I saw this same behaviour, and the worst part was it was almost impossible to know when it would case an issue. I started a thread here that I think may help you solve the Neopixel issue by using the RMT feature to drive the leds.
I never got around to contributing the code back to the neopixel library as it was basically a rewrite of the bit-bang method so didn’t seem like a straightforward fix, plus I had no idea how to properly fix the ESP-IDF dependency.
@TheCrypt0 if you’re struggling let me know and I’ll find a way to share it with you.