I did some more tests yesterday, and I think I found out where the issue happens, but not why still.
Is it possible to see the debug log on any emulaor (printfs ?)
or to attach a debugger to an emulator?

I use a hash map with the free list of hardware sprite id, and get them from this hash map in a loop.
It's a templated hash map and I suspect weird behavior from it from the NeoGeo, on pc it looks fine when I look at the values...
If I don't use this hash map, the game has no glitch and is very fast.
And even when it slows down, I didn't see the issue again.

Anyway, some bug in my code somewhere.