Whenever I have worked on parallel programming, I always plot some variation of the speedup scale.
This is very helpful to detect problems like this one that affects numpy, where you can only see it whenever the contention for some resource is high enough. It might not appear with two or four threads, but sometimes it happens with 10 or 20 threads.
The whole debugging process is quite interesting, using flame graphs, scale plots and the world-famous good-diagram-technique.
Nothing like a good diagram to debug how a system works!