Limit thread count for NumPy/OpenBLAS/MKL

[turswiming]

Set environment variables to limit thread count for NumPy and related libraries.

In certain environments, OpenBLAS/MKL may still have thread pooling.

[Kin-Zhang]

It looks good to me, but I'm wondering is there any speed comparison for this change?

[turswiming]

Thanks for your reply, I will conduct an ablation study and provide environment details few days latter.

This will fix the delay caused by OpenBLAS / MKL in certain environments.

[turswiming]

Using 8 nproc:
With this fix: CPU usage ~800%, 47 data points processed in 11:17 min (avg. 14.4 sec/data)
Without this fix: CPU usage ~1600%, 8 data points processed in 12:30 min (avg. 93.8 sec/data)

The root cause is that when running inside a Docker container, MKL fails to detect the true vCPU limit and instead attempts to utilize all physical CPUs of the host system. This leads to extreme oversubscription, which severely blocks the training procedure and degrades performance.

[turswiming]

environment.yml

[Kin-Zhang]

Thanks for the quick reply.

The root cause is that when running inside a Docker container, MKL fails to detect the true vCPU limit and instead attempts to utilize all physical CPUs of the host system.

Does this mean it only affect the program run inside the Docker or it's a general affect?