Improve benchmark stability with cpusets, nice, and disabling of hyper threading

[rwstauner]

This adds to run_benchmarks.rb several additional linux features which can help to make the benchmark results more reliable:

• set scheduler priority (niceness) high
• setup two cpusets, one for benchmarks and one for everything else to move existing processes off of benchmark cpu
• disable hyper threading for benchmark cpus
• move parent process off of cpu that will be used for benchmarks

You can see this improve the results when running the same ruby version twice:
it increases the number of benchmarks that compare at 1.000 on two different computers:

By putting all this logic into a module we can easily reuse it from yjit-metrics.

I think it would make sense to eventually move the existing checks for turbo into this module as well.

[rwstauner]

For the curious, it looks like it isn't finding any cpuset files in CI, it's only setting the nice value there.

[maximecb]

Looks pretty good to me but I will also tag Alan and Kokubun to review since it's very important that we get benchmarking right.

[k0kubun]

Unlike misc/graph.rb that supports both standalone execution and use as a library or other standalone scripts, these two files (misc/stats.rb and misc/benchmark_mode.rb) only make sense as a library for run_benchmarks.rb. So it feels confusing to me to have them under misc/.

WDYT about moving misc/stats.rb and misc/benchmark_mode.rb to lib/? And it might be also nice to split BenchmarkMode::Xxx into lib/benchmark_mode/xxx.rb?

[rwstauner]

Sounds good

[k0kubun]

Can we move the turbo boost one (and maybe check_pstate as well. I prefer it to still not have disable/disengage though) into BenchmarkMode as well? It seems inconsistent (and thus confusing) that some things live in run_benchmarks.rb and others in BenchmarkMode while they seem to serve the same purpose.

[rwstauner]

Yeah we can move it all under one namespace, I was just waiting to get some general feedback before doing more work on it 👍

[k0kubun]

Part of the point of --turbo was to obviate sudo (when we're only interested in stats or testing the behavior), but this seems to add more sudo requirements that are not disabled by --turbo.

Can we rename --turbo to --no-sudo and let it turn off all include Sudo tasks? I'm also fine with making it aliased to --no-benchmark-mode to simplify the implementation (but --no-benchmark-mode itself seems a bit too long to me to type locally).

[rwstauner]

Thanks for the feedback.
I would like to consolidate it a bit more and yeah we can have a flag to disable everything.
I'll work on it some more.

[eightbitraptor]

Since #434 we now have a lib directory! So there's precedent for library code in there. I think this would be a really good addition.

[eregon]

Can we rename --turbo to --no-sudo

+1 I also run in this situation and specifically I'm on AMD where --turbo doesn't disable all commands using sudo.

[eregon]

Would love if this PR gets merged, I was looking for --no-sudo the other day.