Add some toolchain benchmarks to test_benchmark.py

[sbc100]

These measure the performance of the compiler itself relative to clang.

The output looks like this:

[1/2] test_emcc_compile_hello (test_benchmark.benchmark.test_emcc_compile_hello) ...
        clang: mean: 0.062 (+-0.001) secs  median: 0.062  range: 0.060-0.063  (noise: 2.354%)  (5 runs)   Relative: No baseline recorded yet
         emcc: mean: 0.166 (+-0.004) secs  median: 0.166  range: 0.159-0.171  (noise: 2.374%)  (5 runs)   Relative: 2.69 X slower
ok
[2/2] test_emcc_noop (test_benchmark.benchmark.test_emcc_noop) ...
        clang: mean: 0.044 (+-0.001) secs  median: 0.045  range: 0.042-0.045  (noise: 2.469%)  (5 runs)   Relative: No baseline recorded yet
         emcc: mean: 0.124 (+-0.003) secs  median: 0.124  range: 0.121-0.127  (noise: 2.169%)  (5 runs)   Relative: 2.82 X slower
ok

[kripken]

Can we not add these to the normal toplevel list of benchmarkers? That is, we have a "primes" benchmark, and we could have a "hello world" benchmark. And there could be a Benchmarker that just compiles in emcc or clang and doesn't run. That could run on all the benchmarks in principle.

Then, picking the right Benchmarkers using the env var, and the right benchmarks using benchmark.test_*, you can get the results you want iiuc?

[sbc100]

I lot of the existing benchmarks would not work with these benchmarkers I fear since they anything in the build phase and instead run the compiler in the run phase (which doesn't have any of the build arguments such as filename, shared_args, emcc_args, native_args, native_exec, lib_builder.

For very simple cases it does work to just do the compiling during run. So I added these new benchmarkers to the named list so folks can try them.

[kripken]

Yeah, a lot of other benchmarks might not work - but that's fine I think as users can pick which they want?

Now that we have lines 398-399, do we need 559-562? That is, can't we run these benchmarks the normal way? I am just trying to avoid adding a second "harness" that iterates over benchmarks, that handles using the first benchmarker as the baseline (568-571), etc.

[sbc100]

I tried to re-use do_bennchmark for these, but its different enough I think it warrants keeping this simple/separate do_toolchain_benchmark.

For example, with the toolchain benchmark there is no "src" or "name" to pass.

Maybe we can find a way to merge them later, but this seems OK for now.

[kripken]

Hmm, I'd hope we can make src, name optional somehow, but I guess this new function is not a huge amount of code, so I won't object strongly.

[kripken]

lgtm % typos

[kripken]

Hmm, I'd hope we can make src, name optional somehow, but I guess this new function is not a huge amount of code, so I won't object strongly.

[sbc100]

I added a TODO to try to re-duplicate in the future.