Noob questions + update

[erezsh]

So, I know it took me a long time, but I started working on automatically generating graphs of the benchmarks:

It's SVG, so it's both lightweight and pretty. It's also just an initial attempt, so don't judge it too harshly :)

As part of that effort, I have a few questions:

• When I generate the JSON, it doesn't contain any memory information, just runtime. How can I get that?

• Is seems that the parsers only run for only 1 iteration. How can I fix that? Can I set a minimum of 10 iters, for example?

• Is there an easy switch to disable benchmarks for compilation and only allow parsing?

[goodmami]

Looks good! SVGs will render in GitHub READMEs and you can include custom CSS if it's embedded in the SVG file (see the logos I created for the Wn project and inspect the source for an example). If we want Javascript, though, we'd have to view the site through GitHub Pages as github.com rendering is sanitized for that. There's also native SVG tooltips through the <title> element, but support is unknown for many browsers (works for me on Firefox and Chrome).

• When I generate the JSON, it doesn't contain any memory information, just runtime. How can I get that?

I'm using pytest-benchmark which doesn't yet do memory, but see their issue 28, which has a possible solution. It also links to pytest-austin, which I haven't seen before. Might be worth checking out.

Otherwise, on Unix systems we can use Python's resource.getrusage() function to get the ru_maxrss for the process. This just gets it at one point in time and doesn't poll, so I'm not sure how accurate it would be. We can also use the Unix time command to get memory (see here).

• Is seems that the parsers only run for only 1 iteration. How can I fix that? Can I set a minimum of 10 iters, for example?

The library auto-calibrates the number of iterations and rounds. See here. The time for a round is the average iteration time within a round. The reported stats are based on the rounds. You can adjust the bounds of the time allowed for a round with --benchmark-min-time and --benchmark-max-time. Since the tests are setup to parse one big thing (e.g., a 5Mb JSON file), and this can take a while, it can only do 1 iteration within these bounds (actually it exceeds the max time, but the max is only to determine whether to run another iteration and it doesn't stop the current one). It will always do the minimum number of rounds, which is 5.

If you want to manually specify the number of iterations or rounds, there's the "pedantic mode". I'm fine with auto-calibration, but if we want to test parsing small things many times we'll want multiple iterations, which we might achieve by adjusting the min/max time.

• Is there an easy switch to disable benchmarks for compilation and only allow parsing?

Hmm, I don't know really. I use the benchmark groups (e.g., json and json-compile) so the stats are collected separately, but I'm not sure how to filter these out. The compilation tests are fairly quick to run. Are they causing issues with plotting the SVG?