Additional benchmark statistics

[N-Wouda]

Describe the solution you'd like

Right now PyVRP's CLI calculates a few aggregate measures: avg. iterations, avg. runtime, and avg. objective. These are all simple statistics, and can be quite limited in understanding actual performance since they do not compare against the BKSs. Statistics I am also interested in are e.g., avg/min gap to BKS, and for example avg/max primal integral to BKS, and possibly more I have not yet considered.

We could incorporate something like this into the benchmark output by extending it with a "BKS list" argument, which takes BKSs in the same order as the given instances. These BKSs are then used to compute the statistics listed above.

[leonlan]

One of our main performance indicators is the gap w.r.t. the best-known solutions. I currently go through a cumbersome process to compute the gaps from a benchmark run. It would be very convenient to immediately display the gap after a benchmark run, for example, like this:

Instance     OK  Obj.     Iters. (#)  Time (s)  Gap (%)
-----------  --  -------  ----------  --------  -------
X-n1001-k43   Y  76687.0          25      2.06     5.99
 X-n106-k14   Y  26511.0         372     2.004     0.57

     Avg. objective: 51599
    Avg. iterations: 198
      Avg. run-time: 2.03s
       Total not OK: 0
           Avg. gap: 3.28%

The question is: do we want this, and if so, how do we provide such an interface?

Suppose we do want this. Here is a proposal:

• Pass a bks_dir argument to solve. This directory contains the best-known solutions to each of the instances passed to solve. The solutions are formatted in VRPLIB-style and should be named the same as the instance file, but with the .sol extension instead.
• If bks_dir is passed, we add a "Gap (%)" column to the results and include the "Avg. gap" summary. Otherwise, we keep the results as original.
• If BKS files are missing for the solved instances, then report float('nan'). Or simply raise.

---

Or, as proposed above, passing bks in the order of instances. That's more explicit.

pyvrp instances/CVRP/*.vrp --bks instances/CVRP/*.sol --seed 0 --max_runtime 10

The ordering does not matter because we are sorting the instances. If there is no perfect 1-to-1 matching between the instances and bks, then we can report 'nan' as gap for the instance.