Benchmarking the examples

[dmuenz]

I request that devtools provide some convenient way to time all of a package's examples. This would report both the total time and all of the individual times for each topic, since I believe CRAN evaluates both things.

Maybe this could be added to run_examples(), having it report all the times at the end of its run?

Thanks for your consideration.

[jennybc]

You're going to find that the absolute amount of time varies greatly from machine to machine and from other factors, like what other work is being done. For example, the load on CRAN's machines can differ quite a lot. So you have to sort of accept that you mostly just want to keep examples very snappy.

But if you really want to get some granular info, every time you run R CMD check, it leaves such timings behind. Here's me looking at the timings of vroom's examples:

x <- devtools::check("~/work/vroom")
#> ...
#> ── R CMD check ─────────────────────────────────────────────────────────────────
#> * using log directory ‘/private/tmp/Rtmpx2wvij/file7c2d3d3c37cc/vroom.Rcheck’
#> ...
#> * DONE
#> 
#> Status: 1 NOTE
#> See
#>   ‘/private/tmp/Rtmpx2wvij/file7c2d3d3c37cc/vroom.Rcheck/00check.log’
#> for details.

x$checkdir |>
  list.files(full.names = TRUE, pattern = "*.timings") |>
  readLines() |>
  writeLines()
#> name user    system  elapsed
#> as.col_spec  0.001   0.000   0.001   
#> cols 0.001   0.000   0.001   
#> date_names   0.002   0.001   0.002   
#> gen_tbl  0.028   0.004   0.034   
#> generators   0.005   0.001   0.007   
#> guess_type   0.011   0.001   0.015   
#> locale   0.001   0.000   0.002   
#> output_column    0   0   0   
#> spec 0.037   0.009   0.061   
#> vroom    0.216   0.023   0.229   
#> vroom_altrep 0.001   0.000   0.001   
#> vroom_example    0.001   0.000   0.002   
#> vroom_fwf    0.073   0.004   0.077   
#> vroom_lines  0.000   0.001   0.002   
#> vroom_progress   0   0   0   
#> vroom_str    0.011   0.001   0.012   
#> vroom_write  0.001   0.001   0.002   

We don't use run_examples() much, instead we just exercise the examples in the context of check() and building pkgdown sites. So I don't see devtools gaining additional functionality around examples. Hopefully the timing from R CMD check will help you out.