Add benchmark framework and benchmarks by jonbodner-buf · Pull Request #149 · bufbuild/protovalidate-es

jonbodner-buf · 2026-05-14T14:36:47Z

Mirrors protovalidate-go's validator_bench_test.go in a new private packages/protovalidate-bench workspace so runtime cost can be tracked across changes and compared cross-language.

Uses tinybench, hand-built deterministic fixtures, and writes JSON results to .tmp/bench/.

Adds a checkbench script to diff two runs with a noise-aware regression threshold and non-zero exit on regression, suitable for gating PRs.

Mirrors protovalidate-go's validator_bench_test.go in a new private packages/protovalidate-bench workspace so runtime cost can be tracked across changes and compared cross-language. Uses tinybench, hand-built deterministic fixtures, and writes JSON results to .tmp/bench/. Adds a checkbench script to diff two runs with a noise-aware regression threshold and non-zero exit on regression, suitable for gating PRs.

Eleven near-identical .bench.ts files have collapsed to four: cases.ts lists every (name, schema, fixture) triple in one place, validate.bench.ts iterates it for the per-case validate-time benches, and compile.bench.ts plus standard-schema.bench.ts look up curated subsets by name. Adding a benchmark is now a one-row append to cases.ts plus a fixture in fixtures.ts instead of new-file + import + register call in bench.ts. Bench output is byte-identical: same 17 tasks, same names, same ordering, deltas within the noise floor.

jonbodner-buf · 2026-05-14T21:04:03Z

After this PR is approved there are a series of additional PRs that implement the protovalidate native rule support for ES. Each PR builds on the previous ones.

ajeetdsouza · 2026-05-24T10:26:33Z

+
+The shortcuts `latest` and `previous` resolve to the newest and second-newest
+JSON files in `.tmp/bench/` (by mtime). Calling with only one argument
+defaults the baseline to `previous`.


Calling with only one argument defaults the baseline to previous.

latest and previous would collide with filenames - is there any use case for keyword processing here? I think we should just define the behavior for 0 arguments and 1 argument, and avoid the keywords altogether - that would also make this documentation clearer.

I've updated the logic to the following:

no arguments: use the two most recent files (older is baseline, newer is current)

1 argument: use the named file as the baseline and the latest file as current

2 arguments: use the named files, first is baseline, second is current

ajeetdsouza · 2026-05-24T22:00:09Z

+const BENCH_DIR = ".tmp/bench";
+const DEFAULT_THRESHOLD = 5;
+
+function parseArgs(argv) {


We could probably use parseArgs from node:util here. yargs and commander are also in the dependency tree.

using the node:util parseArgs now. there's also validation of directory and files existing.

Signed-off-by: Jon Bodner <jbodner@buf.build>

timostamm · 2026-05-27T15:11:51Z

+    "generate": "buf generate",
+    "postgenerate": "license-header src/gen",
+    "bench": "tsx src/bench.ts",
+    "checkbench": "node scripts/checkbench.js",


What do you think about promoting checkbench.js to be a sibling to bench.ts?

We do have some vanilla JS scripts in various repositories, but not by choice. Limitations of the repository setup make it difficult to use TS. What we do in those cases is to add typedef annotations (example), which gives some IDE support.

In this case however, the package is purely internal, doesn't have build artifacts, and we can easily use TS.

I can convert it to typescript and move it.

timostamm · 2026-05-27T15:24:36Z

+  "dependencies": {
+    "@bufbuild/protobuf": "^2.11.0",
+    "@bufbuild/protovalidate": "^1.2.0",
+    "tinybench": "^3.1.1"


tinybench is a solid choice 👍

But I recently stumbled upon mitata. It seems to have some really nice features like GC control, minimal overhead, and hardware counters. This could be very useful for incremental performance improvements in cel-es and protobuf-es. Do you think it would be worth taking a look into it here? The features might not be immediately useful for the native rules implementation, but it seems smart to use the same benchmarking tooling across the board.

re-implemented with mitata. I have seen wild swings on the gc/heap numbers, even with the --expose-gc flag set and the .gc('inner') method call added to the mitata benchmarks. On the plus side, the native rules will probably make this better.

I can reproduce the swings.

They go away when .gc("inner") is removed. This is a sharp edge with mitata and our allocation-heavy benchmarks: gc("inner") forces a GC before every sample, but charges it against the per-bench CPU time budget, and the alloc-heavy benches end up with too few samples. There are no knobs in the simple API.

There is also a separate issue: mitata doesn't support process isolation. So the first benchmarks will warm up the JIT, and the following benchmarks hitting the same code paths benefit from it. This means changing the order of benchmarks will change results.

I'm not sure where to go from here, but permanently porting the Go benchmark suite over with similar ergonomics doesn't seem feasible to me without investing significant time. Maybe a very simple script measuring wall-clock time is good enough for confirming the effectiveness of the native rules?

I think that's effectively what tinybench was doing. Should I revert back to it?

The benchmark is now updated to run multiple times and to customize gc behavior depending on the test type (compile tests use gc('inner') others don't). The numbers should be more stable now.

This looks like it fixes most of the issues. If we want to cut the noise completely, could we run separate runs for the timing and heap measurements? With already running it 5x, that might make it too time intensive...the cpu time could be lower for the heap runs though.

…. Update documentation to reflect its current arguments. Signed-off-by: Jon Bodner <jbodner@buf.build>

Signed-off-by: Jon Bodner <jbodner@buf.build>

…idate-es into jbodner/add-benchmarks

Signed-off-by: Jon Bodner <jbodner@buf.build>

… gc settings for different tests. Signed-off-by: Jon Bodner <jbodner@buf.build>

Signed-off-by: Jon Bodner <jbodner@buf.build>

jonbodner-buf added 2 commits May 13, 2026 16:46

jonbodner-buf changed the title ~~Jbodner/add benchmarks~~ Add benchmark framework and benchmarks May 14, 2026

jonbodner-buf requested a review from timostamm May 14, 2026 14:39

fix license header in checkbench.js

a37e50a

jonbodner-buf requested review from ajeetdsouza and removed request for timostamm May 14, 2026 21:03

Merge branch 'main' into jbodner/add-benchmarks

e7af3f8

jonbodner-buf requested a review from ejowers May 22, 2026 15:28

ajeetdsouza reviewed May 24, 2026

View reviewed changes

timostamm self-requested a review May 26, 2026 09:40

jonbodner-buf added 2 commits May 26, 2026 14:43

improve benchmark file selection and argument parsing.

efb400f

improve threshold validation

35b0ab0

Signed-off-by: Jon Bodner <jbodner@buf.build>

timostamm reviewed May 27, 2026

View reviewed changes

jonbodner-buf added 9 commits May 27, 2026 12:32

relocate checkbench to the src directory and convert it to typescript…

edc4761

…. Update documentation to reflect its current arguments. Signed-off-by: Jon Bodner <jbodner@buf.build>

remove unneeded file ignores from biome.json

5ec5fb2

Signed-off-by: Jon Bodner <jbodner@buf.build>

Merge branch 'main' into jbodner/add-benchmarks

f0cee20

fix formatting

a2ffbb2

Signed-off-by: Jon Bodner <jbodner@buf.build>

Merge branch 'jbodner/add-benchmarks' of github.com:bufbuild/protoval…

49b3106

…idate-es into jbodner/add-benchmarks

switch benchmarking from tinybench to mitata

492bc41

Signed-off-by: Jon Bodner <jbodner@buf.build>

fix formatting

70a2969

Signed-off-by: Jon Bodner <jbodner@buf.build>

improve benchmark stability by adding multi-run support and different…

1fc15f5

… gc settings for different tests. Signed-off-by: Jon Bodner <jbodner@buf.build>

additional tweaks to improve noise

52ede79

Signed-off-by: Jon Bodner <jbodner@buf.build>

Conversation

jonbodner-buf commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonbodner-buf commented May 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jonbodner-buf commented May 14, 2026 •

edited

Loading