From 7c2686f05b72a2e21831a98ea85d7edb65d5807d Mon Sep 17 00:00:00 2001 From: pdrobnjak Date: Fri, 13 Feb 2026 13:10:55 +0100 Subject: [PATCH 01/47] Add benchmark-compare script and agent context Co-Authored-By: Claude Opus 4.6 --- CLAUDE.md | 57 ++++++ scripts/benchmark-compare.sh | 361 +++++++++++++++++++++++++++++++++++ 2 files changed, 418 insertions(+) create mode 100755 scripts/benchmark-compare.sh diff --git a/CLAUDE.md b/CLAUDE.md index ee69756150..86bedad48d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,3 +17,60 @@ gofmt -s -l . ``` This command should produce no output if all files are properly formatted. + +## Benchmarking + +### Single scenario + +```bash +GIGA_EXECUTOR=true GIGA_OCC=true DEBUG=true scripts/benchmark.sh +``` + +TPS is logged every 5s as `tps=` (with ANSI color codes). Extract with: + +```bash +grep "tps=" /tmp/sei-benchmark.log | sed 's/\x1b\[[0-9;]*m//g' | sed -n 's/.*tps=\([0-9.]*\).*/\1/p' +``` + +### Parallel multi-scenario comparison + +Use `scripts/benchmark-compare.sh` to run multiple git commits side-by-side: + +```bash +scripts/benchmark-compare.sh \ + pre-opt=fd2e28d74 \ + lazy-cms=82acf458d \ + lazy-cms-fix=37a17fd02 +``` + +Each scenario gets its own binary, home dir, and port set (offset by 100). Results are printed at the end with median/avg/min/max TPS. Raw data in `/tmp/sei-bench/