diff --git a/CLAUDE.md b/CLAUDE.md index ee69756150..6a19a87161 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,3 +17,7 @@ gofmt -s -l . ``` This command should produce no output if all files are properly formatted. + +## Benchmarking + +See [benchmark/CLAUDE.md](benchmark/CLAUDE.md) for benchmark usage, environment variables, and comparison workflows. diff --git a/app/benchmark/benchmark.go b/app/benchmark/benchmark.go index 6d454f064b..05723059cd 100644 --- a/app/benchmark/benchmark.go +++ b/app/benchmark/benchmark.go @@ -17,7 +17,7 @@ // proposalCh := gen.StartProposalChannel(ctx, benchLogger) // // The generator can be configured via JSON config files that follow the sei-load -// LoadConfig format. See scripts/scenarios/ for example configurations. +// LoadConfig format. See benchmark/scenarios/ for example configurations. package benchmark import ( diff --git a/benchmark/CLAUDE.md b/benchmark/CLAUDE.md new file mode 100644 index 0000000000..6ed1b4d992 --- /dev/null +++ b/benchmark/CLAUDE.md @@ -0,0 +1,87 @@ +# Benchmark + +## Single scenario + +```bash +GIGA_EXECUTOR=true GIGA_OCC=true DEBUG=true benchmark/benchmark.sh +``` + +TPS is logged every 5s as `tps=` (with ANSI color codes). To capture output and extract TPS: + +```bash +LOG_FILE=/tmp/bench.log DEBUG=true benchmark/benchmark.sh + +# Extract TPS values +sed 's/\x1b\[[0-9;]*m//g' /tmp/bench.log | sed -n 's/.*tps=\([0-9.]*\).*/\1/p' +``` + +Available scenarios in `benchmark/scenarios/`: `evm.json` (default), `erc20.json`, `mixed.json`, `default.json`. + +```bash +# Use a different scenario +BENCHMARK_CONFIG=benchmark/scenarios/erc20.json benchmark/benchmark.sh +``` + +## Environment variables + +### benchmark.sh + +| Var | Default | Purpose | +|-----|---------|---------| +| `BENCHMARK_PHASE` | `all` | `init` (build+init+configure), `start` (run node), `all` (both) | +| `SEI_HOME` | `$HOME/.sei` | Final chain data dir. If != ~/.sei, init in ~/.sei then `mv` | +| `PORT_OFFSET` | `0` | Added to all ports (RPC, P2P, pprof, gRPC, etc.) | +| `SEID_BIN` | `""` | Pre-built binary path. If set, skip build + copy to ~/go/bin/seid | +| `LOG_FILE` | `""` | Redirect seid output to file | +| `BENCHMARK_CONFIG` | `$SCRIPT_DIR/scenarios/evm.json` | Scenario config file (absolute path resolved from script location) | +| `BENCHMARK_TXS_PER_BATCH` | `1000` | Transactions per batch | +| `GIGA_EXECUTOR` | `false` | Enable evmone-based EVM executor | +| `GIGA_OCC` | `false` | Enable OCC for Giga Executor | +| `DB_BACKEND` | `goleveldb` | Database backend (goleveldb, memdb, cleveldb, rocksdb) | +| `MOCK_BALANCES` | `true` | Use mock balances during benchmark | +| `DISABLE_INDEXER` | `true` | Disable indexer for benchmark (reduces I/O overhead) | +| `DEBUG` | `false` | Print all log output without filtering | + +### benchmark-compare.sh + +Inherits all benchmark.sh vars via delegation. Additionally: + +| Var | Default | Purpose | +|-----|---------|---------| +| `DURATION` | `600` | How long (seconds) to run each node before stopping | +| `GIGA_EXECUTOR` | **`true`** | Overrides benchmark.sh default (false) | +| `GIGA_OCC` | **`true`** | Overrides benchmark.sh default (false) | +| `DB_BACKEND` | `goleveldb` | Forwarded to build and init phases | + +**Note:** `GIGA_EXECUTOR` and `GIGA_OCC` default to `true` in the compare script but `false` in benchmark.sh. The compare script is designed for performance comparison where Giga Executor is typically enabled. + +## Parallel multi-scenario comparison + +Use `benchmark/benchmark-compare.sh` to run multiple git commits side-by-side (minimum 2 scenarios required): + +```bash +benchmark/benchmark-compare.sh \ + pre-opt=fd2e28d74 \ + lazy-cms=82acf458d \ + lazy-cms-fix=37a17fd02 +``` + +Each scenario gets its own binary, home dir, and port set (offset by 100). Results are printed at the end with median/avg/min/max TPS. Raw data in `/tmp/sei-bench/