feat(benches): add benchmarks for historic and latest events scanning #254

yug49 · 2025-12-13T10:54:40Z

Related to #229

Hey!

This PR introduces a benchmarking system for Event Scanner using Criterion.rs to measure performance impact of changes to the scanner. Currently drafting, Bencher CI integration coming in follow-up.

What's Included

New benches Crate Structure

benches/
├── Cargo.toml                           # Benchmark crate config
├── src/
│   └── lib.rs                           # Shared utilities (Anvil setup, contract deployment, event generation)
└── benches/
    ├── historic_scanning.rs             # Historic mode benchmarks
    └── latest_events_scanning.rs        # Latest events mode benchmarks

Benchmarks Implemented

Mode	Event Counts	What It Measures
Historic	10K, 50K, 100K	Time to scan all events from block 0 to latest
Latest Events	100, 1K, 10K, 50K	Time to fetch the N most recent events from a 100K event pool

example of Historic:

How Regression Testing Works

Criterion stores baseline results in target/criterion/<benchmark>/base/. On subsequent runs, it compares new measurements against this baseline and reports:

historic_scanning/events/10000
                        time:   [30.963 ms 36.506 ms 40.598 ms]
                        thrpt:  [246.32 Kelem/s 273.93 Kelem/s 322.97 Kelem/s]
                 change: [-2.12% -1.01% +0.12%] (p = 0.12 > 0.05)
                        No change in performance detected.

If a change introduces a regression, you'll see something like:

                 change: [+15.2% +18.4% +21.1%] (p = 0.00 < 0.05)
                        Performance has regressed.

This makes it easy to catch slowdowns before merging

Running Benchmarks

# All benchmarks
cargo bench --manifest-path benches/Cargo.toml

# Specific benchmark
cargo bench --manifest-path benches/Cargo.toml --bench historic_scanning
cargo bench --manifest-path benches/Cargo.toml --bench latest_events_scanning

# Filter by event count
cargo bench --manifest-path benches/Cargo.toml -- "historic_scanning/events/10000"

Next Steps

Bencher CI integration: Add GitHub Actions workflow for on-demand benchmarking with historical tracking via Bencher

benches/benches/latest_events_scanning.rs

benches/src/lib.rs

Signed-off-by: yug49 <148035793+yug49@users.noreply.github.com>

yug49 · 2025-12-17T04:54:19Z

Hey @0xNeshi
I have made the required changes in the Criterion part as you mentioned.
Please review once, and if everything's good, I can move forward with the Bencher integration.

0xNeshi · 2025-12-17T07:04:45Z

All good 👍

yug49 · 2025-12-21T22:09:32Z

GM @0xNeshi,
I have completed the integration of Bencher with the following setup:

Workflow Architecture

Three GitHub Actions workflows handle different scenarios:

benchmarks.yml - Runs on push to main and manual dispatch. This workflow uploads benchmark results to Bencher and establishes the baseline for regression detection. Path filters ensure benchmarks only run when relevant code changes (src/, benches/, Cargo.toml, Cargo.lock).
pr_benchmarks_run.yml - Runs benchmarks on pull requests. This workflow does not have access to secrets, making it safe for fork PRs. Results are saved as artifacts.
pr_benchmarks_track.yml - Triggered after the PR benchmark run completes. This workflow downloads the artifacts and uploads them to Bencher for comparison against the base branch. It posts comparison results as comments on the PR.

Required Setup

BENCHER_API_TOKEN as a repository secret
BENCHER_PROJECT as a repository variable

How It Works

When code is merged to main, benchmarks run and upload results to establish the baseline. For pull requests, benchmarks run in a fork-safe workflow and results are compared against the baseline. The --start-point-reset flag ensures PR branches remain ephemeral and do not accumulate historical data.

0xNeshi

Excellent work, let's polish now

0xNeshi · 2025-12-22T10:55:26Z

.github/workflows/benchmarks.yml

+        with:
+          egress-policy: audit
+
+      - name: Free up disk space


Is this step required for our use case? Does event scanner benching need this additional space?

Yes, I added this additional step since running benches was taking a lot of space, sometimes it exceeded the github runners' limited space (~14GB).

Here:

Even after adding this step, the github CI was often timing out for the latest events benches, because it was trying to generate 100,000 events which is a very heavy operation that can timeout on CI.

Sometimes, it was getting stuck at this step for > 30 mins:

Run cargo bench --manifest-path benches/Cargo.toml --bench latest_events_scanning 2>&1 | tee latest_results.txt Updating crates.io index Compiling event-scanner v0.9.0-alpha (/home/runner/work/Event-Scanner/Event-Scanner) Compiling event-scanner-benches v0.9.0-alpha (/home/runner/work/Event-Scanner/Event-Scanner/benches) Finished `bench` profile [optimized] target(s) in 17.61s Running benches/latest_events_scanning.rs (target/release/deps/latest_events_scanning-c7b9440175034d98) Gnuplot not found, using plotters backend Setting up environment with 100000 total events...

To counter this, I reduced the total events for this case to 50k from 100k, which I think is also a reasonable number for real world scenarios.

Rust compilation is the main disk consumer, unchanged by event count, the cleanup step runs in <5 seconds, which I think is a negligible cost. It is a cheap safety net that prevents potential flaky failures.

Alright, makes sense.

Once blocks are loaded into Anvil using a dump file, double check if this step becomes redundant and if it's possible to still keep the event count at 100,000.

.github/workflows/benchmarks.yml

0xNeshi · 2025-12-22T11:24:19Z

benches/benches/historic_scanning.rs

+        }
+    }
+
+    assert_eq!(log_count, expected_count, "expected {expected_count} events, got {log_count}");


On second thought, this validation unnecessarily affects bench result, while also being kind of redundant - the project has extensive integration tests with this validation.

Remove the assertions and stop tracking log count

0xNeshi · 2025-12-22T11:28:49Z

benches/benches/historic_scanning.rs

+        let env: BenchEnvironment = rt.block_on(async {
+            let config = BenchConfig::new(event_count);
+            setup_environment(config).await.expect("failed to setup benchmark environment")
+        });


There's really no reason to setup a new Anvil node for each bench run, it just adds to total bench execution time.
Let's make an optimization to the both benches.

Instead of setting up a new Anvil node for each event_count case, let's do the following:

setup a single Anvil node with 100,000 events

get latest block number

for historic bench:

bench with 3 different block ranges that roughly correspond to:

first 1/10 of all blocks

first 1/2 of all blocks

all blocks

for latest events - leave as-is, i.e. bench latest 10k events, then 50k, then 100k (all) events

Related #254 (comment)

0xNeshi · 2025-12-22T11:30:10Z

benches/benches/historic_scanning.rs

+}
+
+fn historic_scanning_benchmark(c: &mut Criterion) {
+    let rt = tokio::runtime::Runtime::new().expect("failed to create tokio runtime");


Create a singleton tokio runtime that can be reused across bench runs.
Something like:

use std::sync::OnceLock; static RUNTIME: OnceLock<tokio::runtime::Runtime> = OnceLock::new(); fn get_runtime() -> &'static tokio::runtime::Runtime { RUNTIME.get_or_init(|| { tokio::runtime::Runtime::new().expect("failed to create tokio runtime") }) } fn historic_scanning_benchmark(c: &mut Criterion) { let rt = get_runtime(); // ... rest of benchmark ... }

0xNeshi · 2025-12-22T11:34:11Z

benches/src/lib.rs

+sol! {
+    // Built directly with solc 0.8.30+commit.73712a01.Darwin.appleclang
+    #[sol(rpc, bytecode="608080604052346015576101b0908161001a8239f35b5f80fdfe6080806040526004361015610012575f80fd5b5f3560e01c90816306661abd1461016157508063a87d942c14610145578063d732d955146100ad5763e8927fbc14610048575f80fd5b346100a9575f3660031901126100a9575f5460018101809111610095576020817f7ca2ca9527391044455246730762df008a6b47bbdb5d37a890ef78394535c040925f55604051908152a1005b634e487b7160e01b5f52601160045260245ffd5b5f80fd5b346100a9575f3660031901126100a9575f548015610100575f198101908111610095576020817f53a71f16f53e57416424d0d18ccbd98504d42a6f98fe47b09772d8f357c620ce925f55604051908152a1005b60405162461bcd60e51b815260206004820152601860248201527f436f756e742063616e6e6f74206265206e6567617469766500000000000000006044820152606490fd5b346100a9575f3660031901126100a95760205f54604051908152f35b346100a9575f3660031901126100a9576020905f548152f3fea2646970667358221220471585b420a1ad0093820ff10129ec863f6df4bec186546249391fbc3cdbaa7c64736f6c634300081e0033")]
+    contract BenchCounter {


Suggested change

contract BenchCounter {

contract Counter {

align name with other contracts in examples

0xNeshi · 2025-12-22T11:44:35Z

benches/benches/historic_scanning.rs

+    // - 10 samples (iterations)
+    // - Long measurement time to accommodate heavy loads
+    group.sample_size(10);
+    group.measurement_time(std::time::Duration::from_secs(120));


Let's also increase warm up time to at least 5 seconds

Co-authored-by: Nenad <xinef.it@gmail.com>

LeoPatOZ · 2025-12-22T15:24:16Z

@0xNeshi what do you think about creating multiple dump files and starting anvil from that state instead of having to recreate it every time we start a bench

0xNeshi · 2025-12-23T07:21:28Z

@0xNeshi what do you think about creating multiple dump files and starting anvil from that state instead of having to recreate it every time we start a bench

Even better 👍

Co-authored-by: Nenad <xinef.it@gmail.com>

0xNeshi reviewed Dec 15, 2025

View reviewed changes

benches/benches/latest_events_scanning.rs Outdated Show resolved Hide resolved

0xNeshi reviewed Dec 15, 2025

View reviewed changes

benches/src/lib.rs Outdated Show resolved Hide resolved

yug49 added 5 commits December 17, 2025 10:17

feat(benches): add Criterion benchmarks for historic scanning mode

47fd087

Add latest events scanning benchmark

329e2a4

Update Cargo.lock

988adea

Signed-off-by: yug49 <148035793+yug49@users.noreply.github.com>

feat(benches): optimize benchmarks and simplify BenchConfig API

8ff77f3

chore: update Cargo.lock with latest dependency versions

fa67e5b

yug49 force-pushed the feat/benchmarks branch from 465a88a to fa67e5b Compare December 17, 2025 04:49

yug49 added 9 commits December 21, 2025 13:30

Merge branch 'OpenZeppelin:main' into feat/benchmarks

7ec2a3d

feat: add Bencher CI integration for performance tracking

28c052c

fix: remove invalid --noplot flag from benchmark command

f7a3d81

perf: run benchmarks in parallel jobs to avoid disk space issues

70c6b17

fix: add disk cleanup step to free space before benchmarks

b8146e8

perf: reduce latest events benchmark from 100K to 50K events

a53baef

fix: correct Bencher integration for baseline tracking

e541a36

fix: use dynamic branch name in Bencher upload

651bf30

fix: remove --ci-only flag from PR tracking workflow

44e4ead

yug49 marked this pull request as ready for review December 21, 2025 22:10

yug49 requested review from LeoPatOZ and pepebndc as code owners December 21, 2025 22:10

0xNeshi reviewed Dec 22, 2025

View reviewed changes

yug49 and others added 4 commits December 22, 2025 19:32

Update .github/workflows/benchmarks.yml

d5d7e9e

Co-authored-by: Nenad <xinef.it@gmail.com>

Update .github/workflows/benchmarks.yml

4825627

Co-authored-by: Nenad <xinef.it@gmail.com>

Update .github/workflows/benchmarks.yml

6a1e923

Co-authored-by: Nenad <xinef.it@gmail.com>

Merge branch 'OpenZeppelin:main' into feat/benchmarks

6f036a4

refactor: update benchmark config and remove PR workflows

23c741c

yug49 and others added 9 commits December 26, 2025 04:10

Update .github/workflows/benchmarks.yml

dc8f23d

Co-authored-by: Nenad <xinef.it@gmail.com>

Merge branch 'OpenZeppelin:main' into feat/benchmarks

798f9c5

Remove log count tracking and assertion from historic scanning benchmark

ec4ed63

Rename BenchCounter to Counter

78ed3bd

bench: add 5s warm-up time to benchmark groups

97cc1d9

feat: use singleton tokio runtime in benchmarks

4b71d12

refactor(benches): replace panics with proper error handling

ae58054

Update .github/workflows/benchmarks.yml

29acfb8

Co-authored-by: Nenad <xinef.it@gmail.com>

fix: make benchmark jobs consistent with dependency versions

089d5af

feat(benches): add benchmarks for historic and latest events scanning #254

Are you sure you want to change the base?

feat(benches): add benchmarks for historic and latest events scanning #254

Uh oh!

Conversation

yug49 commented Dec 13, 2025

What's Included

New benches Crate Structure

Benchmarks Implemented

How Regression Testing Works

Running Benchmarks

Next Steps

Uh oh!

Uh oh!

Uh oh!

yug49 commented Dec 17, 2025

Uh oh!

0xNeshi commented Dec 17, 2025

Uh oh!

yug49 commented Dec 21, 2025

Uh oh!

0xNeshi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LeoPatOZ commented Dec 22, 2025

Uh oh!

0xNeshi commented Dec 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants