Skip to content

fix(ci): raise AVM check-circuit per-test timeout to 120s#23969

Draft
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-cc-timeout-120s
Draft

fix(ci): raise AVM check-circuit per-test timeout to 120s#23969
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-cc-timeout-120s

Conversation

@AztecBot

@AztecBot AztecBot commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Problem

The avm-check-circuit job failed on next (run 27215957791, exit code 124 = timeout).

The GitHub job failure is just the symptom; the actual failure was a single AVM check-circuit test hitting its per-test timeout:

FAILED: bb-avm avm_check_circuit -v --avm-inputs .../e2e_multiple_blobs/avm-circuit-inputs-tx-0x017d645b...bin (35s) (code: 124)
run_test_cmd '...ISOLATE=1:TIMEOUT=30s:NAME=avm_cc_e2e_multiple_blobs_0x017d645b ...'

This is not a constraint failure — the circuit is valid. The individual test log shows it was simply killed mid-check after exceeding the 30s budget:

  • Trace generation: ~16s
  • Running check (with skippable) circuit over 700560 rows started, then timeout: sending signal TERM at ~30s

Root cause

avm_check_circuit_cmds runs every dumped tx with TIMEOUT=30s and the default 2 CPUs / 8g. Most e2e txs produce small AVM traces that clear check-circuit in 4–8s, but a few (notably e2e_multiple_blobs, ~700k circuit rows) need ~30–40s for trace generation plus check-circuit at 2 CPUs. The 30s budget was right on the edge, so this tx flakes. The triggering commit (#23963, a CI spot-instance change) is unrelated to the AVM — this is a pre-existing borderline-timeout test, not a regression.

Change

Raise the per-test timeout 30s → 120s in avm_check_circuit_cmds.

A longer timeout — rather than raising CPUS — is the targeted fix: the parallel scheduler acquires CPUS cores per test from a global semaphore, so bumping CPUS from 2 to 4 would halve concurrency (96 → 48 on a 192-core runner) for every tx, penalizing the hundreds of small txs to help a handful of large outliers. A longer timeout only lets the rare large tx hold its slot a little longer, with zero impact on the common case. 120s gives ~3x headroom over the observed ~36s worst case.

Validation

The failing run shows the test executing correctly (valid trace, valid in-progress check) and being killed purely by the wall-clock timeout, so additional time resolves it. The change is a single comment + timeout-value edit in the CI test-command generator.


Created by claudebox · group: slackbot

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jun 9, 2026
@AztecBot

AztecBot commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

Flakey Tests

🤖 says: This CI run detected 1 tests that failed, but were tolerated due to a .test_patterns.yml entry.

\033FLAKED\033 (8;;http://ci.aztec-labs.com/1ae79e4b27d3d437�1ae79e4b27d3d4378;;�):  yarn-project/end-to-end/scripts/run_test.sh simple src/e2e_p2p/sentinel_status_slash.parallel.test.ts "slashes the proposer with INACTIVITY when checkpoint validation records invalid" (237s) (code: 0) group:e2e-p2p-epoch-flakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant