Challenge redesigns: 33% → 56.5% baseline (reconciled with main)

[jobordu]

Summary

Post-PR-#1 work on the benchmark suite. PR #1 squash-merged the regression
gate into main; this branch then continued with challenge redesigns while
main moved independently. This PR reconciles the divergence and lands the
redesign work.

Baseline trajectory: 33% → 52% → 55.2% → 56.5% (130/230)

What's in this PR

• Challenge redesigns across 12 files (09, 11, 17, 19–25) —
  reverse-flow, convergence, integration, config-hooks, formal-models, tests,
  and documentation challenges rebuilt for detectable mutations on active
  solver layers.
• lib/mutator.cjs — richer file-modify ops (content / append /
  find+replace) that the redesigned challenges depend on.
• lib/runner.cjs — runSolveFull hang-fix flags; createIsolatedRoot
  deep-copies mutable dirs (COPY_DIRS) so --parallel workers can't clobber
  the real project repo through symlinks.

Reconciliation with main

Merged origin/main to bring its infra forward (workflow_dispatch trigger,
coderlm symbol queries, per-category regression detection, gitignore cleanup).
Conflicts resolved as hybrid: main's isolation model + HEAD's mutator:

File	Resolution
lib/runner.cjs	main's snapshot/restore + coderlm enabled; kept HEAD's hang-fix + COPY_DIRS
lib/mutator.cjs	HEAD's rich mutation ops
bin/nf-benchmark.cjs	main's snapshot model + per-category regression; kept HEAD's parallel path
challenges/05,06,25	HEAD's redesigns (same challenge IDs both sides)

baseline.json and results/trend.jsonl are now untracked, per main's
gitignore (b03d8de) — the baseline is owned by the consuming tool's repo.

Verification

• All 230 challenges validate clean (npm run validate).
• Pre-existing test debt unchanged: the npm test script (node --test test/)
  is independently broken, and a few loadResults adversarial tests race on
  the shared results/ directory. Not gated by CI (benchmark.yml runs the
  benchmark, not unit tests).

🤖 Generated with Claude Code

Summary by CodeRabbit

• Chores
  • Updated benchmark challenge definitions to refocus test scenarios across multiple suites.
  • Enhanced file mutation logic to support more flexible modification strategies, including conditional content replacement and append operations.
  • Refined solver invocation with optimized parameters and improved test directory isolation.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR refactors the benchmark challenge suite from simulating failure modes to discovery-based testing for untraced functionality. Twelve challenge JSON files are updated to create C→R and T→R detection scenarios using CommonJS modules under bin/, while two support files extend the mutation engine and solver integration to enable these new test patterns with faster execution and flexible isolation.

Changes

Benchmark Challenge Suite Refactoring

Layer / File(s)	Summary
Documentation & Traceability Alignment Benchmarks challenges/05-documentation.json, challenges/06-cross-layer-alignment.json	BENCH-051–060 replace CLI/env/file references with nonexistent ones to test documentation discovery. BENCH-063 and BENCH-067 shift from unit-test coverage corruption to traceability matrix status/structure corruption tests, targeting l3_to_tc layer.
Reverse-Flow & Expert Benchmarks challenges/09-reverse-flow.json, challenges/11-expert-challenges.json, challenges/17-devops-deployment.json	BENCH-089–095 create untraced feature handlers, config validators, and state machines under bin/rf-*.cjs. BENCH-106 detects low alignment scores across layers. BENCH-109 discovers untraced rollback handler. New BENCH-139 introduces untraced deployment orchestrator with rollout phases.
Convergence Benchmarks challenges/19-convergence-additional.json	BENCH-146–155 replace convergence failure scenarios with two formal-lint triggers (wave_count, total_layers) and eight untraced module discovery cases (signal aggregator, threshold detector, monitor, drift calculator, oscillation detector, stability scorer, wave analyzer, residual forecaster). All target c_to_r with 300-second timeout.
Additional Reverse-Flow, Integration & Config Benchmarks challenges/20-reverse-flow-additional.json, challenges/21-integration-additional.json, challenges/22-config-hooks-additional.json	BENCH-156–165 and BENCH-166–185 shift from rollback/integration failure mutations to creating untraced CommonJS modules for data export, batch processing, scheduling, rate limiting, crypto, metrics, health checking, API clients, validators, adapters, webhooks, connection pools, message queues, OAuth, file watching, caching, monitoring, and hook utilities—all c_to_r discovery with 300-second timeout.
Formal Models, Tests & Documentation Benchmarks challenges/23-formal-models-additional.json, challenges/24-tests-additional.json, challenges/25-documentation-additional.json	BENCH-186–190 shift from TLA+/Alloy modifications to formal-lint triggers and validator module discovery. BENCH-191–195 introduce orphan test files targeting t_to_r. BENCH-196–200 update markdown snippets with contradictory/outdated capability claims and nonexistent file references, all reduced to 300-second timeout.

Mutation Engine & Solver Integration

Layer / File(s)	Summary
Flexible File Mutation Strategy lib/mutator.cjs	file-modify logic is extended from append-only to support mutation.content (full replacement), mutation.append (append), regex find/replace, and fallback marker. Enables precise file state control for new benchmark patterns.
Solver Invocation & Isolation Setup lib/runner.cjs	runSolveFull now uses --fast mode with feature skips (--skip-heatmap, --skip-proximity, --no-auto-commit) instead of --no-timeout for faster detection-only benchmarks. createIsolatedRoot introduces COPY_DIRS set to support configurable deep-copy vs. symlink isolation strategy, replacing fixed .planning-only approach.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

Benchmark modules hop from bin to C→R,
Orphan tests and docs now travel far,
Mutations bend to content, append, and find—
Discovery engines, untraced but designed. 🐰✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Challenge redesigns: 33% → 56.5% baseline (reconciled with main)' accurately summarizes the main objective: redesigning challenges and reconciling work with the main branch while improving the baseline from 33% to 56.5%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ci/benchmark-regression-gate

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint skipped: no ESLint configuration detected in root package.json. To enable, add eslint to devDependencies.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

challenges/05-documentation.json (1)

236-239: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Align BENCH-060 timeout with the redesigned 300s benchmark standard.

Line 238 uses timeout_seconds: 600 while the redesigned suite is standardized to 300s. This introduces inconsistent runtime and can distort regression-gate behavior for this category.

Suggested fix

-      "timeout_seconds": 600
+      "timeout_seconds": 300

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@challenges/05-documentation.json` around lines 236 - 239, The BENCH-060
scoring block currently sets "timeout_seconds": 600 under the "scoring" object;
update this to the standardized 300s by changing "timeout_seconds" from 600 to
300 in the same "scoring" object (keep "method": "detection_only" unchanged) so
the BENCH-060 entry aligns with the redesigned 300s benchmark standard.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@challenges/24-tests-additional.json`:
- Line 12: The generated test string incorrectly destructures assert from
node:test; replace the import "const { test, assert } = require('node:test');"
with separate imports so test comes from node:test and assert from
node:assert/strict (e.g. import or require test and require assert from
'node:assert/strict'), and update every generated test content occurrence that
contains that bad import (the test string referencing
rf-nondeterministic-handler.cjs) so runtime no longer throws when running the
tests.

In `@lib/mutator.cjs`:
- Around line 200-205: The current branches in the mutation application use
truthiness (mutation.content, mutation.append, mutation.find) which prevents
explicit empty values like content: "" and allows no-op find/replace to pass;
update the logic to check for explicit field presence (e.g.,
Object.prototype.hasOwnProperty.call(mutation, 'content') and similarly for
'append', 'find', 'replace') and then apply the operation; for the find/replace
branch ensure mutation.find is a non-empty pattern (or valid regex) and that the
replacement actually changes the file (compute newContent = content.replace(new
RegExp(mutation.find, 'g'), mutation.replace) and if newContent === content or
mutation.find is empty/invalid, treat it as an error/failed mutation instead of
writing a no-op.

In `@lib/runner.cjs`:
- Around line 191-214: The current symlink logic (loop over entries using
fs.readdirSync, fs.symlinkSync) leaves writable paths porous because
runSolveFull and other code can write outside COPY_DIRS; change the strategy to
isolate writable files by default: instead of symlinking everything not in
COPY_DIRS, copy the entire projectRoot into tempDir (using fs.cpSync with
recursive) and then optionally create symlinks only for explicitly
immutable/heavy paths (or exclude known large paths like node_modules, .git) so
that COPY_DIRS semantics are inverted; reference COPY_DIRS, entries,
projectRoot, tempDir, runSolveFull, fs.symlinkSync and fs.cpSync when locating
the code to replace — implement either "copy-all-then-symlink-immutable" or
"symlink-only-immutable" approach and ensure any derived writable locations
(solver output dirs) are included in the copied set.

---

Outside diff comments:
In `@challenges/05-documentation.json`:
- Around line 236-239: The BENCH-060 scoring block currently sets
"timeout_seconds": 600 under the "scoring" object; update this to the
standardized 300s by changing "timeout_seconds" from 600 to 300 in the same
"scoring" object (keep "method": "detection_only" unchanged) so the BENCH-060
entry aligns with the redesigned 300s benchmark standard.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 76f79788-b28e-4f6b-a0b8-d9caa8ecea11

📥 Commits

Reviewing files that changed from the base of the PR and between b03d8de and 3329104.

📒 Files selected for processing (14)

• challenges/05-documentation.json
• challenges/06-cross-layer-alignment.json
• challenges/09-reverse-flow.json
• challenges/11-expert-challenges.json
• challenges/17-devops-deployment.json
• challenges/19-convergence-additional.json
• challenges/20-reverse-flow-additional.json
• challenges/21-integration-additional.json
• challenges/22-config-hooks-additional.json
• challenges/23-formal-models-additional.json
• challenges/24-tests-additional.json
• challenges/25-documentation-additional.json
• lib/mutator.cjs
• lib/runner.cjs

[coderabbitai]

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

In current Node.js documentation, does require('node:test')exportassert, or should assertions come from node:assert/node:assert/strict?

💡 Result:

The node:test module does not export assert [1][2]. Assertions should be imported from the built-in node:assert or node:assert/strict modules [3][4][5]. It is standard practice to use node:assert/strict when writing tests with the Node.js test runner to ensure strict equality comparisons and clearer error messages [3][6][7]. Example usage: import { test } from 'node:test'; import assert from 'node:assert/strict'; test('example test', => { assert.strictEqual(1, 1); });

Citations:

• 1: https://nodejs.org/dist/latest/docs/api/test.html
• 2: https://nodejs.org/api/test.html
• 3: https://nodejs.dev/en/learn/test-runner/using-test-runner
• 4: https://glebbahmutov.com/blog/trying-node-test-runner/
• 5: https://dev.to/armorbreak/how-i-test-my-nodejs-apps-a-practical-guide-2026-42i0
• 6: https://nearform.com/digital-community/demystifying-node-js-test-runner-assertions/
• 7: https://medium.com/@farhad.gulizada/node-js-assertion-testing-guide-d1fc5771a9e6

---

Fix invalid assertion import in generated test content.

The node:test module does not export assert. Destructuring it from node:test will cause a runtime error. Import assert from node:assert/strict instead.

Suggested fix (apply to lines 12, 33, 54, 75, 96)

- const { test, assert } = require('node:test');
+ const test = require('node:test');
+ const assert = require('node:assert/strict');

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	"content": "const { test, assert } = require('node:test');\nlet mod;\ntry { mod = require('./rf-nondeterministic-handler.cjs'); } catch (e) { mod = null; }\n\ntest('handler processes random input', () => {\n assert.ok(mod, 'module should load');\n});",
	const test = require('node:test');
	const assert = require('node:assert/strict');
	let mod;
	try { mod = require('./rf-nondeterministic-handler.cjs'); } catch (e) { mod = null; }
	test('handler processes random input', () => {
	assert.ok(mod, 'module should load');
	});

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@challenges/24-tests-additional.json` at line 12, The generated test string
incorrectly destructures assert from node:test; replace the import "const {
test, assert } = require('node:test');" with separate imports so test comes from
node:test and assert from node:assert/strict (e.g. import or require test and
require assert from 'node:assert/strict'), and update every generated test
content occurrence that contains that bad import (the test string referencing
rf-nondeterministic-handler.cjs) so runtime no longer throws when running the
tests.

[coderabbitai]

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Treat mutation fields as explicit directives, and reject no-op replacements.

Line 200 currently makes content: "" impossible because the branch is selected by truthiness, not field presence. Line 204 also accepts a missed find pattern as a successful mutation, which can silently turn a benchmark case into a no-op while still being scored as mutated.

💡 Suggested fix

-        if (mutation.content) {
+        if (Object.prototype.hasOwnProperty.call(mutation, 'content')) {
           fs.writeFileSync(fullPath, mutation.content);
-        } else if (mutation.append) {
+        } else if (Object.prototype.hasOwnProperty.call(mutation, 'append')) {
           fs.writeFileSync(fullPath, content + '\n' + mutation.append);
         } else if (mutation.find && mutation.replace !== undefined) {
-          fs.writeFileSync(fullPath, content.replace(new RegExp(mutation.find, 'g'), mutation.replace));
+          const next = content.replace(new RegExp(mutation.find, 'g'), mutation.replace);
+          if (next === content) {
+            throw new Error(`file-modify pattern did not match: ${mutation.find}`);
+          }
+          fs.writeFileSync(fullPath, next);
         } else {

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	if (mutation.content) {
	fs.writeFileSync(fullPath, mutation.content);
	} else if (mutation.append) {
	fs.writeFileSync(fullPath, content + '\n' + mutation.append);
	} else if (mutation.find && mutation.replace !== undefined) {
	fs.writeFileSync(fullPath, content.replace(new RegExp(mutation.find, 'g'), mutation.replace));
	if (Object.prototype.hasOwnProperty.call(mutation, 'content')) {
	fs.writeFileSync(fullPath, mutation.content);
	} else if (Object.prototype.hasOwnProperty.call(mutation, 'append')) {
	fs.writeFileSync(fullPath, content + '\n' + mutation.append);
	} else if (mutation.find && mutation.replace !== undefined) {
	const next = content.replace(new RegExp(mutation.find, 'g'), mutation.replace);
	if (next === content) {
	throw new Error(`file-modify pattern did not match: ${mutation.find}`);
	}
	fs.writeFileSync(fullPath, next);
	} else {

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/mutator.cjs` around lines 200 - 205, The current branches in the mutation
application use truthiness (mutation.content, mutation.append, mutation.find)
which prevents explicit empty values like content: "" and allows no-op
find/replace to pass; update the logic to check for explicit field presence
(e.g., Object.prototype.hasOwnProperty.call(mutation, 'content') and similarly
for 'append', 'find', 'replace') and then apply the operation; for the
find/replace branch ensure mutation.find is a non-empty pattern (or valid regex)
and that the replacement actually changes the file (compute newContent =
content.replace(new RegExp(mutation.find, 'g'), mutation.replace) and if
newContent === content or mutation.find is empty/invalid, treat it as an
error/failed mutation instead of writing a no-op.

[coderabbitai]

⚠️ Potential issue | 🔴 Critical | 🏗️ Heavy lift

The isolation is still porous for writes outside COPY_DIRS.

Line 200 symlinks every path not listed in COPY_DIRS, but this benchmark still runs all 230 challenges and runSolveFull can perform real writes. Any mutation or solver edit that lands outside this hard-coded set will still hit the source repo through the symlink, so parallel workers can continue to interfere with each other and with the checked-out tree.

The safe fix is to isolate all writable paths, not just a curated subset — e.g. copy the whole workspace except explicitly immutable/heavy paths, or derive the copy set from the challenge target plus known solver output locations before creating symlinks.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/runner.cjs` around lines 191 - 214, The current symlink logic (loop over
entries using fs.readdirSync, fs.symlinkSync) leaves writable paths porous
because runSolveFull and other code can write outside COPY_DIRS; change the
strategy to isolate writable files by default: instead of symlinking everything
not in COPY_DIRS, copy the entire projectRoot into tempDir (using fs.cpSync with
recursive) and then optionally create symlinks only for explicitly
immutable/heavy paths (or exclude known large paths like node_modules, .git) so
that COPY_DIRS semantics are inverted; reference COPY_DIRS, entries,
projectRoot, tempDir, runSolveFull, fs.symlinkSync and fs.cpSync when locating
the code to replace — implement either "copy-all-then-symlink-immutable" or
"symlink-only-immutable" approach and ensure any derived writable locations
(solver output dirs) are included in the copied set.