Redesign 59 structurally-broken challenges + fix runner repeatability

[jobordu]

Summary

Stacked on #2. Fixes the structurally-broken challenges found while triaging the
benchmark toward an honest 100%, and a runner bug that made file-create
challenges non-reproducible.

The problem

Triage of all 230 challenges found ~59 that no solver could ever pass:

• Phantom-file challenges (12–18, 26): file-modify on src/database.js,
  src/components/*.js, Dockerfile, etc. — files that don't exist in the target
  project (QGSD, a planning/formal-methods CLI). file-modify on a missing file
  is a silent no-op → no residual change → unmeasurable.
• Dead-layer challenges (03, 04, 07): only target t_to_c/f_to_c,
  which the benchmark's --fast invocation deliberately skips → residual is
  always -1 → undetectable.

Changes

59 challenges redesigned to the verified c_to_r file-create pattern
(BENCH-188): create an untraced bin/rf-*.cjs utility with no @req
annotation, which the code→requirements layer flags as a coverage gap. Each new
file has a unique name, distinct realistic JS content, and honest title/tags.
Verified: BENCH-111 scores PASS (c_to_r residual 124→125, mutation detected).

Runner repeatability fix (lib/runner.cjs): the snapshot model runs solve
in place, but restoreSnapshot only cleaned .planning/formal — so the 125
file-create challenges (116 into bin/) left their files behind. On a repeat
run the file already exists, the residual doesn't increase, and the challenge
falsely fails. createSnapshot now records pre-existing files in the
create-guard dirs; restoreSnapshot deletes anything a challenge added.
.jsonl is also snapshotted so the solver's trend log is restored — runs now
leave the project pristine.

Verification

• All 230 challenges schema-valid (npm run validate).
• All 125 file-create target_files unique.
• BENCH-111 PASS, twice, leaving QGSD with 0 dirty files.

A full re-measurement (multi-hour serial run) is the follow-up step to get the
new baseline score.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 53595417-e273-4c92-8840-1fcaa96fdf4a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch benchmark/redesign-broken-challenges

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}