hypatia: detect stale continue-on-error masks tied to closed issues (from #333 cohort, pattern 5)

[hyperpolymath]

Detector spec (from hypatia#333 Pattern 5)

Pattern 5 — continue-on-error: true on a job, paired with documentation that says "remove this when X lands"

Severity: medium (gate-quality erosion; the workflow drifts further from green main-line)

Detection (Hypatia.Rules.WorkflowAudit):

• For each job with continue-on-error: true:
  • Scan the job's leading comments for phrases like until #N lands, until #N closes, remove when #N.
  • If a referenced issue is now closed or merged (via GitHub API), flag as "stale continue-on-error mask".
• Bonus: detect continue-on-error on a job whose name matches an entry in branch-protection's required_status_checks.contexts (structural contradiction).

Worked example (this session):

• affinescript/.github/workflows/ci.yml vscode-smoke had continue-on-error: true with the comment "Remove the continue-on-error: true line when chore(deps): bump bb8 from 0.8.6 to 0.9.1 #104 publishes the adapter to npm." chore(deps): bump bb8 from 0.8.6 to 0.9.1 #104 is still open so this didn't trigger as stale, but the related affinescript#380 fixed the root cause and removed the mask anyway.

Remediation guidance to emit:

If the gating issue has closed, drop continue-on-error: true and let the job report accurately. If the issue is still open but the workflow logic has changed (e.g. root cause fixed differently), also drop and update the comment.

Implementation pointers

• Detection algorithm: For each continue-on-error: true job, scan leading comments for until #N lands/until #N closes/remove when #N; resolve #N via GitHub API; flag if closed/merged. Bonus: contradiction check against branch-protection required contexts.
• Real-world example: affinescript/.github/workflows/ci.yml vscode-smoke had continue-on-error: true with comment "Remove the continue-on-error: true line when chore(deps): bump bb8 from 0.8.6 to 0.9.1 #104 publishes the adapter to npm."
• Landed fix (reference): affinescript#380 (root cause fixed; mask removed even though chore(deps): bump bb8 from 0.8.6 to 0.9.1 #104 still open).
• Rule statement: If the gating issue has closed, drop continue-on-error: true and let the job report accurately. If the issue is still open but the workflow logic has changed (e.g. root cause fixed differently), also drop and update the comment.

Acceptance

• Rule encoded in hypatia (file path follows existing rule naming convention — lib/rules/<name>.ex if Elixir, or matching the repo's rule DSL)
• Test fixture exercising the positive case + at least one negative case
• Smoke test passes against the cited landed-fix repo

Source cohort: hypatia#333.

[hyperpolymath]

cicd triage 2026-05-30 — scoped plan + park

PR #403 ships the first batch of 4 rules from this cohort (#336, #337, #338, #390). This issue is in the next batch; landing it requires more substantial work than the first four:

Detection scans for continue-on-error: true jobs, reads leading comments for until #N / remove when #N / etc. patterns, then gh API closed-issue check. Add to lib/rules/workflow_audit.ex as WF023.

The issue body is the implementation spec (detection algorithm + worked example + remediation guidance — well-formed). Next-batch PR will pick this up. Leaving open with cicd label.

[hyperpolymath]

Detector landed in PR #411 (merged, d460d6c): Hypatia.Rules.WorkflowAudit.check_stale_continue_on_error/2 (WF023) — flags a continue-on-error: true job whose "until/remove when #N" comment names an issue in the injected closed-issue set.

• Rule encoded (WF023)
• Test fixtures (positive + open-issue + no-coe negatives) — test/rules/group_b_detectors_test.exs
• Smoke test against cited landed-fix repo (needs live issue state)

Remaining (keeping this open): it's a standalone check_* taking closed_issues as a parameter (deliberately not wired into audit/3, which has no issue-state signal). Wiring the live issue-state lookup into the scan flow + the smoke test are the follow-up. The "bonus" branch-protection-contradiction check is also still open.

---

Generated by Claude Code

[hyperpolymath]

Closed by main's WF023 (check_stale_continue_on_error) which lands the detector this issue requested. The rule's docstring header explicitly cites this issue. The cohort-superseded variant in hypatia#409 (PR's WF024) was redundant; #409 has been closed as partially superseded.