Skip to content

ci: drop continue-on-error on test-run step (bisect_ppx)#409

Merged
hyperpolymath merged 1 commit into
mainfrom
ci/drop-soft-fail-on-test-step
May 28, 2026
Merged

ci: drop continue-on-error on test-run step (bisect_ppx)#409
hyperpolymath merged 1 commit into
mainfrom
ci/drop-soft-fail-on-test-step

Conversation

@hyperpolymath
Copy link
Copy Markdown
Owner

Summary

The "Run tests with bisect_ppx instrumentation" step in .github/workflows/ci.yml carried continue-on-error: true. Result: test failures (including QTT typechecker test regressions) were being reported as CI passes.

What changes

Removes one line: continue-on-error: true from the test-run step at line 178.

What is NOT changed

Intentional soft-fail patterns preserved on non-test steps:

  • Lint with odoc (line 105) — docs lint.
  • Build bench targets + Run benches (lines 127, 131) — benchmarks.
  • Generate HTML coverage report (line 183) — coverage tooling artefact (failure here shouldn't mask test results).

Pattern context

Same anti-pattern class as:

  • echo-types' N5Falsifier xfail gate (resolved via echo-types#136).
  • echidna's Tier-2/3 live-test soft-fail (resolved via echidna#118).

The principle: "CI green" must reliably mean "tests pass". Soft-failing test steps breaks that invariant.

Refs

Estate prover-CI hygiene audit, recorded at:
`standards/adoption-readiness-grades/AUDIT-FINDINGS-2026-05-28.adoc`.

Test plan

  • CI runs on this PR (hard-fails on real test regressions instead of silently masking).

🤖 Generated with Claude Code

The "Run tests with bisect_ppx instrumentation" step in
`.github/workflows/ci.yml` carried `continue-on-error: true`. Result:
test failures (including QTT typechecker test regressions) reported
as a CI pass.

The intention was apparently to keep coverage-report generation
working even when tests fail, but the better split is:
- TESTS must hard-fail on regression (this commit).
- Coverage-report generation may soft-fail (kept as `continue-on-error`
  on the "Generate HTML coverage report" step, since coverage tooling
  artefact issues should not mask test results).

This is the same anti-pattern class as echo-types' N5Falsifier xfail
gate (resolved via echo-types#136) and echidna's Tier-2/3 live-test
soft-fail (resolved via echidna#118).

The other 4 `continue-on-error` instances in this file are preserved:
- `Lint with odoc` (line 105) — docs lint, not proof validity.
- `Build bench targets` + `Run benches` (lines 127, 131) — benchmarks.
- `Generate HTML coverage report` (line 183) — coverage tooling
  artefact, not proof validity.

Refs hygiene audit 2026-05-28
(`standards/adoption-readiness-grades/AUDIT-FINDINGS-2026-05-28.adoc`).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@hyperpolymath hyperpolymath merged commit d39e9b1 into main May 28, 2026
14 of 25 checks passed
@hyperpolymath hyperpolymath deleted the ci/drop-soft-fail-on-test-step branch May 28, 2026 08:40
@github-actions
Copy link
Copy Markdown

🔍 Hypatia Security Scan

Findings: 82 issues detected

Severity Count
🔴 Critical 4
🟠 High 11
🟡 Medium 67

⚠️ Action Required: Critical security issues found!

View findings
[
  {
    "reason": "Action actions/checkout@v6 needs attention",
    "type": "unpinned_action",
    "file": "publish-jsr.yml",
    "action": "pin_sha",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Action denoland/setup-deno@v2 needs attention",
    "type": "unpinned_action",
    "file": "publish-jsr.yml",
    "action": "pin_sha",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in affine-vscode-publish.yml",
    "type": "unknown",
    "file": "affine-vscode-publish.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in casket-pages.yml",
    "type": "unknown",
    "file": "casket-pages.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in casket-pages.yml",
    "type": "unknown",
    "file": "casket-pages.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in ci.yml",
    "type": "unknown",
    "file": "ci.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in ci.yml",
    "type": "unknown",
    "file": "ci.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in ci.yml",
    "type": "unknown",
    "file": "ci.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in ci.yml",
    "type": "unknown",
    "file": "ci.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Issue in ci.yml",
    "type": "unknown",
    "file": "ci.yml",
    "action": "flag",
    "rule_module": "workflow_audit",
    "severity": "medium"
  }
]

Powered by Hypatia Neurosymbolic CI/CD Intelligence

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant