Skip to content

test(policy): e2e integration tests for require_pinned_constraint (follow-up to #1494)#1505

Merged
danielmeppiel merged 1 commit into
mainfrom
test/policy-require-pinned-e2e
May 27, 2026
Merged

test(policy): e2e integration tests for require_pinned_constraint (follow-up to #1494)#1505
danielmeppiel merged 1 commit into
mainfrom
test/policy-require-pinned-e2e

Conversation

@danielmeppiel
Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #1494 that closes the user-promise gaps in test coverage
left by the original PR. #1494 shipped 65 unit tests plus a partial
e2e file (tests/integration/test_policy_pinned_constraint_e2e.py
covering only the block-aborts and warn-emits paths). This PR adds the
remaining CliRunner-based end-to-end coverage for every user-observable
promise the policy field makes.

No production code changes.

Promises covered

Promise Test Tier
B -- pinned dep proceeds past the gate (caret, bare exact) TestPromiseBPinnedDepPassesPolicyGate (x2) e2e
C -- policy_gate forwards direct_dep_keys (transitive bleed-through guard from #1494 Copilot follow-up) TestPromiseCDirectDepKeysWiring e2e (wiring)
D + G -- block exits 1 and diagnostic cites dep + pinning hint + check name inside violation block TestPromiseDGBlockDiagnosticAndExitCode e2e
E -- backward compat: require_pinned_constraint: false (default) does not block TestPromiseERequirePinnedFalseDoesNotBlock e2e
F -- --dry-run previews "Would be blocked" without raising or mutating disk TestPromiseFDryRunPreviewsViolation e2e

6 tests added across 5 test classes. New directory tests/integration/policy/
(mirrors the existing tests/integration/marketplace/ convention). Two
fixtures under tests/fixtures/policy/require_pinned/ (block + off variants).

Mutation-break evidence

Each test was certified by removing the production guard, confirming
the test fails, and restoring. Per the test-coverage-expert north star:
"If this code silently drifts six months from now, will any test
fail loudly enough that a maintainer will see it before a user does?"

Test Production guard mutated Result with mutation
Promise B _check_pinned_constraints: if reason is None: continue -> force violation 2 failed (both Promise B tests)
Promise C direct_dep_keys=... dropped from all 3 forwarding sites (policy_gate, policy_target_check, install_preflight) 1 failed
Promise D+G violations.append(f"{key}: {hint}") -> "MUTATION-NO-REF" 1 failed
Promise E if not policy.require_pinned_constraint: return ... early-exit removed 1 failed
Promise F if enforcement == "block" and not dry_run: -> if enforcement == "block": 1 failed

All mutations restored; final test run is green.

Lint evidence

$ uv run --extra dev ruff check src/ tests/
All checks passed!

$ uv run --extra dev ruff format --check src/ tests/
1107 files already formatted

$ uv run --extra dev python -m pylint --disable=all --enable=R0801 \
    --min-similarity-lines=10 --fail-on=R0801 src/apm_cli/
Your code has been rated at 10.00/10

$ bash scripts/lint-auth-signals.sh
[+] auth-signal lint clean

Test evidence

$ uv run --extra dev pytest tests/integration/policy tests/unit/policy \
    tests/integration/test_policy_pinned_constraint_e2e.py -q
813 passed, 1 xfailed, 17 subtests passed in 2.81s

The 1 xfailed is the pre-existing #1488 sentinel, unrelated to this PR.

Observation worth filing separately

While writing Promise B, I noticed that the =1.2.3 alternate
exact-version syntax is classified by _constraint_pinning.py as
BARE_BRANCH (rendered as "bare branch '=1.2.3' tracks a moving
tip"
), even though the #1494 commit message lists exact versions among
the pinned forms. Only the bare 1.2.3 form is recognized as pinned.

This is a real UX gap but is out of scope for a test-only PR. The new
e2e uses the bare form and adds an in-file comment pointing at the
gap. A follow-up issue would be appropriate.

How to test

git fetch origin test/policy-require-pinned-e2e
git checkout test/policy-require-pinned-e2e
uv run --extra dev pytest tests/integration/policy -v

…#1494)

Adds 6 CliRunner-based e2e tests that close the user-promise gaps left
by #1494's existing unit + partial-e2e coverage:

- Promise B: pinned dep (caret range, bare exact version) passes the
  policy gate under enforcement=block + require_pinned_constraint=true.
- Promise C: policy_gate forwards direct_dep_keys to the runner --
  regression trap for the transitive bleed-through guard added in the
  #1494 Copilot follow-up.
- Promise D + G: block exits with code 1 and the diagnostic cites the
  offending dep ref, the pinning hint, and the check name inside the
  violation block (not just upstream resolver noise).
- Promise E: backward compat -- require_pinned_constraint=false (the
  default) does NOT block an unbounded dep, even at enforcement=block.
- Promise F: --dry-run previews a 'Would be blocked by policy' line
  citing the dep without aborting or mutating the filesystem.

Each test was verified with a mutation-break gate: removing the
production guard makes the corresponding test fail; restoring it
passes (see PR body for the 5 mutations + evidence).

Surfaces a separate observation worth filing: the '=1.2.3' alternate
exact-version syntax is currently classified as BARE_BRANCH by
_constraint_pinning.py (only bare '1.2.3' is recognized). The
e2e uses the bare form per the documented contract.

No production code changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 27, 2026 11:46
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@danielmeppiel danielmeppiel merged commit 196712e into main May 27, 2026
20 of 21 checks passed
@danielmeppiel danielmeppiel deleted the test/policy-require-pinned-e2e branch May 27, 2026 12:12
@danielmeppiel danielmeppiel restored the test/policy-require-pinned-e2e branch May 27, 2026 12:17
danielmeppiel added a commit that referenced this pull request May 27, 2026
The constraint classifier in '_constraint_pinning.py' relied on
'is_semver_range' from 'apm_cli.deps.registry.semver' to recognise
valid semver ranges. That helper's '_RANGE_OPERATORS' tuple omitted
the '=' prefix, so any user who wrote the npm- and cargo-style
explicit-equality form ('=1.2.3') in 'apm.yml' got the constraint
mis-classified as BARE_BRANCH. Under 'policy.dependencies.require_
pinned_constraint: true', the install was blocked with a confusing
"bare branch '=1.2.3' tracks a moving tip" diagnostic.

Fix: teach both 'deps/registry/semver.py' (parse-time gate) and
'marketplace/semver.py' (runtime range matcher) to accept '=X.Y.Z'
as an exact pin. The classifier then flows through the existing
semver-range probe and returns None (pinned) for '=1.2.3',
'=1.2.3-beta.1', '=0.0.1', etc.

Scope decision:
- Accept: bare '1.2.3' and '=1.2.3' (npm / cargo precedent;
  cargo treats '=1.2.3' as the stricter explicit pin).
- Reject: '==1.2.3' (pip-style is not part of node-semver; users
  who write it get a clear violation pointing at the supported form
  rather than silent acceptance of the wrong dialect).

Regression traps:
- tests/unit/policy: 5 parametrised cases plus a registry-source
  case and a '==' rejection case.
- tests/unit/registry: '=1.2.3' / '=0.0.1' / '=1.2.3-beta.1'
  added to the accepted-ranges parametrize; '==1.2.3' / '=garbage'
  / '=1.2' added to the rejection set.
- tests/unit/marketplace: 'satisfies_range' positive + prerelease
  + invalid-spec cases for the '=' operator.
- tests/integration/policy: existing 'test_bare_exact_version_does
  _not_trigger_block' extended to include '=1.2.3' alongside
  '1.2.3'; the documented '=1.2.3 is a known gap' caveat is
  removed.

Mutation-break verified: deleting '=' from '_RANGE_OPERATORS'
fails the unit + e2e regression traps; deleting the '=' branch
in 'marketplace/semver.py' fails the satisfies_range trap.

Follow-up to #1505 (cannot fold; #1505 already merged).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
danielmeppiel added a commit that referenced this pull request May 27, 2026
…#1506)

* fix(policy): classify '=1.2.3' explicit-equality as pinned constraint

The constraint classifier in '_constraint_pinning.py' relied on
'is_semver_range' from 'apm_cli.deps.registry.semver' to recognise
valid semver ranges. That helper's '_RANGE_OPERATORS' tuple omitted
the '=' prefix, so any user who wrote the npm- and cargo-style
explicit-equality form ('=1.2.3') in 'apm.yml' got the constraint
mis-classified as BARE_BRANCH. Under 'policy.dependencies.require_
pinned_constraint: true', the install was blocked with a confusing
"bare branch '=1.2.3' tracks a moving tip" diagnostic.

Fix: teach both 'deps/registry/semver.py' (parse-time gate) and
'marketplace/semver.py' (runtime range matcher) to accept '=X.Y.Z'
as an exact pin. The classifier then flows through the existing
semver-range probe and returns None (pinned) for '=1.2.3',
'=1.2.3-beta.1', '=0.0.1', etc.

Scope decision:
- Accept: bare '1.2.3' and '=1.2.3' (npm / cargo precedent;
  cargo treats '=1.2.3' as the stricter explicit pin).
- Reject: '==1.2.3' (pip-style is not part of node-semver; users
  who write it get a clear violation pointing at the supported form
  rather than silent acceptance of the wrong dialect).

Regression traps:
- tests/unit/policy: 5 parametrised cases plus a registry-source
  case and a '==' rejection case.
- tests/unit/registry: '=1.2.3' / '=0.0.1' / '=1.2.3-beta.1'
  added to the accepted-ranges parametrize; '==1.2.3' / '=garbage'
  / '=1.2' added to the rejection set.
- tests/unit/marketplace: 'satisfies_range' positive + prerelease
  + invalid-spec cases for the '=' operator.
- tests/integration/policy: existing 'test_bare_exact_version_does
  _not_trigger_block' extended to include '=1.2.3' alongside
  '1.2.3'; the documented '=1.2.3 is a known gap' caveat is
  removed.

Mutation-break verified: deleting '=' from '_RANGE_OPERATORS'
fails the unit + e2e regression traps; deleting the '=' branch
in 'marketplace/semver.py' fails the satisfies_range trap.

Follow-up to #1505 (cannot fold; #1505 already merged).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: document =1.2.3 explicit-equality pin form

Fold panel-recommended follow-ups into the same PR:
- reference/policy-schema.md: add =1.5.3 OK example and ==1.5.3 FAIL example
- consumer/manage-dependencies.md: add registry semver constraint table
  with explicit note that pip-style == is unsupported
- apm-usage/governance.md: name =1.2.3 alongside bare 1.2.3 in the
  pinned-constraint remediation column
- CHANGELOG.md: normalise spelling (recognised -> recognized) for
  consistency with surrounding entries

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@danielmeppiel danielmeppiel mentioned this pull request May 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants