test(policy): e2e integration tests for require_pinned_constraint (follow-up to #1494)#1505
Merged
Merged
Conversation
…#1494) Adds 6 CliRunner-based e2e tests that close the user-promise gaps left by #1494's existing unit + partial-e2e coverage: - Promise B: pinned dep (caret range, bare exact version) passes the policy gate under enforcement=block + require_pinned_constraint=true. - Promise C: policy_gate forwards direct_dep_keys to the runner -- regression trap for the transitive bleed-through guard added in the #1494 Copilot follow-up. - Promise D + G: block exits with code 1 and the diagnostic cites the offending dep ref, the pinning hint, and the check name inside the violation block (not just upstream resolver noise). - Promise E: backward compat -- require_pinned_constraint=false (the default) does NOT block an unbounded dep, even at enforcement=block. - Promise F: --dry-run previews a 'Would be blocked by policy' line citing the dep without aborting or mutating the filesystem. Each test was verified with a mutation-break gate: removing the production guard makes the corresponding test fail; restoring it passes (see PR body for the 5 mutations + evidence). Surfaces a separate observation worth filing: the '=1.2.3' alternate exact-version syntax is currently classified as BARE_BRANCH by _constraint_pinning.py (only bare '1.2.3' is recognized). The e2e uses the bare form per the documented contract. No production code changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot stopped reviewing on behalf of
danielmeppiel due to an error
May 27, 2026 11:48
danielmeppiel
added a commit
that referenced
this pull request
May 27, 2026
The constraint classifier in '_constraint_pinning.py' relied on
'is_semver_range' from 'apm_cli.deps.registry.semver' to recognise
valid semver ranges. That helper's '_RANGE_OPERATORS' tuple omitted
the '=' prefix, so any user who wrote the npm- and cargo-style
explicit-equality form ('=1.2.3') in 'apm.yml' got the constraint
mis-classified as BARE_BRANCH. Under 'policy.dependencies.require_
pinned_constraint: true', the install was blocked with a confusing
"bare branch '=1.2.3' tracks a moving tip" diagnostic.
Fix: teach both 'deps/registry/semver.py' (parse-time gate) and
'marketplace/semver.py' (runtime range matcher) to accept '=X.Y.Z'
as an exact pin. The classifier then flows through the existing
semver-range probe and returns None (pinned) for '=1.2.3',
'=1.2.3-beta.1', '=0.0.1', etc.
Scope decision:
- Accept: bare '1.2.3' and '=1.2.3' (npm / cargo precedent;
cargo treats '=1.2.3' as the stricter explicit pin).
- Reject: '==1.2.3' (pip-style is not part of node-semver; users
who write it get a clear violation pointing at the supported form
rather than silent acceptance of the wrong dialect).
Regression traps:
- tests/unit/policy: 5 parametrised cases plus a registry-source
case and a '==' rejection case.
- tests/unit/registry: '=1.2.3' / '=0.0.1' / '=1.2.3-beta.1'
added to the accepted-ranges parametrize; '==1.2.3' / '=garbage'
/ '=1.2' added to the rejection set.
- tests/unit/marketplace: 'satisfies_range' positive + prerelease
+ invalid-spec cases for the '=' operator.
- tests/integration/policy: existing 'test_bare_exact_version_does
_not_trigger_block' extended to include '=1.2.3' alongside
'1.2.3'; the documented '=1.2.3 is a known gap' caveat is
removed.
Mutation-break verified: deleting '=' from '_RANGE_OPERATORS'
fails the unit + e2e regression traps; deleting the '=' branch
in 'marketplace/semver.py' fails the satisfies_range trap.
Follow-up to #1505 (cannot fold; #1505 already merged).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
danielmeppiel
added a commit
that referenced
this pull request
May 27, 2026
…#1506) * fix(policy): classify '=1.2.3' explicit-equality as pinned constraint The constraint classifier in '_constraint_pinning.py' relied on 'is_semver_range' from 'apm_cli.deps.registry.semver' to recognise valid semver ranges. That helper's '_RANGE_OPERATORS' tuple omitted the '=' prefix, so any user who wrote the npm- and cargo-style explicit-equality form ('=1.2.3') in 'apm.yml' got the constraint mis-classified as BARE_BRANCH. Under 'policy.dependencies.require_ pinned_constraint: true', the install was blocked with a confusing "bare branch '=1.2.3' tracks a moving tip" diagnostic. Fix: teach both 'deps/registry/semver.py' (parse-time gate) and 'marketplace/semver.py' (runtime range matcher) to accept '=X.Y.Z' as an exact pin. The classifier then flows through the existing semver-range probe and returns None (pinned) for '=1.2.3', '=1.2.3-beta.1', '=0.0.1', etc. Scope decision: - Accept: bare '1.2.3' and '=1.2.3' (npm / cargo precedent; cargo treats '=1.2.3' as the stricter explicit pin). - Reject: '==1.2.3' (pip-style is not part of node-semver; users who write it get a clear violation pointing at the supported form rather than silent acceptance of the wrong dialect). Regression traps: - tests/unit/policy: 5 parametrised cases plus a registry-source case and a '==' rejection case. - tests/unit/registry: '=1.2.3' / '=0.0.1' / '=1.2.3-beta.1' added to the accepted-ranges parametrize; '==1.2.3' / '=garbage' / '=1.2' added to the rejection set. - tests/unit/marketplace: 'satisfies_range' positive + prerelease + invalid-spec cases for the '=' operator. - tests/integration/policy: existing 'test_bare_exact_version_does _not_trigger_block' extended to include '=1.2.3' alongside '1.2.3'; the documented '=1.2.3 is a known gap' caveat is removed. Mutation-break verified: deleting '=' from '_RANGE_OPERATORS' fails the unit + e2e regression traps; deleting the '=' branch in 'marketplace/semver.py' fails the satisfies_range trap. Follow-up to #1505 (cannot fold; #1505 already merged). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: document =1.2.3 explicit-equality pin form Fold panel-recommended follow-ups into the same PR: - reference/policy-schema.md: add =1.5.3 OK example and ==1.5.3 FAIL example - consumer/manage-dependencies.md: add registry semver constraint table with explicit note that pip-style == is unsupported - apm-usage/governance.md: name =1.2.3 alongside bare 1.2.3 in the pinned-constraint remediation column - CHANGELOG.md: normalise spelling (recognised -> recognized) for consistency with surrounding entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #1494 that closes the user-promise gaps in test coverage
left by the original PR. #1494 shipped 65 unit tests plus a partial
e2e file (
tests/integration/test_policy_pinned_constraint_e2e.pycovering only the block-aborts and warn-emits paths). This PR adds the
remaining CliRunner-based end-to-end coverage for every user-observable
promise the policy field makes.
No production code changes.
Promises covered
TestPromiseBPinnedDepPassesPolicyGate(x2)policy_gateforwardsdirect_dep_keys(transitive bleed-through guard from #1494 Copilot follow-up)TestPromiseCDirectDepKeysWiringTestPromiseDGBlockDiagnosticAndExitCoderequire_pinned_constraint: false(default) does not blockTestPromiseERequirePinnedFalseDoesNotBlock--dry-runpreviews "Would be blocked" without raising or mutating diskTestPromiseFDryRunPreviewsViolation6 tests added across 5 test classes. New directory
tests/integration/policy/(mirrors the existing
tests/integration/marketplace/convention). Twofixtures under
tests/fixtures/policy/require_pinned/(block + off variants).Mutation-break evidence
Each test was certified by removing the production guard, confirming
the test fails, and restoring. Per the test-coverage-expert north star:
"If this code silently drifts six months from now, will any test
fail loudly enough that a maintainer will see it before a user does?"
_check_pinned_constraints:if reason is None: continue-> force violationdirect_dep_keys=...dropped from all 3 forwarding sites (policy_gate,policy_target_check,install_preflight)violations.append(f"{key}: {hint}")->"MUTATION-NO-REF"if not policy.require_pinned_constraint: return ...early-exit removedif enforcement == "block" and not dry_run:->if enforcement == "block":All mutations restored; final test run is green.
Lint evidence
Test evidence
The 1 xfailed is the pre-existing
#1488sentinel, unrelated to this PR.Observation worth filing separately
While writing Promise B, I noticed that the
=1.2.3alternateexact-version syntax is classified by
_constraint_pinning.pyasBARE_BRANCH(rendered as "bare branch '=1.2.3' tracks a movingtip"), even though the #1494 commit message lists exact versions among
the pinned forms. Only the bare
1.2.3form is recognized as pinned.This is a real UX gap but is out of scope for a test-only PR. The new
e2e uses the bare form and adds an in-file comment pointing at the
gap. A follow-up issue would be appropriate.
How to test