Skip to content

Comments

ci: auto-approve integration tests for internal PRs#98

Open
danielmeppiel wants to merge 4 commits intomainfrom
fix/auto-approve-internal-prs
Open

ci: auto-approve integration tests for internal PRs#98
danielmeppiel wants to merge 4 commits intomainfrom
fix/auto-approve-internal-prs

Conversation

@danielmeppiel
Copy link
Collaborator

@danielmeppiel danielmeppiel commented Feb 24, 2026

Problem

Every PR — including internal ones from repo collaborators — triggers the Integration Tests (PR) workflow which requires manual environment approval before running. This creates unnecessary friction:

  • Maintainers must navigate to Actions → find the waiting run → click "Review deployments" → approve
  • Each branch update triggers a new approval request (we just had 6 queued at once after rebasing open PRs)
  • Internal contributors already have write access, so the approval gate adds no security value for their PRs

Additionally, 4 integration test files hardcoded .venv/bin/apm as the binary path, bypassing the PR artifact binary that CI places on PATH. This meant integration tests were silently testing main's source code instead of the PR's binary, defeating the purpose of fork-safe artifact testing.

Changes

1. Auto-approve internal PRs (ci-integration.yml)

Split the approve job into two:

Job Trigger condition Approval
approve-fork PR from a fork (head_repository != github.repository) Manual environment gate (unchanged)
approve-internal PR from the same repo Auto-approved (no gate)

Downstream jobs (smoke-testintegration-testsrelease-validation) run if either approval job succeeds (the other is skipped).

2. Fix integration test binary resolution (test_ado_e2e.py, test_mixed_deps.py, test_skill_compile.py, test_skill_install.py)

Inverted binary lookup priority in 4 test files: prefer shutil.which("apm") (finds the PR binary on PATH in CI) over the hardcoded .venv/bin/apm (main's source install). Falls back to venv path for local development where PATH includes the venv.

Without this fix: integration tests for ADO, mixed deps, skills, and skill compilation run against main's source code, not the PR's artifact — regressions go undetected.

3. Upgrade CodeQL Action (codeql.yml)

Updated CodeQL Action from v3 to v4 (Node.js 16 deprecation).

Security model (unchanged for forks)

  • Fork PRs still require manual approval via the integration-tests environment protection rules
  • The workflow still checks out main (never PR code) for test scripts
  • Fork artifacts are still isolated — a maintainer reviews the fork's code before secrets are exposed to artifacts built from it
  • Internal PRs skip the gate because contributors already have write access to push directly to main if needed

Validation

Fork PRs still require manual environment approval to prevent
untrusted artifacts from accessing secrets. Internal PRs (same repo)
skip the gate since contributors already have write access.

This eliminates manual approval overhead for the common case while
preserving the security gate for external contributions.
Copilot AI review requested due to automatic review settings February 24, 2026 11:59
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements auto-approval for integration tests on internal PRs while maintaining the manual approval security gate for fork PRs. It addresses the workflow friction where repo collaborators with write access must manually approve environment protection rules for their own PRs, which provides no security benefit since they can already push directly to main.

Changes:

  • Split the single approve job into approve-fork (requires manual environment approval) and approve-internal (auto-approved)
  • Update downstream job dependencies to run if either approval path succeeds using always() conditional logic
  • Update workflow comments to reflect the dual approval path architecture

@danielmeppiel danielmeppiel added this to the 0.7.4 milestone Feb 24, 2026
The test harness in 4 integration test files hardcoded .venv/bin/apm
as the primary binary, falling back to PATH only when the venv didn't
exist. In CI (ci-integration.yml), the PR binary artifact is placed on
PATH via symlink, but uv sync also creates .venv/bin/apm from main's
source. The tests always found the venv binary first, effectively
testing main's code instead of the PR's.

Invert the priority: prefer shutil.which('apm') (finds PATH binary in
CI, or venv binary locally since venv activates onto PATH), falling
back to the hardcoded .venv path only when not on PATH.

Affected files:
- test_ado_e2e.py
- test_mixed_deps.py
- test_skill_compile.py
- test_skill_install.py
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Comment on lines +33 to 35
# Fallback to local dev venv
apm_path = Path(__file__).parent.parent.parent / ".venv" / "bin" / "apm"

Copy link

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing final fallback to "apm" when venv path doesn't exist. The other three test files modified in this PR (test_skill_install.py, test_skill_compile.py, test_mixed_deps.py) all include a final return "apm" statement after checking the venv path. This ensures the test can still attempt to find apm on the system PATH if neither the CI artifact nor the venv binary are available.

Without this fallback, if the venv doesn't exist and shutil.which("apm") returns None, the code will use a Path object instead of a string for apm_path, which could cause unexpected behavior when constructing the command string.

Suggested change
# Fallback to local dev venv
apm_path = Path(__file__).parent.parent.parent / ".venv" / "bin" / "apm"
# Fallback to local dev venv, then final fallback to "apm"
venv_apm = Path(__file__).parent.parent.parent / ".venv" / "bin" / "apm"
if venv_apm.exists():
apm_path = str(venv_apm)
else:
# If the venv binary is not available, still attempt to use "apm" on PATH
apm_path = "apm"

Copilot uses AI. Check for mistakes.
Copy link
Collaborator

@SebastienDegodez SebastienDegodez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants