Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions .github/workflows/e2e.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: E2E (self-dogfood)

on:
issue_comment:
types: [created, edited]
discussion_comment:
types: [created, edited]
pull_request:
types: [opened, synchronize, reopened]
push:
branches:
- main

permissions:
contents: read

concurrency:
group: e2e-${{ github.event.comment.id || github.ref }}
cancel-in-progress: false

jobs:
registered:
if: github.event_name == 'pull_request' || github.event_name == 'push'
runs-on: ubuntu-latest
steps:
- run: echo "E2E workflow is registered. Real dogfood runs only on issue_comment / discussion_comment."

dogfood:
if: |
(github.event_name == 'issue_comment' || github.event_name == 'discussion_comment') &&
startsWith(github.event.comment.body, '.')
runs-on: ubuntu-latest
environment: e2e
permissions:
contents: read
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- uses: ./.github/actions/setup

- run: pnpm build

- id: command
uses: ./
continue-on-error: true
with:
command: test
allowed_contexts: issue,pull_request,discussion

- name: Verify
env:
ACTUAL_OUTCOME: ${{ steps.command.outcome }}
ACTUAL_CONTINUE: ${{ steps.command.outputs.continue }}
ACTUAL_COMMAND: ${{ steps.command.outputs.command }}
ACTUAL_PARAMS: ${{ steps.command.outputs.params }}
ACTUAL_NUMBER: ${{ steps.command.outputs.number }}
ACTUAL_CONTEXT: ${{ steps.command.outputs.context }}
ACTUAL_ISSUE_NUMBER: ${{ steps.command.outputs.issue_number }}
ACTUAL_COMMENT_ID: ${{ steps.command.outputs.comment_id }}
ACTUAL_ACTOR: ${{ steps.command.outputs.actor }}
run: pnpm tsx scripts/e2e-verify.ts
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,10 @@ $ mise install
$ pnpm i
```

### E2E (self-dogfood)

The action is validated against real GitHub event payloads via an environment-gated workflow. See [`docs/e2e.md`](./docs/e2e.md) for the comment convention, security model, and required repo settings.

## LICENSE

See [LICENSE][license].
Expand Down
203 changes: 203 additions & 0 deletions docs/e2e.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
# E2E (self-dogfood) workflow

The workflow at `.github/workflows/e2e.yaml` exercises the bundled `dist/index.js`
against real GitHub event payloads from this repository. It runs the action under
test against `issue_comment` (on issues and on PRs) and `discussion_comment`
events, and asserts the emitted outputs against an expectation directive embedded
in the triggering comment.

The matrix of `vitest` integration tests in `src/main.integration.test.ts` provides
fast, exhaustive coverage. This workflow complements that suite by validating the
real runtime path (`runs.using: node24` + payload shape) before a release ships.

## Comment convention

The directive is carried as a single-quoted JSON string under the reserved
`_expect_` key in the command's params, so the action's own parser sees it as a
regular params value:

```
.test foo=bar, _expect_='{"continue":"true","command":"test","params":{"foo":"bar"},"context":"issue"}'
```

The action parses this comment as `command=test, params={foo:"bar", _expect_:"<JSON>"}`
and emits the full `params` (including `_expect_`) as the `params` output. The
verification step then extracts `params._expect_`, JSON-decodes it, and compares
against the other observable outputs.

### Why not `# expect: ...` on a separate line?

The action's parser is strict: anything in the comment body after `.command
key=value, ...` that isn't a comma-separated key=value pair (or end-of-input)
causes a parse error and `core.setFailed`. A trailing `# expect:` line breaks
the parser before verification can run. Putting the directive inside a params
value is the only way to keep the directive in the same comment without
modifying the production parser.

### Rules

- The directive value is **a JSON string**, not raw JSON. Quote it with single
quotes so the embedded double-quoted JSON does not collide with the param's
delimiter.
- If the comment has no `_expect_` key (or the action did not emit `params` at
all), the verification step falls back to a smoke check. The smoke check
asserts that the action emitted a `continue` output and that its step outcome
is not `failure`.
- Keys not listed in the directive are not checked.
- Unknown directive keys are reported as a mismatch (typo guard). The allowed
keys are `failed`, `params`, `continue`, `command`, `number`, `context`,
`issue_number`, `comment_id`, and `actor`. The internal `outcome` is asserted
via `failed` rather than directly.
- The `failed` value must be a JSON boolean (`true` or `false`). A string
`"true"` is reported as a mismatch (typo guard).
- A directive value of `null` means **must be unset** — the corresponding
output must be empty. For example, `"issue_number": null` asserts that the
action did not emit `issue_number` (regression guard for the soft-deprecation
contract on `discussion_comment` events).
- The `_expect_` key is stripped from `actual.params` before comparison, so
directive-bearing comments do not need to enumerate it under
`expected.params`.
- Numeric JSON values are coerced via `String(...)` before comparison
(GitHub Actions outputs are always strings).

### Examples

Positive case (action must succeed with the listed params):

```
.test name=alice, _expect_='{"continue":"true","command":"test","params":{"name":"alice"},"context":"issue"}'
```

Smoke run (no directive — only verifies the action ran end-to-end):

```
.test foo=bar
```

### Limitations of the directive carrier

The directive can only be carried on comments that produce `continue=true`,
because the action only emits the `params` output on that path
(`src/main.ts:110-112`). For negative cases — filtered `allowed_contexts`,
command not in the allow-list, malformed params — the directive cannot reach
the verification step. Use the smoke run convention for those, and confirm the
filter behavior manually in the workflow log (Verify step shows
`actual.continue` and `actual.outcome`).

## Security model

The primary security boundary is the GitHub `Environment` named `e2e`, configured
with required reviewers. Every workflow run pauses until a reviewer explicitly
approves it.

- Workflow source guarantee: GitHub Actions runs `issue_comment` and
`discussion_comment` workflows from the repository default branch. A fork PR
cannot mutate the e2e workflow file or scripts in a way that affects the run.
- Token scope: the workflow runs with the default `GITHUB_TOKEN` and an explicit
`permissions: contents: read`. No personal access tokens or app installations
are used.
- No secret usage in the action under test: the action only parses public
comment text. A leaked approval has limited blast radius — the worst case is
a CI minute burned on parsing a comment.

### Approval-then-edit race (known caveat)

A comment author can edit the comment body between the moment a reviewer
approves the `e2e` environment and the moment the workflow runs. The workflow
re-reads `github.event.comment.body` at run start (via the action's own payload
access), so an edited directive will be evaluated, not the body the reviewer
saw.

Mitigation is operational rather than code-side:

- Reviewers should screenshot or paste the comment body at approval time and
compare it with the workflow's Verify step output.
- The risk is bounded because the e2e job has no write permissions and the
action under test parses only public comment text.

A code-side mitigation (re-reading the comment body via `gh api` after
approval) is intentionally out of scope; it would require additional
`pull-requests: write` or `issues: write` scope that contradicts the
minimal-permissions design.

## Required repo settings

These are one-time admin tasks. Without them, the workflow will fail to start.

1. **Enable Discussions** on `knowledge-work/command-action`.
Settings → General → Features → check `Discussions`.

2. **Create the `e2e` Environment**.
Settings → Environments → `New environment` → name `e2e` → enable
`Required reviewers` and add the `knowledge-work` org members who maintain
this repo as required reviewers.

The list of reviewers should match the org members who already approve releases.

## Operational tradeoffs

By design, the `dogfood` job only runs when the comment body starts with a `.`
character (`startsWith(github.event.comment.body, '.')`). This filters out chat
or review comments so that the reviewer is not asked to approve the `e2e`
environment every time someone posts an unrelated comment.

The alternative (running on every comment) was considered and rejected because
it produces approval noise that drowns out real dogfood events.

If org policy or operational preferences change, this trade-off can be revisited
by editing the `dogfood.if` expression in `.github/workflows/e2e.yaml`.

## Smoke run

After Discussions and the `e2e` environment are configured, exercise each event
kind once with a passing directive and once with a deliberately wrong directive.

### Positive runs

1. **Issue comment** — open or use an existing issue and post:

```
.test foo=bar, _expect_='{"continue":"true","command":"test","params":{"foo":"bar"},"context":"issue"}'
```

Approve the `e2e` environment when prompted; confirm the workflow run passes.

2. **PR comment** — open or use an existing PR and post the same comment, but
change `"context":"issue"` to `"context":"pull_request"`. Approve and verify.

3. **Discussion comment** — open or use a discussion and post the same comment,
but with `"context":"discussion"` and add `"issue_number":null` (the soft-
deprecation regression guard — `issue_number` must be unset on discussions).

### Negative runs

For each of the three event kinds, post a comment with a deliberately wrong
directive (for example, change one of the asserted values in the directive).
The workflow must fail with a clear `E2E mismatch:` message naming the failing
field.

### What "complete" looks like

- Six total dogfood runs (3 event kinds × positive + negative).
- All three positive runs end with `E2E expectations matched.` in the
Verify step.
- All three negative runs fail in the Verify step with a clear mismatch
message.

A failure outside the Verify step (for example, the action step itself crashed)
indicates a real regression in the action, not in the harness.

## How verification works

The verification logic lives in two files:

- `src/e2e-verify.ts` — pure functions (`extractExpectDirective`, `compare`)
with unit tests in `src/e2e-verify.test.ts`. The runtime ships nothing here;
these are test-only utilities re-used by the workflow.
- `scripts/e2e-verify.ts` — the CLI shim invoked by the workflow's Verify
step. It reads environment variables (`ACTUAL_*`), calls the pure functions,
and exits non-zero on any mismatch.

`dist/` drift is not re-asserted by this workflow because the existing
`.github/workflows/ci.yaml` job already gates it on every PR and every push to
`main`.
48 changes: 48 additions & 0 deletions scripts/e2e-verify.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import process from 'node:process';
import { type ActualOutputs, compare, extractExpectDirective } from '../src/e2e-verify.js';

const env = (key: string): string => process.env[key] ?? '';

const actual: ActualOutputs = {
outcome: env('ACTUAL_OUTCOME'),
continue: env('ACTUAL_CONTINUE'),
command: env('ACTUAL_COMMAND'),
params: env('ACTUAL_PARAMS'),
number: env('ACTUAL_NUMBER'),
context: env('ACTUAL_CONTEXT'),
issue_number: env('ACTUAL_ISSUE_NUMBER'),
comment_id: env('ACTUAL_COMMENT_ID'),
actor: env('ACTUAL_ACTOR'),
};

let directive: Record<string, unknown> | null;
try {
directive = extractExpectDirective(actual.params);
} catch (e) {
console.error(`Failed to parse the dogfood directive — ${(e as Error).message}`);
process.exit(1);
}

if (directive === null) {
if (actual.continue === '') {
console.error('Smoke test failed: action emitted no "continue" output.');
process.exit(1);
}
if (actual.outcome === 'failure') {
console.error('Smoke test failed: action step outcome was "failure" but no directive declared it.');
process.exit(1);
}
console.log('Smoke test passed (no directive present in params).');
process.exit(0);
}

const mismatches = compare(directive, actual);
if (mismatches.length > 0) {
console.error('E2E mismatch:');
for (const m of mismatches) {
console.error(' - ' + m);
}
process.exit(1);
}

console.log('E2E expectations matched.');
Loading