ci(e2e): wire Playwright suite into DTT with a run summary

[stephenliang]

Wires the Playwright e2e suite into the automated deploy-time test train (DTT), and fixes why its #infra-test messages were missing report links and counts. Builds on the existing test:playwright_ui task (already run by Drone), verified to run manually on the DTT.

DTT wiring

Adds :playwright_ui to test:ui_all, so the DTT runs it in parallel with the Cucumber/eyes suites. It is non-blocking (rescues its own failure), so it reports to #infra-test but never fails the train. No double-run in Drone — that path is ci:run_ui_tests, which doesn't touch ui_all.

Provisioning (flake fix)

Drops the per-run playwright install-deps from the shared CI script: it shelled out to apt on every DTT run (root, network, hang-prone). The version-aware playwright install stays — a ~0.5s no-op when the pinned browser build is present, fetching only after a version bump. OS-level deps are now provisioned once out-of-band.

Reporting (the #infra-test message)

Root cause: the html/json reporters were gated on CI, which the DTT daemon doesn't set — so DTT produced no report dir and no machine-readable results, hence no link and no counts. Introduces an isAutomated signal (CI || PLAYWRIGHT_CI, the latter set by yarn test:ui:ci) that is true in both Drone and DTT; local test:ui:local stays lean (list reporter only). The task now posts a Cucumber-style summary — pass/fail/skip counts, flaky, duration, and report links at start and finish.

flowchart LR
  subgraph Drone
    A[ui_tests.sh] --> R["rake test:playwright_ui"]
  end
  subgraph DTT
    B["test:ui_all"] --> R
  end
  R --> S["run-playwright-tests-ci.sh"]
  S --> P["yarn test:ui:ci sets PLAYWRIGHT_CI"]
  P --> C{"isAutomated? CI or PLAYWRIGHT_CI"}
  C -->|yes| RPT["list + html + json"]
  C -->|local| LST["list only"]
  RPT --> U["upload report to S3"]
  RPT --> J["parse results.json"]
  U --> M["infra-test summary: counts, duration, links"]
  J --> M

Loading

Links

• Jira: n/a

Testing story

• yarn typecheck, Ruby/shell syntax, and pre-commit lint all pass.
• Reporter gating verified functionally: no env → list only (no artifacts); PLAYWRIGHT_CI=1 → playwright-report/ + test-results/results.json generated.
• Real helper code exercised against a real results.json plus synthetic mixes (fail/flaky/skip) and the missing-file fallback.
• End-to-end TARGET_URL=https://test-studio.code.org bundle exec rake test:playwright_ui against the test env: 9/9 passed, real S3 report link in the posted message.
• playwright install confirmed as a 0.46s no-op when browsers are present.
• Full DTT path (test:ui_all) not run locally — it dispatches the eyes/SauceLabs/Device Farm suites, which need remote infra; statically confirmed :playwright_ui is in the list.

Deployment notes

• One-time on the DTT daemon: ensure browser OS deps are present (sudo yarn exec playwright install-deps), since the per-run install-deps is removed. A future Playwright bump that adds a new system lib would need this re-run once.

🤖 Generated with Claude Code