Skip to content

ci: add subprocess-based tutorial tests#58

Merged
thephez merged 3 commits intomainfrom
test-add-testing
Mar 12, 2026
Merged

ci: add subprocess-based tutorial tests#58
thephez merged 3 commits intomainfrom
test-add-testing

Conversation

@thephez
Copy link
Collaborator

@thephez thephez commented Mar 12, 2026

Summary

  • Add test infrastructure that runs tutorial scripts as-is via subprocess execution and validates their output
  • Uses Node.js built-in node:test — zero new dependencies
  • Read-only tests (9 tutorials) run on push/PR; write tests (12 tutorials) run via manual workflow dispatch

Details

Test architecture

Each tutorial is spawned as a child process with child_process.execFile. The harness captures stdout/stderr, checks exit codes, and validates output against expected patterns. Environment variables are injected to pass state between dependent tutorials (e.g., contract ID from registration flows into document submission).

Files added

  • test/run-tutorial.mjs — subprocess runner
  • test/assertions.mjs — shared validation helpers (assertTutorialSuccess, extractId)
  • test/read-only.test.mjs — 9 read-only tutorial tests
  • test/read-write.test.mjs — 12 sequential write tutorial tests with state passing
  • .github/workflows/test-tutorials.yml — CI with choice dropdown (read-only / read-write / all)

Scripts

  • npm test / npm run test:read-only — read-only tests
  • npm run test:read-write — write pipeline (sequential, consumes testnet credits)
  • npm run test:all — both suites

Not yet testable (hardcoded values, no env var override)

  • document-update.mjs, document-delete.mjsDOCUMENT_ID hardcoded
  • name-register.mjsNAME_LABEL hardcoded
  • identity-update-disable-key.mjsKEY_ID hardcoded

Test plan

  • npm run test:read-only passes (9/9)
  • npm run test:read-write passes with funded testnet wallet
  • CI workflow triggers correctly on push and manual dispatch

Summary by CodeRabbit

  • Tests
    • Added dedicated read-only and read-write tutorial test suites and a combined test command; CI workflow to run them on main and PRs and via manual dispatch.
  • Documentation
    • New "Testing" section with usage and environment guidance for running tutorial tests.
  • Chores
    • Added test runner scripts and helper utilities to orchestrate, time, and validate tutorial executions.

thephez added 2 commits March 12, 2026 12:13
Excludes identity update (disable key), register name, document update, document delete
@coderabbitai
Copy link

coderabbitai bot commented Mar 12, 2026

📝 Walkthrough

Walkthrough

Adds end-to-end tutorial testing: a GitHub Actions workflow, new npm test scripts, subprocess-based test runner and assertion utilities, plus read-only and read-write test suites that execute tutorial scripts and validate outputs.

Changes

Cohort / File(s) Summary
CI/CD Workflow
​.github/workflows/test-tutorials.yml
New GitHub Actions workflow "Tutorial Tests" with two jobs (test-read-only, test-read-write), triggers for push/PR/main and manual dispatch, job timeouts, concurrency group for read-write, and env/secret wiring.
NPM scripts
package.json
Replaced default test script to run read-only suite; added test:read-only, test:read-write, and test:all scripts with explicit Node test runner invocations and timeouts.
Test utilities
test/assertions.mjs, test/run-tutorial.mjs
Added runTutorial(scriptPath, options) to spawn tutorial subprocesses with merged env, timeout, and standardized output shape; added assertion helpers assertTutorialSuccess, extractFromOutput, and extractId to validate and parse tutorial outputs.
Test suites
test/read-only.test.mjs, test/read-write.test.mjs
Added read-only suite that runs multiple tutorials and checks expected/error patterns; added sequential read-write suite that runs identity, contract, and document tutorials, captures IDs into shared state, and conditions some tests on captured values.
Docs
README.md
Added a "Testing" section describing how tests run as subprocesses, required environment configuration, and the new npm test commands.

Sequence Diagram(s)

sequenceDiagram
  participant GH as GitHub Actions
  participant Runner as Ubuntu VM (job)
  participant Node as Node/npm
  participant Script as Tutorial script
  participant Asserts as Test assertions

  GH->>Runner: trigger job (push/PR/dispatch)
  Runner->>Runner: checkout code\nsetup Node.js\nnpm ci
  Runner->>Node: npm run test:read-*/test:all
  Node->>Script: spawn node <tutorial script>
  Script-->>Node: stdout/stderr, exit
  Node->>Asserts: pass output to assertions
  Asserts-->>Node: success/failure result
  Node-->>Runner: test job exit status
  Runner-->>GH: job result
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 Hops and hops, I run each test,

Scripts and workflows put to quest,
Read-only first, then writes parade,
IDs collected, assertions made,
I nibble bugs and celebrate the rest.

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately captures the main objective: adding test infrastructure based on subprocesses for tutorial validation, directly aligned with the primary change across all modified files.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch test-add-testing
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/test-tutorials.yml:
- Around line 40-44: The "Run read-only tutorial tests" job is exporting the
secret PLATFORM_MNEMONIC into the step environment even for pull_request runs;
remove PLATFORM_MNEMONIC from the env block (or conditionally inject it only for
trusted runs) so that the step that runs npm run test:read-only does not receive
the funded mnemonic; alternatively, move any tests that require
PLATFORM_MNEMONIC behind a trusted/manual workflow or a separate job that only
runs on push/dispatch so the secret is never exposed to untrusted PRs.

In `@package.json`:
- Around line 9-12: Add an engines constraint to package.json so contributors
run a supported Node version for the test CLI flags: update package.json to
include "engines": { "node": ">=20.11.0" } (this ensures the test scripts
"test", "test:read-only", "test:read-write", and "test:all" that use
--test-timeout work correctly).

In `@test/read-write.test.mjs`:
- Around line 20-28: Add assertions that the spawned subprocess exited
successfully: after the existing assert.match on result.stdout,
assert.strictEqual(result.exitCode, 0, `Expected zero exit code.\nSTDOUT:
${result.stdout}\nSTDERR: ${result.stderr}`) and
assert.strictEqual(result.killed, false, `Process was killed
unexpectedly.\nSTDOUT: ${result.stdout}\nSTDERR: ${result.stderr}`) so the test
requires both the "Identity registered!" output and a successful process
termination; keep the existing stdout/stderr diagnostic strings and the current
workaround that reads result.stdout/result.stderr.
- Around line 107-109: The test currently silently continues when
extractId(result.stdout) returns falsy, leaving state.dataContractId unset;
update the block around extractId(result.stdout) so that if no id is returned
you immediately fail the test (e.g., throw an Error or use an assertion) with a
clear message including the stdout, otherwise set state.dataContractId to the
id; refer to the extractId function result and state.dataContractId assignment
to implement the early-fail behavior so downstream tests don’t silently skip.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ee8bac4f-baa9-49c4-805a-727861c13b15

📥 Commits

Reviewing files that changed from the base of the PR and between fdda0ce and 94e7b21.

📒 Files selected for processing (6)
  • .github/workflows/test-tutorials.yml
  • package.json
  • test/assertions.mjs
  • test/read-only.test.mjs
  • test/read-write.test.mjs
  • test/run-tutorial.mjs

Fail early when extractId returns null so downstream tests don't silently skip. Also support the $id key format used by document output.
Update readme
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
test/read-write.test.mjs (1)

20-28: ⚠️ Potential issue | 🟠 Major

Assert process success here as well.

This still goes green on a non-zero exit or timeout as long as stdout contains Identity registered!. Keep the special-case output check if needed, but also require result.killed === false and result.exitCode === 0 so the subprocess actually succeeded.

Suggested tightening
     assert.match(
       result.stdout,
       /Identity registered!/,
       `Expected identity registration output.\nSTDOUT: ${result.stdout}\nSTDERR: ${result.stderr}`,
     );
-    console.log(result.stdout)
+    assert.equal(
+      result.killed,
+      false,
+      `identity-register timed out.\nSTDOUT: ${result.stdout}\nSTDERR: ${result.stderr}`,
+    );
+    assert.equal(
+      result.exitCode,
+      0,
+      `identity-register exited with code ${result.exitCode}.\nSTDOUT: ${result.stdout}\nSTDERR: ${result.stderr}`,
+    );
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/read-write.test.mjs` around lines 20 - 28, The test currently only
asserts stdout contains "Identity registered!" via the assert.match call;
tighten it by also asserting the child process actually succeeded: add checks
that result.killed === false and result.exitCode === 0 immediately after the
assert.match (or before) so the test fails on non-zero exit or timeout;
reference the existing result object fields result.killed and result.exitCode
and the assert.match usage to locate where to insert the two additional
assertions.
🧹 Nitpick comments (1)
README.md (1)

57-65: Document npm run test:read-only explicitly too.

npm test is enough for the default path, but the PR also adds a named test:read-only script. Listing that alias here would keep the README aligned with the workflow wording and make the available entry points clearer for contributors.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@README.md` around lines 57 - 65, Update the README test section to explicitly
mention the named script alias "npm run test:read-only" alongside "npm test" so
the documentation matches the new package.json scripts; reference both "npm
test" (default/read-only) and "npm run test:read-only" (explicit read-only
alias), and keep the descriptions for "npm run test:read-write" and "npm run
test:all" unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@test/read-write.test.mjs`:
- Around line 20-28: The test currently only asserts stdout contains "Identity
registered!" via the assert.match call; tighten it by also asserting the child
process actually succeeded: add checks that result.killed === false and
result.exitCode === 0 immediately after the assert.match (or before) so the test
fails on non-zero exit or timeout; reference the existing result object fields
result.killed and result.exitCode and the assert.match usage to locate where to
insert the two additional assertions.

---

Nitpick comments:
In `@README.md`:
- Around line 57-65: Update the README test section to explicitly mention the
named script alias "npm run test:read-only" alongside "npm test" so the
documentation matches the new package.json scripts; reference both "npm test"
(default/read-only) and "npm run test:read-only" (explicit read-only alias), and
keep the descriptions for "npm run test:read-write" and "npm run test:all"
unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bf0594df-be38-410b-8590-d690ab198098

📥 Commits

Reviewing files that changed from the base of the PR and between 94e7b21 and bd90eb8.

📒 Files selected for processing (3)
  • README.md
  • test/assertions.mjs
  • test/read-write.test.mjs

@thephez thephez merged commit 905392b into main Mar 12, 2026
6 of 9 checks passed
@thephez thephez deleted the test-add-testing branch March 12, 2026 20:21
@thephez thephez mentioned this pull request Mar 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant