Add the code testing agent and /add-tests workflow#7710
Add the code testing agent and /add-tests workflow#7710JanKrivanek wants to merge 1 commit intomainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new /add-tests slash-command workflow (via gh-aw) intended to generate unit tests for PR changes, and introduces a reusable “code testing agent” skill + agent definitions to drive the generation pipeline.
Changes:
- Added
/add-testsworkflow source (add-tests.md), shared repo build setup, and compiled lock workflow (add-tests.lock.yml). - Introduced a new
code-testing-agentskill (prompt + .NET extension guidance). - Added a set of
code-testing-*agent definitions (research/plan/implement/build/test/fix/lint orchestration).
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/shared/repo-build-setup.md | Repo-specific build steps shared by the add-tests workflow. |
| .github/workflows/add-tests.md | gh-aw workflow manifest/prompt that defines the /add-tests behavior. |
| .github/workflows/add-tests.lock.yml | Compiled GitHub Actions workflow that executes the agentic flow. |
| .github/skills/code-testing-agent/unit-test-generation.prompt.md | Baseline guidance for generating high-quality unit tests (80% target). |
| .github/skills/code-testing-agent/SKILL.md | Skill wrapper describing the Research→Plan→Implement pipeline and state files. |
| .github/skills/code-testing-agent/extensions/dotnet.md | .NET-specific build/test/lint guidance for test generation. |
| .github/aw/actions-lock.json | Locks SHAs for selected actions used by the workflow framework. |
| .github/agents/code-testing-generator.agent.md | Orchestrator agent for the RPI test-generation pipeline. |
| .github/agents/code-testing-researcher.agent.md | Research sub-agent to map codebase/tests/commands. |
| .github/agents/code-testing-planner.agent.md | Planning sub-agent to produce phased test plan. |
| .github/agents/code-testing-implementer.agent.md | Implementation sub-agent to write tests and verify build/test. |
| .github/agents/code-testing-builder.agent.md | Build sub-agent guidance. |
| .github/agents/code-testing-tester.agent.md | Test execution sub-agent guidance. |
| .github/agents/code-testing-fixer.agent.md | Compilation error fixing sub-agent guidance. |
| .github/agents/code-testing-linter.agent.md | Formatting/linting sub-agent guidance. |
Copilot's findings
Comments suppressed due to low confidence (1)
.github/workflows/add-tests.md:72
- This section tells the agent to use the
task toolto invokecode-testing-generator, but the workflow doesn’t grant that tool (seetools:allowlist and compiled--allow-toollist). Update either the tool allowlist or the instructions so the workflow isn’t asking the agent to do something it cannot do.
### Step 3: Generate Tests
Use the `code-testing-generator` agent (defined at `.github/agents/code-testing-generator.agent.md`) via the task tool to generate tests:
1. Follow the Research → Plan → Implement pipeline from the skill
- Files reviewed: 15/15 changed files
- Comments generated: 4
| activation: | ||
| needs: pre_activation | ||
| if: "needs.pre_activation.outputs.activated == 'true' && (github.event_name == 'issue_comment' && (startsWith(github.event.comment.body, '/add-tests ') || startsWith(github.event.comment.body, '/add-tests\n') || github.event.comment.body == '/add-tests') && github.event.issue.pull_request != null)" | ||
| runs-on: ubuntu-slim |
There was a problem hiding this comment.
runs-on: ubuntu-slim is not used anywhere else in this repo (all other workflows use ubuntu-latest). If this repo doesn't have a self-hosted runner labeled ubuntu-slim, these jobs will never start. Consider switching to ubuntu-latest (or a pinned GitHub-hosted runner like ubuntu-24.04) for pre_activation, activation, safe_outputs, and conclusion jobs, or document where ubuntu-slim comes from.
| runs-on: ubuntu-slim | |
| runs-on: ubuntu-latest |
| tools: | ||
| github: | ||
| toolsets: [pull_requests, repos] | ||
| edit: |
There was a problem hiding this comment.
The prompt instructs using the task tool to invoke the code-testing-generator agent, but this workflow’s tools: allowlist only includes github, edit, and bash (no task). In the compiled workflow, the Copilot CLI invocation also doesn’t allow a task tool. Either add the missing tool capability (if supported by gh-aw) or rewrite the instructions to not depend on task/sub-agents so the workflow can actually follow them.
This issue also appears on line 68 of the same file.
| edit: | |
| edit: | |
| task: |
| run: ./build.sh | ||
|
|
||
| - name: Put dotnet on the path | ||
| run: echo "PATH=$PWD/.dotnet:$PATH" >> $GITHUB_ENV |
There was a problem hiding this comment.
This step modifies PATH by writing a full PATH=... line to $GITHUB_ENV. GitHub Actions provides $GITHUB_PATH specifically for appending directories to PATH, which avoids accidentally clobbering PATH and handles quoting more safely. Consider switching this to append $PWD/.dotnet to $GITHUB_PATH instead of rewriting PATH via $GITHUB_ENV.
| run: echo "PATH=$PWD/.dotnet:$PATH" >> $GITHUB_ENV | |
| run: echo "$PWD/.dotnet" >> $GITHUB_PATH |
| - name: Build | ||
| run: ./build.sh | ||
| - name: Put dotnet on the path | ||
| run: echo "PATH=$PWD/.dotnet:$PATH" >> $GITHUB_ENV |
There was a problem hiding this comment.
This step writes PATH=$PWD/.dotnet:$PATH into $GITHUB_ENV, which rewrites PATH rather than appending to it. Prefer using $GITHUB_PATH to add $PWD/.dotnet to PATH so you don’t risk clobbering PATH and to align with GitHub Actions’ intended mechanism. (Since this file is generated, the fix should be made in the source .github/workflows/shared/repo-build-setup.md and then recompiled.)
| run: echo "PATH=$PWD/.dotnet:$PATH" >> $GITHUB_ENV | |
| run: echo "$PWD/.dotnet" >> "$GITHUB_PATH" |
There was a problem hiding this comment.
This is the same from dotnet/skills right?
I'm adding it already in #7707 and did automation for it.
It was pending markdownlint violations that I fixed in dotnet/skills#513
| - `src/TestFramework/` → `test/UnitTests/TestFramework.UnitTests/` | ||
| - `src/Adapter/MSTest.TestAdapter/` → `test/UnitTests/MSTestAdapter.UnitTests/` | ||
| - `src/Adapter/MSTestAdapter.PlatformServices/` → `test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/` | ||
| - `src/Adapter/MSTest.Engine/` → `test/UnitTests/MSTest.Engine.UnitTests/` |
There was a problem hiding this comment.
For MSTest, more often than not, unit tests are extremely useless. We almost always prefer an integration test, except for MSTest.Analyzers.
Motivation
Add the code testing agent and it's hooking into slash commend
This should help driving the test coverage for the changes added by PRs
Adapted from: dotnet/machinelearning#7602
cc: @Evangelink