Skip to content

Add the code testing agent and /add-tests workflow#7710

Open
JanKrivanek wants to merge 1 commit intomainfrom
dev/jankrivanek/add-test-ghaw
Open

Add the code testing agent and /add-tests workflow#7710
JanKrivanek wants to merge 1 commit intomainfrom
dev/jankrivanek/add-test-ghaw

Conversation

@JanKrivanek
Copy link
Copy Markdown
Member

@JanKrivanek JanKrivanek commented Apr 10, 2026

Motivation

Add the code testing agent and it's hooking into slash commend
This should help driving the test coverage for the changes added by PRs

Adapted from: dotnet/machinelearning#7602

cc: @Evangelink

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new /add-tests slash-command workflow (via gh-aw) intended to generate unit tests for PR changes, and introduces a reusable “code testing agent” skill + agent definitions to drive the generation pipeline.

Changes:

  • Added /add-tests workflow source (add-tests.md), shared repo build setup, and compiled lock workflow (add-tests.lock.yml).
  • Introduced a new code-testing-agent skill (prompt + .NET extension guidance).
  • Added a set of code-testing-* agent definitions (research/plan/implement/build/test/fix/lint orchestration).
Show a summary per file
File Description
.github/workflows/shared/repo-build-setup.md Repo-specific build steps shared by the add-tests workflow.
.github/workflows/add-tests.md gh-aw workflow manifest/prompt that defines the /add-tests behavior.
.github/workflows/add-tests.lock.yml Compiled GitHub Actions workflow that executes the agentic flow.
.github/skills/code-testing-agent/unit-test-generation.prompt.md Baseline guidance for generating high-quality unit tests (80% target).
.github/skills/code-testing-agent/SKILL.md Skill wrapper describing the Research→Plan→Implement pipeline and state files.
.github/skills/code-testing-agent/extensions/dotnet.md .NET-specific build/test/lint guidance for test generation.
.github/aw/actions-lock.json Locks SHAs for selected actions used by the workflow framework.
.github/agents/code-testing-generator.agent.md Orchestrator agent for the RPI test-generation pipeline.
.github/agents/code-testing-researcher.agent.md Research sub-agent to map codebase/tests/commands.
.github/agents/code-testing-planner.agent.md Planning sub-agent to produce phased test plan.
.github/agents/code-testing-implementer.agent.md Implementation sub-agent to write tests and verify build/test.
.github/agents/code-testing-builder.agent.md Build sub-agent guidance.
.github/agents/code-testing-tester.agent.md Test execution sub-agent guidance.
.github/agents/code-testing-fixer.agent.md Compilation error fixing sub-agent guidance.
.github/agents/code-testing-linter.agent.md Formatting/linting sub-agent guidance.

Copilot's findings

Comments suppressed due to low confidence (1)

.github/workflows/add-tests.md:72

  • This section tells the agent to use the task tool to invoke code-testing-generator, but the workflow doesn’t grant that tool (see tools: allowlist and compiled --allow-tool list). Update either the tool allowlist or the instructions so the workflow isn’t asking the agent to do something it cannot do.
### Step 3: Generate Tests

Use the `code-testing-generator` agent (defined at `.github/agents/code-testing-generator.agent.md`) via the task tool to generate tests:

1. Follow the Research → Plan → Implement pipeline from the skill
  • Files reviewed: 15/15 changed files
  • Comments generated: 4

activation:
needs: pre_activation
if: "needs.pre_activation.outputs.activated == 'true' && (github.event_name == 'issue_comment' && (startsWith(github.event.comment.body, '/add-tests ') || startsWith(github.event.comment.body, '/add-tests\n') || github.event.comment.body == '/add-tests') && github.event.issue.pull_request != null)"
runs-on: ubuntu-slim
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runs-on: ubuntu-slim is not used anywhere else in this repo (all other workflows use ubuntu-latest). If this repo doesn't have a self-hosted runner labeled ubuntu-slim, these jobs will never start. Consider switching to ubuntu-latest (or a pinned GitHub-hosted runner like ubuntu-24.04) for pre_activation, activation, safe_outputs, and conclusion jobs, or document where ubuntu-slim comes from.

Suggested change
runs-on: ubuntu-slim
runs-on: ubuntu-latest

Copilot uses AI. Check for mistakes.
tools:
github:
toolsets: [pull_requests, repos]
edit:
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The prompt instructs using the task tool to invoke the code-testing-generator agent, but this workflow’s tools: allowlist only includes github, edit, and bash (no task). In the compiled workflow, the Copilot CLI invocation also doesn’t allow a task tool. Either add the missing tool capability (if supported by gh-aw) or rewrite the instructions to not depend on task/sub-agents so the workflow can actually follow them.

This issue also appears on line 68 of the same file.

Suggested change
edit:
edit:
task:

Copilot uses AI. Check for mistakes.
run: ./build.sh

- name: Put dotnet on the path
run: echo "PATH=$PWD/.dotnet:$PATH" >> $GITHUB_ENV
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step modifies PATH by writing a full PATH=... line to $GITHUB_ENV. GitHub Actions provides $GITHUB_PATH specifically for appending directories to PATH, which avoids accidentally clobbering PATH and handles quoting more safely. Consider switching this to append $PWD/.dotnet to $GITHUB_PATH instead of rewriting PATH via $GITHUB_ENV.

Suggested change
run: echo "PATH=$PWD/.dotnet:$PATH" >> $GITHUB_ENV
run: echo "$PWD/.dotnet" >> $GITHUB_PATH

Copilot uses AI. Check for mistakes.
- name: Build
run: ./build.sh
- name: Put dotnet on the path
run: echo "PATH=$PWD/.dotnet:$PATH" >> $GITHUB_ENV
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step writes PATH=$PWD/.dotnet:$PATH into $GITHUB_ENV, which rewrites PATH rather than appending to it. Prefer using $GITHUB_PATH to add $PWD/.dotnet to PATH so you don’t risk clobbering PATH and to align with GitHub Actions’ intended mechanism. (Since this file is generated, the fix should be made in the source .github/workflows/shared/repo-build-setup.md and then recompiled.)

Suggested change
run: echo "PATH=$PWD/.dotnet:$PATH" >> $GITHUB_ENV
run: echo "$PWD/.dotnet" >> "$GITHUB_PATH"

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Member

@Youssef1313 Youssef1313 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same from dotnet/skills right?

I'm adding it already in #7707 and did automation for it.

It was pending markdownlint violations that I fixed in dotnet/skills#513

Comment on lines +58 to +61
- `src/TestFramework/` → `test/UnitTests/TestFramework.UnitTests/`
- `src/Adapter/MSTest.TestAdapter/` → `test/UnitTests/MSTestAdapter.UnitTests/`
- `src/Adapter/MSTestAdapter.PlatformServices/` → `test/UnitTests/MSTestAdapter.PlatformServices.UnitTests/`
- `src/Adapter/MSTest.Engine/` → `test/UnitTests/MSTest.Engine.UnitTests/`
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For MSTest, more often than not, unit tests are extremely useless. We almost always prefer an integration test, except for MSTest.Analyzers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants