feat: add conformance test infrastructure for v1.x #1518
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Port the conformance test setup from main (v2) to v1.x, adapting for the single-package SDK structure and v1.x API conventions. Includes: - Everything server (all MCP features: tools, resources, prompts, logging, completions, elicitation, sampling, SSE resumability) - Everything client (20 scenarios: initialize, tools_call, auth/*, elicitation, sse-retry) - Shell script to orchestrate server conformance tests - CI workflow running both client and server conformance on v1.x pushes and PRs (continue-on-error: true) - Root package.json conformance scripts Key adaptations from main: - Single package imports (../../src/...) instead of workspace packages - StreamableHTTPServerTransport instead of NodeStreamableHTTPServerTransport - extra.sendNotification/sendRequest instead of ctx.mcpReq.notify/send - zod v3 imports instead of zod/v4 - npm instead of pnpm - .tool()/.resource()/.prompt() API (same on both branches) Test results: 30 server scenarios (40 checks) all pass, 19 client scenarios (240 checks) all pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use registerTool/registerResource/registerPrompt (non-deprecated APIs) instead of tool/resource/prompt, matching main's pattern exactly - Restore prompt title metadata (was lost when using deprecated prompt() API) - Remove auth/pre-registration and auth/resource-mismatch scenarios that were added to v1.x but don't exist in main - Pin conformance runner to 0.1.10 to match main - Add back _authCodePromise field in OAuth provider to match main Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
commit: |
- Import OAuth types from src/shared/auth.js (where they're exported) - Import FetchLike from src/shared/transport.js - Use schema objects for setRequestHandler (SubscribeRequestSchema, etc.) - Use deprecated .tool()/.prompt() APIs (registerTool has zod v3 type recursion issues on v1.x) - Use CreateMessageResultSchema for sampling sendRequest - Add ts-expect-error for 2 unavoidable zod v3 type depth limits Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Conformance CI now gates merges: unexpected failures block PRs, but known failures in the baseline are tolerated. Stale baseline entries (scenarios that start passing) also block, forcing cleanup. Current baseline: - tools_call: conformance runner bug (fix merged in PR #141, pending release) - auth/pre-registration: needs client handler (not yet in main either) Also pins conformance to 0.1.11. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…eline - Prettier formatting for conformance-baseline.yml - dns-rebinding-protection partially fails in CI (GitHub Actions environment differs from localhost for binding checks) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add localhostHostValidation() middleware to reject requests with non-localhost Host headers. This is what createMcpExpressApp() does automatically — our manual express setup was missing it. Removes dns-rebinding-protection from the server expected-failures baseline since the test now passes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the separate test/conformance/package.json — conformance runner is now a root devDependency. No more separate npm install step in CI. Scripts updated to run from repo root. Shell script paths adjusted. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
felixweinberger
force-pushed
the
fweinberger/conformance-v1x
branch
from
1f9d6a0 to
12a161f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ports the conformance test setup from main (v2) to v1.x, adapting for the single-package SDK structure and v1.x API conventions.
Motivation and Context
The v1.x branch has no conformance test infrastructure. This adds it so conformance can be validated on v1.x, and the tier-check tool can produce actual conformance results.
Components
Key adaptations from main
How Has This Been Tested?
Server: 30 scenarios (40 checks), all pass.
Client: 23 scenarios, 265 passed, 2 failed (known conformance runner bugs — `tools_call` hits GHSA-345p-7cg4-v4c7 guard in runner's test server, fix pending in modelcontextprotocol/conformance#141).
Breaking Changes
None — adds test infrastructure only.
Types of changes
Checklist