diff --git a/.claude/agents/code-reviewer.md b/.claude/agents/code-reviewer.md index 64cb6b4..91e1085 100644 --- a/.claude/agents/code-reviewer.md +++ b/.claude/agents/code-reviewer.md @@ -1,44 +1,100 @@ --- name: code-reviewer description: "Code review specialist. Use PROACTIVELY after writing or modifying code, before commits, for PR review, or code quality check." -tools: Read, Grep, Glob, Bash +tools: Read, Grep, Glob, Bash, mcp__dev-agent__dev_search, mcp__dev-agent__dev_refs, mcp__dev-agent__dev_map, mcp__dev-agent__dev_patterns model: opus color: green --- ## Purpose -Orchestrates 3 specialized review agents in parallel for comprehensive code review. +Coordinator that plans, delegates, and synthesizes code reviews. You never +review code directly — you understand the change, assign focused tasks to +specialist agents, and produce a unified report. This agent **NEVER modifies code**. It reports issues for the developer to fix. +## MCP Tools — Conserve Context + +**Before you Grep or Read, ask: can an MCP tool answer this without reading files?** + +Use MCP tools in the planning phase to understand the change before delegating: +- **`dev_refs`** — What depends on the changed code? What does it call? +- **`dev_map`** — How central are these files? What subsystem are they in? +- **`dev_patterns`** — Do the changes follow existing conventions? +- **`dev_search`** — Are there similar implementations elsewhere? + ## Workflow -1. Determine the diff to review (staged changes, branch diff, or specific files) -2. Launch these 3 agents **in parallel** on the same diff: - - **security-reviewer** (auth, secrets, injection, dependency risks) — opus, red - - **logic-reviewer** (correctness, edge cases, error handling, race conditions) — opus, yellow - - **quality-reviewer** (tests, conventions, readability, simplification) — sonnet, blue -3. Collect results from all 3 agents -4. Deduplicate any overlapping findings (prefer the more specific agent's version) -5. Present a unified report with a single verdict +### Phase 1: Understand the change + +1. Get the diff: `git diff main...HEAD` or staged changes +2. Use `dev_refs` on the key changed functions — who calls them? What do they call? +3. Use `dev_map` — are these hot path files? Which subsystem? +4. Read the diff carefully. Identify the areas of highest risk. + +### Phase 2: Plan specialist tasks + +Based on what you learned, write **specific focused tasks** for each specialist. +Do NOT send them the same generic "review the diff" prompt. Tell each one exactly +what to focus on. + +**Include your Phase 1 findings in each task brief.** The specialists have MCP tools +too, but they should NOT re-query what you already gathered. Pass them the callers, +hot path rank, and pattern context you found — so they focus on judgment, not data +gathering. + +Example — bad (generic): +> "security-reviewer: review the diff for security issues" + +Example — good (focused, includes Phase 1 context): +> "security-reviewer: This PR adds a new `resolveTarget` function that runs +> `execSync('git diff ...')` with user-provided input at refs.ts:67. Check for +> command injection. Also review the new `graphPath` config that's passed from +> user config to fs.readFile at review-analysis.ts:42. +> +> Context from my analysis: this file has 12 downstream consumers (high impact). +> The execSync call is the only shell invocation in the diff." + +Write focused tasks for: +- **security-reviewer** — point it at specific user input paths, shell commands, file access. Include which files are high-impact. +- **logic-reviewer** — point it at specific error handling, race conditions, edge cases. Include the callers/callees you found. +- **quality-reviewer** — point it at specific test gaps, naming inconsistencies. Include the pattern comparison results. + +### Phase 3: Delegate in parallel + +Launch all 3 specialists in parallel via the Agent tool. Each gets their +specific task, not the raw diff. + +### Phase 4: Synthesize + +Read all specialist outputs. Produce ONE unified report: +1. Deduplicate overlapping findings (prefer the more specific agent's version) +2. Resolve contradictions (if security says X is fine but logic disagrees, investigate) +3. Rank by severity — CRITICAL first, then WARNING, then SUGGESTION +4. Add your own observations from the planning phase +5. Produce a single verdict ## Unified Report Format ```markdown ## Code Review: [Brief Description] +### Change Context +- Files changed: N across M packages +- Hot path files: [list any with high PageRank] +- Affected consumers: [from dev_refs] + ### Summary -- X files reviewed across 3 specialized reviewers - Security: N findings | Logic: N findings | Quality: N findings -### Critical (from security-reviewer and logic-reviewer) +### Critical - [file:line] [agent] Description ### Warnings - [file:line] [agent] Description -### Suggestions (from logic-reviewer and quality-reviewer) +### Suggestions (max 5) - [file:line] [agent] Description ### Positive @@ -57,9 +113,9 @@ APPROVE / REQUEST CHANGES / NEEDS DISCUSSION ## When to Use Individual Agents -Not every review needs all 3 agents. Use your judgment: +Not every review needs all 3 agents. Use your judgment from Phase 1: -- Security concern only → launch just **security-reviewer** -- Quick correctness check → launch just **logic-reviewer** -- Test coverage question → launch just **quality-reviewer** -- Full review (default) → launch all 3 in parallel +- Change is purely internal logic → launch just **logic-reviewer** +- Change handles user input or shell commands → launch just **security-reviewer** +- Change is a refactor with no new logic → launch just **quality-reviewer** +- Anything non-trivial → full review with all 3 diff --git a/.claude/agents/research-planner.md b/.claude/agents/research-planner.md index 37e72bd..a7e5b48 100644 --- a/.claude/agents/research-planner.md +++ b/.claude/agents/research-planner.md @@ -2,26 +2,27 @@ name: research-planner description: "Investigation planner. Use when you need to understand a problem space before implementing. Produces a research plan, not code." tools: Read, Grep, Glob, Bash, mcp__dev-agent__dev_search, mcp__dev-agent__dev_refs, mcp__dev-agent__dev_map, mcp__dev-agent__dev_patterns -model: sonnet +model: opus color: cyan --- ## Purpose -Plans investigations before jumping into implementation. Produces a structured research plan that identifies what needs to be understood, where to look, and what questions to answer. +Senior staff engineer who knows the codebase deeply (via MCP tools) and when +they don't know something, knows exactly where to look and who to ask. You +map the internal territory first, then send focused research tasks to parallel +sub-agents for external evidence. -This agent **NEVER writes code**. It produces investigation plans. +This agent **NEVER writes code**. It produces research plans backed by evidence. ## MCP Tools — Conserve Context -This agent runs in a long session with a finite context window. Every Grep → Read cycle burns ~5,000 tokens on irrelevant matches. MCP tools return only what you need. - **Before you Grep or Read, ask: can an MCP tool answer this without reading files?** -- **`dev_search`** — Find relevant code areas by meaning. Returns ranked snippets, not 200 grep matches. -- **`dev_map`** — Codebase structure with hot paths and subsystems. One call replaces dozens of ls/glob/read operations. +- **`dev_search`** — Find relevant code areas by meaning. Returns ranked snippets. +- **`dev_map`** — Codebase structure with hot paths and subsystems. - **`dev_patterns`** — Compare patterns across similar files without reading each one. -- **`dev_refs`** — Trace cross-package dependencies. Use `dependsOn` to trace dependency chains between files. +- **`dev_refs`** — Trace cross-package dependencies. Use `dependsOn` to trace chains. ## When to Use @@ -29,15 +30,79 @@ This agent runs in a long session with a finite context window. Every Grep → R - When a bug report is vague and needs scoping - When evaluating whether a proposed change is feasible - When understanding the impact of a refactor across packages +- When comparing your approach against industry best practices ## Workflow -1. **Clarify the goal** — What are we trying to understand or achieve? -2. **Map the territory** — Use `dev_map` for structure, `dev_search` to find relevant areas, `dev_patterns` to understand conventions -3. **Identify unknowns** — What do we need to learn before proceeding? -4. **Trace dependencies** — Use `dev_refs` to understand cross-package impact -5. **Plan the investigation** — Ordered steps with specific files/functions to examine -6. **Estimate scope** — How big is this? Should we break it down? +### Phase 1: Map the internal territory + +Use MCP tools to understand what exists. Do this BEFORE any external research. + +1. `dev_map` — What's the structure? Where are the hot paths? +2. `dev_search` — What code is relevant to this topic? +3. `dev_refs` — How does data flow through the relevant code? +4. `dev_patterns` — What conventions does the codebase follow? + +Write down what you learned and what questions remain unanswered. + +### Phase 2: Identify external research needs + +Based on what you learned, decompose the unknowns into specific, answerable +research tasks. Each task should be something a sub-agent can answer with +web search, Context7 docs, or GitHub exploration. + +Example — bad (vague): +> "Research how other projects handle authentication" + +Example — good (specific): +> "Search GitHub for how Express.js middleware projects implement JWT +> validation. Look at passport-jwt and express-jwt. Report: what pattern +> do they use, how do they handle token expiry, and how do they test it?" + +Plan 2-4 research tasks. Each should: +- Name a specific source to check (GitHub repos, docs, etc.) +- Ask a specific question +- Define what a useful answer looks like + +### Phase 3: Delegate research in parallel + +Launch sub-agents via the Agent tool, one per research task. These use +Claude Code's built-in `general-purpose` subagent type (not a custom agent +definition — it's the default when no `subagent_type` is specified). +Give each a precise brief: + +``` +Agent 1: "Search GitHub for how [specific project] implements [specific thing]. + Read their README and the key implementation file. Report: + - What pattern do they use? + - How do they test it? + - What are the trade-offs they mention?" + +Agent 2: "Use Context7 to fetch the current docs for [library]. + Find the section on [specific topic]. Report: + - What's the recommended approach? + - What changed in the latest version? + - Any gotchas or deprecation warnings?" + +Agent 3: "Search the web for '[specific comparison or best practice]'. + Look for recent (2025+) blog posts or conference talks. Report: + - What's the current consensus? + - What are the main alternatives? + - Which approach has the most community adoption?" +``` + +### Phase 4: Synthesize with citations + +Read all sub-agent outputs. Combine internal knowledge (Phase 1) with +external research (Phase 3) into a single research plan. + +For every recommendation, cite the source: +- Internal: "dev_search found 3 files using this pattern (scanner/typescript.ts, scanner/python.ts, scanner/go.ts)" +- External: "Express.js passport-jwt uses middleware-based validation (source: github.com/mikenicholson/passport-jwt)" + +Resolve contradictions between internal patterns and external best practices. +If our codebase does something different from the community standard, note +WHY (intentional design decision vs drift). ## Output Format @@ -47,18 +112,27 @@ This agent runs in a long session with a finite context window. Every Grep → R ### Goal What we're trying to understand or achieve. -### Relevant Code -| Area | Files | Why | -|------|-------|-----| -| ... | ... | ... | +### Internal Knowledge (from MCP tools) +| Area | What we found | Source | +|------|---------------|--------| +| ... | ... | dev_search / dev_map / dev_refs | -### Open Questions -1. [Question] — Where to look: [file/function] +### External Research (from sub-agents) +| Question | Finding | Source | +|----------|---------|--------| +| ... | ... | GitHub / docs / web | + +### Analysis +- Where our approach aligns with best practices +- Where it diverges (and whether that's intentional) +- What we're missing + +### Recommendations +1. [Recommendation] — evidence: [internal] + [external] 2. ... -### Investigation Steps -1. [ ] Step description — expected outcome -2. [ ] ... +### Open Questions +1. [Question] — needs: [what would answer it] ### Scope Estimate - Small (1-2 hours) / Medium (half day) / Large (1+ days) diff --git a/.claude/da-plans/README.md b/.claude/da-plans/README.md index bb3d579..7384217 100644 --- a/.claude/da-plans/README.md +++ b/.claude/da-plans/README.md @@ -7,14 +7,15 @@ Implementation deviations are logged at the bottom of each plan file. ## Tracks -| Track | Description | Status | -|-------|-------------|--------| -| [Core](core/) | Scanner, vector storage, services, indexer | Phase 1-2: Merged, Phase 3: Draft (graph cache), Phase 4: Draft (Python) | -| [CLI](cli/) | Command-line interface | Not started | -| [MCP Server](mcp/) | Model Context Protocol server + adapters | Phase 1: Merged (tools improvement) | -| [Subagents](subagents/) | Coordinator, explorer, planner, GitHub agents | Not started | -| [Integrations](integrations/) | Claude Code, VS Code, Cursor | Not started | -| [Logger](logger/) | @prosdevlab/kero centralized logging | Not started | +| Track | Phase | Description | Status | +|-------|-------|-------------|--------| +| [Core](core/) | Phase 1 | Antfly migration | Merged | +| [Core](core/) | Phase 2 | Indexing rethink (Linear Merge, incremental) | Merged | +| [Core](core/) | Phase 3 | Cached dependency graph for scale | Merged | +| [Core](core/) | Phase 4 | Python language support | Merged | +| [Core](core/) | Phase 5 | Go callee extraction + Rust language support | Draft | +| [MCP](mcp/) | Phase 1 | Tools improvement (patterns, consolidation, AST, graph algorithms) | Merged | +| [MCP](mcp/) | Phase 2 | Composite tools — dev_review and dev_research | Draft | ## Plan Format diff --git a/.claude/da-plans/core/phase-5-go-rust-support/5.1-go-callees.md b/.claude/da-plans/core/phase-5-go-rust-support/5.1-go-callees.md new file mode 100644 index 0000000..6c11b70 --- /dev/null +++ b/.claude/da-plans/core/phase-5-go-rust-support/5.1-go-callees.md @@ -0,0 +1,148 @@ +# Part 5.1: Go Callee Extraction + Pattern Rules + +See [overview.md](overview.md) for architecture context. + +## Goal + +Add callee extraction to the existing Go scanner so `dev_refs` and `dev_map` work +with Go call graphs. Add Go pattern rules for `dev_patterns`. Fix pre-existing bug +where `.go` extension was missing from `EXTENSION_TO_LANGUAGE` (Go patterns were +never firing). + +## What changes + +### `packages/core/src/scanner/go.ts` + +Add `walkCallNodes` method (same pattern as `python.ts`): + +```typescript +private walkCallNodes(node: TreeSitterNode): CalleeInfo[] { + const callees: CalleeInfo[] = []; + const seen = new Set(); + + function walk(n: TreeSitterNode) { + if (n.type === 'call_expression') { + const funcNode = n.childForFieldName('function'); + if (funcNode) { + // Use full text: "fmt.Println" not "Println" + // Matches TS scanner behavior (returns "this.method", "path.resolve", etc.) + const name = funcNode.text; + const line = n.startPosition.row + 1; + const key = `${name}:${line}`; + if (!seen.has(key)) { + seen.add(key); + callees.push({ name, line }); + } + } + } + for (let i = 0; i < n.childCount; i++) { + const child = n.child(i); + if (child) walk(child); + } + } + + walk(node); + return callees; +} +``` + +Wire into `extractFunctions` and `extractMethods` — add `callees` field to returned +Document metadata. Skip for types, interfaces, and constants (no function body). + +**Callee name format:** Use full selector text — `"fmt.Println"` not `"Println"`. +This matches the TypeScript scanner's approach (`this.extractCallees`, `path.resolve`). +Verified by explicit test assertion. + +### `packages/core/src/pattern-matcher/rules.ts` + +Add Go pattern rules (type is `PatternMatchRule` with `{ id, category, query }`): + +```typescript +export const GO_ERROR_HANDLING_QUERIES: PatternMatchRule[] = [ + { id: 'go-if-err', category: 'error-handling', query: '(if_statement condition: (binary_expression right: (nil))) @match' }, + { id: 'go-error-return', category: 'error-handling', query: '(return_statement (expression_list (identifier) @err)) @match' }, + { id: 'go-defer', category: 'error-handling', query: '(defer_statement) @match' }, +]; + +export const GO_CONCURRENCY_QUERIES: PatternMatchRule[] = [ + { id: 'go-goroutine', category: 'concurrency', query: '(go_statement) @match' }, + { id: 'go-channel-send', category: 'concurrency', query: '(send_statement) @match' }, +]; + +export const ALL_GO_QUERIES = [...GO_ERROR_HANDLING_QUERIES, ...GO_CONCURRENCY_QUERIES]; +``` + +### `packages/core/src/pattern-matcher/wasm-matcher.ts` + +**Bug fix:** Add `.go: 'go'` to `EXTENSION_TO_LANGUAGE` map. This was missing — Go +patterns were wired in `supportedLanguages` but `resolveLanguage('.go')` returned +`undefined`, so Go pattern rules never fired. + +Add a test: `expect(resolveLanguage('.go')).toBe('go')` to prevent regression. + +### `packages/core/src/services/pattern-analysis-service.ts` + +Add `go: ALL_GO_QUERIES` to `QUERIES_BY_LANGUAGE` map. + +## Test fixture + +Create `packages/core/src/scanner/__fixtures__/go-callees.go`: + +```go +package main + +import ( + "fmt" + "os" + "strings" +) + +func processInput(input string) string { + trimmed := strings.TrimSpace(input) + fmt.Println("Processing:", trimmed) + return trimmed +} + +func main() { + result := processInput(os.Args[1]) + fmt.Println(result) + os.Exit(0) +} + +type Server struct { + host string +} + +func (s *Server) Start() error { + fmt.Println("Starting server on", s.host) + return nil +} + +func (s *Server) handleRequest(data string) { + processed := processInput(data) + fmt.Println("Handled:", processed) +} +``` + +## Tests + +Add to `packages/core/src/scanner/__tests__/go.test.ts`: + +| Test | What it verifies | +|------|-----------------| +| functions have callees | `processInput` calls `strings.TrimSpace`, `fmt.Println` | +| callee name is full selector | `calleeNames.toContain('fmt.Println')` not just `'Println'` | +| methods have callees | `Server.Start` calls `fmt.Println` | +| callee line numbers correct | Line numbers match fixture | +| callee deduplication | Same call on same line not duplicated | +| interfaces have no callees | Interface docs have undefined callees | +| Go pattern rules fire | `if err != nil`, `defer`, `go` patterns detected | +| resolveLanguage('.go') works | Bug fix regression test | + +## Commit + +``` +feat(core): add Go callee extraction and pattern rules + +Also fix: .go was missing from EXTENSION_TO_LANGUAGE — Go patterns never fired. +``` diff --git a/.claude/da-plans/core/phase-5-go-rust-support/5.2-rust-scanner.md b/.claude/da-plans/core/phase-5-go-rust-support/5.2-rust-scanner.md new file mode 100644 index 0000000..75d907d --- /dev/null +++ b/.claude/da-plans/core/phase-5-go-rust-support/5.2-rust-scanner.md @@ -0,0 +1,405 @@ +# Part 5.2: Rust Scanner with Full Extraction + +See [overview.md](overview.md) for architecture context. + +## Goal + +Build a complete Rust scanner using tree-sitter-rust WASM. Extract functions, structs, +enums, traits, methods (from impl blocks), imports, callees, and doc comments. + +## Step 0: Validate grammar node names + +**Before writing any scanner code**, run a throwaway validation test: + +```typescript +it('validates Rust grammar node names', async () => { + const code = ` + pub fn hello() { println!("hi"); } + struct Foo { x: i32 } + impl Foo { + fn bar(&self) { self.baz(); } + fn baz(&self) {} + } + trait MyTrait { fn do_thing(&self); } + use std::io; + `; + const tree = await parseCode(code, 'rust'); + // Log the tree to verify node names + function printTree(node, indent = 0) { + console.log(' '.repeat(indent) + node.type + (node.isNamed ? '' : ' [anonymous]')); + for (let i = 0; i < node.childCount; i++) printTree(node.child(i), indent + 2); + } + printTree(tree.rootNode); +}); +``` + +Expected to confirm: +- `function_item` for `fn` declarations +- `struct_item` for structs +- `impl_item` for impl blocks +- `trait_item` for traits +- `use_declaration` for imports +- `call_expression` for function calls (including `self.baz()`) +- `macro_invocation` for `println!` +- `visibility_modifier` for `pub` + +**Keep this test as a reference** — don't delete it after validation. + +## What changes + +### `packages/core/src/scanner/rust-queries.ts` (new) + +S-expression queries validated against tree-sitter-rust grammar: + +```typescript +export const RUST_QUERIES = { + functions: '(function_item name: (identifier) @name) @definition', + structs: '(struct_item name: (type_identifier) @name) @definition', + enums: '(enum_item name: (type_identifier) @name) @definition', + traits: '(trait_item name: (type_identifier) @name) @definition', + implMethods: `(impl_item + type: (_) @receiver + body: (declaration_list + (function_item name: (identifier) @name) @definition))`, + imports: '(use_declaration argument: (_) @path) @definition', + typeAliases: '(type_item name: (type_identifier) @name) @definition', +}; +``` + +**Note:** Query syntax must be validated by Step 0 before use. + +### `packages/core/src/scanner/rust.ts` (new) + +~400-500 lines. Follows `python.ts` pattern: + +``` +RustScanner implements Scanner + language = 'rust' + capabilities = { syntax: true, types: true, documentation: true } + + canHandle(filePath) → .rs extension + scan(files, repoRoot, logger, onProgress) + for each file: + parseCode(content, 'rust') + extractFunctions(tree) → Document[] + extractStructs(tree) → Document[] + extractEnums(tree) → Document[] + extractTraits(tree) → Document[] + extractMethods(tree) → Document[] (from impl blocks) + extractImports(tree) → Document[] + extractFromFile(filePath, repoRoot) → single-file extraction +``` + +Key implementation details: + +**Export detection:** +``` +pub fn foo() → exported: true +fn bar() → exported: false +pub(crate) → exported: true (visible within crate) +``` +Check for `visibility_modifier` child on `function_item` / `struct_item` / etc. + +**Method naming (impl blocks):** +```rust +impl Server { + fn handle_request(&self) { ... } +} +``` +Use `impl_item`'s `type` field to get the receiver type. This field holds the +concrete type in BOTH `impl Type` and `impl Trait for Type` forms — no positional +child walking needed. + +Document name: `"Server.handle_request"`. + +For `impl Trait for Type`: +```rust +impl Handler for Server { + fn handle(&self) { ... } +} +``` +Document name: `"Server.handle"` — `impl_item.type` field gives `Server` (the +concrete type), not `Handler` (the trait). + +**Generic type parameter stripping:** +```rust +impl Container { + pub fn show(&self) -> String { ... } +} +``` +The `type` field text is `Container`. Strip generic params: `.replace(/<.*>/, '')`. +Document name: `"Container.show"` not `"Container.show"`. +This mirrors the Go scanner's approach at `go.ts:434` which strips `[...]` for Go generics. + +**Doc comment extraction:** +```rust +/// This is a doc comment +/// for the function below +fn documented() { ... } +``` +tree-sitter-rust represents these as `line_comment` nodes with text starting `///`. +Walk preceding siblings of a definition node, collecting `line_comment` nodes whose +text starts with `///`. Strip the `/// ` prefix and join with newlines. + +**Known limitation:** Block doc comments (`/** */`) are not extracted in v1. They're +represented as `block_comment` in tree-sitter-rust. Track as future work in scratchpad. + +**Callee extraction:** +Same `walkCallNodes` pattern as Go/Python. tree-sitter-rust uses `call_expression` +for both bare function calls AND method calls (`obj.method()`). The method call case +has a `field_expression` as the function child: + +``` +(call_expression + function: (field_expression ← self.process_request + value: (self) + field: (field_identifier))) ← process_request + +(call_expression + function: (identifier)) ← local_function + +(call_expression + function: (scoped_identifier)) ← std::io::stdin +``` + +There is NO separate `method_call` node type — `call_expression` covers all cases. +Walking `call_expression` nodes (as Go and Python do) catches everything. + +**Macros are NOT callees.** `println!`, `vec!`, `format!` appear as `macro_invocation` +nodes, not `call_expression`. They are intentionally excluded from callee extraction. +This is a design decision, not a bug. Track "Rust macro callees" in scratchpad as +future work if users request it. + +**Generated file detection:** +Skip files matching: +- `target/` directory (build output) +- Files with `// Code generated` header + +**Async detection:** +```rust +async fn fetch() { ... } +``` +Check for `async` keyword child on `function_item` node (tree-sitter-rust exposes it +directly in the AST). More reliable than regex against full text. Step 0 validation +should confirm this child exists. Fallback to regex if tree-sitter doesn't expose it. + +**Signature extraction:** +Take the `fn` line (skip attribute lines like `#[derive(...)]`). + +### `packages/core/src/scanner/tree-sitter.ts` + +Add `'rust'` to `TreeSitterLanguage` union type. + +### `packages/core/src/scanner/index.ts` + +Import `RustScanner`, register in `createDefaultRegistry()`, export. + +### `packages/core/src/pattern-matcher/rules.ts` + +Add Rust pattern rules (type is `PatternMatchRule` with `{ id, category, query }` + `@match` capture): + +```typescript +export const RUST_ERROR_HANDLING_QUERIES: PatternMatchRule[] = [ + { id: 'rust-try-operator', category: 'error-handling', query: '(try_expression) @match' }, + { id: 'rust-match', category: 'error-handling', query: '(match_expression) @match' }, +]; + +export const RUST_UNSAFE_QUERIES: PatternMatchRule[] = [ + { id: 'rust-unsafe-block', category: 'unsafe', query: '(unsafe_block) @match' }, +]; + +export const RUST_TYPE_QUERIES: PatternMatchRule[] = [ + { id: 'rust-impl-block', category: 'types', query: '(impl_item) @match' }, + { id: 'rust-trait-def', category: 'types', query: '(trait_item) @match' }, +]; + +export const ALL_RUST_QUERIES = [ + ...RUST_ERROR_HANDLING_QUERIES, + ...RUST_UNSAFE_QUERIES, + ...RUST_TYPE_QUERIES, +]; +``` + +### `packages/core/src/services/pattern-analysis-service.ts` + +Add `rust: ALL_RUST_QUERIES` to `QUERIES_BY_LANGUAGE` map. + +### `packages/core/src/pattern-matcher/wasm-matcher.ts` + +Add `.rs: 'rust'` to `EXTENSION_TO_LANGUAGE` map. +Add `'rust'` to `supportedLanguages` set. + +## Test fixtures + +### `packages/core/src/scanner/__fixtures__/rust-simple.rs` + +```rust +use std::collections::HashMap; +use std::io::{self, Read}; + +/// A simple key-value store +pub struct Store { + data: HashMap, +} + +impl Store { + /// Create a new empty store + pub fn new() -> Self { + Store { data: HashMap::new() } + } + + /// Get a value by key + pub fn get(&self, key: &str) -> Option<&String> { + self.data.get(key) + } + + fn internal_cleanup(&mut self) { + self.data.clear(); + } +} + +/// Process input from stdin +pub fn process_input() -> io::Result { + let mut buffer = String::new(); + io::stdin().read_to_string(&mut buffer)?; + Ok(buffer) +} + +fn helper() -> bool { + true +} + +/// Only visible within the crate +pub(crate) fn crate_visible() -> bool { + helper() +} + +pub enum Status { + Active, + Inactive, + Error(String), +} + +pub trait Processor { + fn process(&self, input: &str) -> String; +} + +/// Async function for testing async detection +pub async fn fetch_data(url: &str) -> Result> { + Ok(url.to_string()) +} +``` + +### `packages/core/src/scanner/__fixtures__/rust-complex.rs` + +```rust +use std::fmt; + +/// Server handles HTTP requests +pub struct Server { + host: String, + port: u16, +} + +pub trait Handler { + fn handle(&self, request: &str) -> Result>; +} + +impl Handler for Server { + fn handle(&self, request: &str) -> Result> { + let processed = self.process_request(request)?; + Ok(processed) + } +} + +impl Server { + pub fn new(host: &str, port: u16) -> Self { + Server { host: host.to_string(), port } + } + + fn process_request(&self, data: &str) -> Result> { + let trimmed = data.trim(); + let result = format!("{}:{} - {}", self.host, self.port, trimmed); + Ok(result) + } +} + +impl fmt::Display for Server { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "Server({}:{})", self.host, self.port) + } +} + +/// Generic container — tests type parameter stripping +pub struct Container { + value: T, +} + +impl Container { + pub fn show(&self) -> String { + self.value.to_string() + } +} + +fn transform(input: &str) -> String { + input.to_uppercase() +} + +/// Tests callee extraction inside closures +pub fn process_items(items: Vec) -> Vec { + items.iter().map(|x| transform(x)).collect() +} + +/// Tests that field access is NOT a callee +pub fn read_server_host(s: &Server) -> String { + let host = s.host.clone(); // field access — NOT a callee + host.to_uppercase() // method call — IS a callee +} +``` + +### `packages/core/src/scanner/__fixtures__/rust-malformed.rs` + +```rust +fn broken( { + // This file intentionally has a syntax error + let x = +} +``` + +## Tests + +Create `packages/core/src/scanner/__tests__/rust.test.ts`: + +| Test | What it verifies | +|------|-----------------| +| Step 0: grammar validation | Node names match expectations (keep as reference) | +| canHandle('.rs') returns true | Scanner claims Rust files | +| canHandle('.go') returns false | Doesn't claim other languages | +| extract functions | `process_input`, `helper` extracted with correct lines | +| pub vs non-pub | `process_input` exported, `helper` not | +| pub(crate) visibility | `crate_visible` → exported: true | +| extract structs | `Store`, `Server`, `Container` extracted | +| extract enums | `Status` extracted | +| extract traits | `Processor`, `Handler` extracted | +| extract methods from impl | `Store.new`, `Store.get`, `Store.internal_cleanup` | +| method naming includes type | Name is `Store.new` not just `new` | +| generic impl type param stripping | `Container.show` not `Container.show` | +| impl Trait for Type methods | `Server.handle` uses concrete type, `Server.fmt` for Display | +| extract imports | `std::collections::HashMap`, `std::io` captured | +| callee extraction | `Server.handle` calls `self.process_request`. `process_request` calls `data.trim`. | +| macros NOT in callees | `format!`, `println!` not in callee list | +| callees inside closures | `transform` captured from `.map(\|x\| transform(x))` | +| field access NOT a callee | `s.host` (field) excluded, `host.to_uppercase()` (method) included | +| doc comments | `Store` has "A simple key-value store" docstring | +| method doc comments | `Store.new` has "Create a new empty store" | +| async detection via AST | `fetch_data` detected as async via keyword child node | +| signatures | Function signatures include `fn` keyword and params | +| generated file skipping | Files in `target/` skipped | +| isTestFile for tests/ dir | `tests/integration.rs` flagged as test | +| malformed file doesn't crash | Scanner returns empty documents for `rust-malformed.rs` | + +## Commit + +``` +feat(core): add Rust scanner with full extraction +``` diff --git a/.claude/da-plans/core/phase-5-go-rust-support/5.3-wiring-and-verification.md b/.claude/da-plans/core/phase-5-go-rust-support/5.3-wiring-and-verification.md new file mode 100644 index 0000000..b9a0939 --- /dev/null +++ b/.claude/da-plans/core/phase-5-go-rust-support/5.3-wiring-and-verification.md @@ -0,0 +1,143 @@ +# Part 5.3: Pipeline Wiring, Docs, and Local Verification + +See [overview.md](overview.md) for architecture context. + +## Goal + +Wire Go patterns and Rust into the full pipeline. Update docs. Run local verification +against real open source repos. + +## What changes + +### `packages/dev-agent/scripts/copy-wasm.js` + +Add `'rust'` to `SUPPORTED_LANGUAGES` array so `tree-sitter-rust.wasm` is copied to +the dist bundle. + +### `packages/core/src/utils/test-utils.ts` + +Add Rust entries: + +```typescript +// In TEST_PATTERNS map +rs: (f: string) => f.includes('/tests/') || path.basename(f).endsWith('_test.rs'), + +// In TEST_PATH_GENERATORS map +rs: (base: string) => { + const name = path.basename(base, '.rs'); + const dir = path.dirname(base); + return [ + path.join(dir, '..', 'tests', `${name}.rs`), + path.join(dir, `${name}_test.rs`), + ]; +}, +``` + +### `CLAUDE.md` + +Update scanner description to include Rust: +``` +core/ # Scanner (ts-morph, tree-sitter for Python/Go/Rust), vector storage (Antfly), services +``` + +### `website/content/index.mdx` + +Add Go and Rust to the Multi-Language feature list. + +### `website/content/updates/index.mdx` + +Add release entry for v0.12.0. + +### `website/content/latest-version.ts` + +Update to v0.12.0. + +### Changeset + +```yaml +--- +'@prosdevlab/dev-agent': minor +--- + +Go callee extraction and Rust language support + +- Go: callee extraction for functions and methods — dev_refs now traces Go call chains +- Go: pattern rules for error handling (if err != nil), goroutines, defer, channels +- Rust: full scanner — functions, structs, enums, traits, impl methods, imports, callees +- Rust: pattern rules for Result/Option handling, unsafe blocks, match expressions +- All MCP tools (dev_search, dev_refs, dev_map, dev_patterns) work with Go callees and Rust +``` + +## Local verification + +### Go — cli/cli + +```bash +cd /tmp +git clone --depth 1 https://github.com/cli/cli.git gh-cli +cd gh-cli + +# Index with local build +node /path/to/dev-agent/packages/dev-agent/dist/cli.js index + +# Verify +node /path/to/dev-agent/packages/dev-agent/dist/cli.js map --depth 2 +node /path/to/dev-agent/packages/dev-agent/dist/cli.js refs "NewCmdRoot" +node /path/to/dev-agent/packages/dev-agent/dist/cli.js search "authentication" +``` + +Expected: +- Index completes without crash +- `dev map` shows Go files in hot paths +- `dev refs` shows callers/callees for Go functions +- `dev search` finds Go code by meaning + +### Rust — BurntSushi/ripgrep + +```bash +cd /tmp +git clone --depth 1 https://github.com/BurntSushi/ripgrep.git +cd ripgrep + +# Index with local build +node /path/to/dev-agent/packages/dev-agent/dist/cli.js index + +# Verify +node /path/to/dev-agent/packages/dev-agent/dist/cli.js map --depth 2 +node /path/to/dev-agent/packages/dev-agent/dist/cli.js refs "main" +node /path/to/dev-agent/packages/dev-agent/dist/cli.js search "grep pattern matching" +``` + +Expected: +- Index completes without crash +- `dev map` shows Rust files in hot paths +- `dev refs` shows callers/callees for Rust functions +- `dev search` finds Rust code by meaning + +### What to look for + +- **Crash on parse:** Large or complex files that tree-sitter can't handle. Log the file + and skip gracefully. +- **Missing callees:** If `dev refs` shows a function with 0 callees that clearly calls + other functions, the callee extraction has a bug. +- **Wrong method names:** If impl methods show as bare names (`handle`) instead of + qualified names (`Server.handle`), the impl block association is broken. +- **Hot paths empty:** If `dev map` shows no hot paths, callees aren't being indexed + correctly. + +Track any issues found in `.claude/scratchpad.md`. + +## Tests + +| Test | What it verifies | +|------|-----------------| +| copy-wasm includes rust WASM | `tree-sitter-rust.wasm` in dist | +| test-utils Rust patterns | `isTestFile` detects `tests/` and `_test.rs` | +| test-utils Rust findTestFile | Generates correct test paths | +| Changeset valid | `pnpm changeset status` passes | + +## Commit + +``` +docs: update language lists and add changelog for Phase 5 +``` diff --git a/.claude/da-plans/core/phase-5-go-rust-support/overview.md b/.claude/da-plans/core/phase-5-go-rust-support/overview.md new file mode 100644 index 0000000..5afb611 --- /dev/null +++ b/.claude/da-plans/core/phase-5-go-rust-support/overview.md @@ -0,0 +1,275 @@ +# Phase 5: Go Callee Extraction + Rust Language Support + +**Status:** Draft + +## Context + +Phase 4 added Python support. The scanner pipeline now handles TypeScript/JavaScript +(ts-morph), Go (tree-sitter), Python (tree-sitter), and Markdown (remark). + +Go has a working scanner but **no callee extraction** — functions and methods are indexed +but the call graph is empty for Go files. This means `dev_refs` can't trace Go call chains +and `dev_map` hot paths miss Go dependencies. + +Rust has **no scanner at all**. `tree-sitter-rust.wasm` is already bundled in +`tree-sitter-wasms@0.1.13` — zero new deps needed. The registry already maps `.rs` to +`rust` in `getExtensionsForLanguage`. The wiring is ready, just needs the scanner. + +### What exists + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Language │ Scanner │ Callees │ Patterns │ Test Detection │ +├─────────────┼─────────┼─────────┼──────────┼─────────────────┤ +│ TypeScript │ ts-morph │ ✓ │ ✓ │ .test., .spec. │ +│ JavaScript │ ts-morph │ ✓ │ ✓ │ .test., .spec. │ +│ Python │ tree-sitter │ ✓ │ ✓ │ test_*, *_test │ +│ Go │ tree-sitter │ ✗ │ ✗ │ _test.go │ +│ Rust │ none │ ✗ │ ✗ │ none │ +│ Markdown │ remark │ n/a │ n/a │ n/a │ +└─────────────┴─────────────┴─────────┴──────────┴─────────────────┘ +``` + +### What this phase delivers + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Language │ Scanner │ Callees │ Patterns │ Test Detection │ +├─────────────┼─────────┼─────────┼──────────┼─────────────────┤ +│ Go │ tree-sitter │ ✓ │ ✓ │ _test.go │ +│ Rust │ tree-sitter │ ✓ │ ✓ │ tests/, _test.rs │ +└─────────────┴─────────────┴─────────┴──────────┴─────────────────┘ +``` + +After Phase 5, all MCP tools (`dev_search`, `dev_refs`, `dev_map`, `dev_patterns`) work +with Go and Rust automatically. + +--- + +## Proposed architecture + +### Go callee extraction + +``` +┌──────────────────────────────────────────────────┐ +│ go.ts (existing) │ +│ │ +│ extractFunctions() ──► now includes callees │ +│ extractMethods() ──► now includes callees │ +│ │ │ +│ ▼ │ +│ walkCallNodes(node) │ +│ recursive walk, same pattern as python.ts │ +│ node.type === 'call_expression' │ +│ callee text from 'function' child │ +│ line from node.startPosition.row │ +└──────────────────────────────────────────────────┘ + +Go call_expression structure: + (call_expression + function: (selector_expression ← fmt.Println + operand: (identifier) ← fmt + field: (field_identifier)) ← Println + arguments: (argument_list ...)) + + (call_expression + function: (identifier) ← localFunc + arguments: (argument_list ...)) +``` + +### Rust scanner + +``` +┌──────────────────────────────────────────────────┐ +│ rust.ts (new) │ +│ │ +│ canHandle('.rs') │ +│ scan(files, repoRoot) │ +│ ├── extractFunctions() ← function_item │ +│ ├── extractStructs() ← struct_item │ +│ ├── extractEnums() ← enum_item │ +│ ├── extractTraits() ← trait_item │ +│ ├── extractMethods() ← impl_item > fn │ +│ ├── extractImports() ← use_declaration │ +│ └── extractCallees() ← call_expression │ +│ │ +│ rust-queries.ts │ +│ S-expression queries for each node type │ +└──────────────────────────────────────────────────┘ + +Rust impl block association: + (impl_item + type: (type_identifier) @receiver ← Server + body: (declaration_list + (function_item + name: (identifier) @name ← handle_request + ...))) + + → document name: "Server.handle_request" +``` + +### Pipeline wiring (both languages) + +``` +Touch points: + tree-sitter.ts → add 'rust' to TreeSitterLanguage union + scanner/index.ts → import + register RustScanner + wasm-matcher.ts → add '.go': 'go' and '.rs': 'rust' + copy-wasm.js → add 'rust' to SUPPORTED_LANGUAGES + test-utils.ts → add Rust entries to TEST_PATTERNS + TEST_PATH_GENERATORS + pattern-analysis.ts → add 'go' and 'rust' to QUERIES_BY_LANGUAGE map + rules.ts → add ALL_GO_QUERIES and ALL_RUST_QUERIES +``` + +--- + +## Parts + +| Part | Description | Risk | +|------|-------------|------| +| [5.1](./5.1-go-callees.md) | Go callee extraction + Go pattern rules + fixture + tests | Low — existing scanner, additive | +| [5.2](./5.2-rust-scanner.md) | Rust scanner + queries + pattern rules + fixtures + tests | Medium — new scanner, impl block logic | +| [5.3](./5.3-wiring-and-verification.md) | Pipeline wiring, docs, local real-repo verification | Low — mechanical | + +--- + +## Decisions + +| Decision | Rationale | Alternatives | +|----------|-----------|-------------| +| Validate grammar before building | Step 0 parse test confirms node names. Prevents building on wrong assumptions. | Trust docs: risky, grammar may differ from documentation | +| Recursive AST walk for callees (not query) | Matches Python pattern, handles nested calls naturally | tree-sitter query: harder to capture all nesting levels | +| `call_expression` for all Rust calls | tree-sitter-rust uses `call_expression` for bare calls AND method calls (via `field_expression` child). No separate `method_call` node. | N/A — grammar doesn't offer alternatives | +| Full selector text for Go callees | `"fmt.Println"` not `"Println"`. Matches TS scanner behavior, gives agents package context. | Short name only: loses context | +| `impl_item.type` field for method naming | Use the `type` field on `impl_item` — holds the concrete type in both `impl Type` and `impl Trait for Type`. Strip generic params (``) for document name. | Positional child walking: fragile, depends on child order | +| Strip generic type params from impl | `Container.show` → `Container.show`. Use `.replace(/<.*>/, '')` (mirrors Go scanner's `replace(/\[.*\]/, '')` for type params). | Keep full generic name: hurts search matching | +| Skip macro callees (`println!`, `vec!`) | Macros are `macro_invocation` nodes, not `call_expression`. Different semantics. Explicit negative test locks in decision. | Include: would need `macro_invocation` node handling | +| `pub` keyword for export detection | Rust's visibility is explicit. `pub fn` = exported, `fn` = private. All `visibility_modifier` variants (`pub`, `pub(crate)`, `pub(super)`) → exported: true. | N/A — pragmatic v1 | +| Async via AST child node, not regex | tree-sitter-rust exposes `async` as a keyword child on `function_item`. More reliable than regex against full text. | Regex: fragile if text includes "async" in comments | +| Doc comments via `///` prefix only | tree-sitter-rust exposes as `line_comment` nodes. `//!` inner docs and `/** */` block docs deferred to v2. | Include all doc forms: more complex, diminishing returns | +| Test detection: `tests/` dir + `_test.rs` | Covers integration tests and the common convention. Inline `#[cfg(test)]` deferred. | Parse `#[cfg(test)]`: would flag functions inside test modules — more complex | +| Self-contained fixtures, real-repo local test | Unit tests with fixtures for CI. Clone real repos for manual verification. | Real-repo in CI: too slow, too flaky | +| Type is `PatternMatchRule` with `{ id, category, query }` | Matches existing interface and all rules in `rules.ts`. All queries need `@match` capture alias. | N/A — compiler enforces | + +--- + +## Risk register + +| Risk | Likelihood | Impact | Mitigation | +|------|-----------|--------|------------| +| tree-sitter-rust grammar node names differ from docs | Medium | Low | Step 0 validation test confirms names before building. Keep test as reference. | +| `impl` block association misses trait impls | Medium | Low | Handle both `impl Type` and `impl Trait for Type` in v1. Test both forms. | +| Generic impl names include type params | Medium | Low | Strip `<...>` from type name. Explicit test: `Container.show` not `Container.show`. | +| Go callee extraction too noisy (stdlib calls) | Low | Low | Callees already include all calls in TS/Python — consistent | +| Rust WASM grammar large or slow | Low | Low | Python WASM is 476KB, Go is similar. Lazy-loaded per file. | +| Real-repo test finds edge cases we can't handle | Medium | Low | Track in scratchpad as known limitations. Don't block on edge cases. | +| Malformed Rust files crash scanner | Low | Medium | Explicit test with `rust-malformed.rs` fixture. Scanner returns empty, no crash. | +| `.go` extension missing from pattern matcher | Already broken | Low | Fix in Part 5.1 with regression test for `resolveLanguage('.go')` | +| Block doc comments (`/** */`) missed | Low | Low | Track in scratchpad. `///` covers 95%+ of real Rust code. | + +--- + +## Test strategy + +| Test | Priority | What it verifies | +|------|----------|-----------------| +| Go: callee extraction from functions | P0 | `walkCallNodes` returns correct callees | +| Go: callee name format (full selector) | P0 | `"fmt.Println"` not `"Println"` | +| Go: callee extraction from methods | P0 | Methods have callees with correct names/lines | +| Go: callee deduplication | P1 | Same call on same line not duplicated | +| Go: no callees for interfaces/types | P1 | Non-callable types return no callees | +| Go: pattern rules fire | P1 | Error handling, goroutine, defer patterns detected | +| Go: resolveLanguage('.go') | P0 | Bug fix regression test | +| Rust: Step 0 grammar validation | P0 | Confirm node names before building scanner | +| Rust: canHandle('.rs') | P0 | Scanner claims Rust files | +| Rust: extract functions | P0 | Free functions with name, line, signature, exported | +| Rust: extract methods from impl | P0 | `Type.method` naming, correct line numbers | +| Rust: impl Trait for Type | P0 | `Server.handle` uses concrete type, `Server.fmt` for Display | +| Rust: extract structs/enums/traits | P0 | All type definitions captured | +| Rust: extract imports | P0 | `use` declarations captured | +| Rust: callee extraction | P0 | Function calls and method calls in callees | +| Rust: macros NOT in callees | P0 | `println!`, `format!` excluded from callee list | +| Rust: doc comment extraction | P1 | `///` comments extracted as docstrings | +| Rust: pub vs non-pub | P1 | `pub fn` → exported: true, `fn` → exported: false | +| Rust: pattern rules fire | P1 | Result/Option, unsafe, match patterns detected | +| Rust: isTestFile for tests/ dir | P1 | Files in tests/ directory flagged | +| Rust: malformed file resilience | P0 | Scanner returns empty documents, no crash | +| Rust: generic impl type param stripping | P0 | `Container.show` not `Container.show` | +| Rust: `pub(crate)` visibility | P1 | `pub(crate) fn` → exported: true | +| Rust: callees inside closures | P1 | `transform` in `.map(\|x\| transform(x))` captured | +| Rust: async via AST child | P1 | `async fn` detected via keyword child, not regex | +| Rust: `self.field` NOT a callee | P1 | Field access excluded, only `self.method()` calls | +| Local: `dev index` on cli/cli (Go) | P0 | Indexes without crash, callees populated | +| Local: `dev index` on ripgrep (Rust) | P0 | Indexes without crash, functions/structs captured | +| Local: `dev map` on both repos | P1 | Hot paths show real Go/Rust files | +| Local: `dev refs` on both repos | P1 | Callers/callees work for Go/Rust symbols | + +--- + +## Verification checklist + +### Automated (CI) +- [ ] Go callee tests pass +- [ ] Rust scanner tests pass (all extraction types) +- [ ] Pattern rules tests pass for Go and Rust +- [ ] `pnpm build && pnpm test` passes +- [ ] `pnpm typecheck` clean +- [ ] `pnpm lint` clean + +### Manual (local, real repos) + +**Go — clone `cli/cli`:** +```bash +cd /tmp && git clone --depth 1 https://github.com/cli/cli.git gh-cli +cd gh-cli && dev index +dev map --depth 2 # Should show Go hot paths +dev refs "NewCmdRoot" # Should find callers/callees +dev search "authentication" # Should find Go auth code +``` + +**Rust — clone `BurntSushi/ripgrep`:** +```bash +cd /tmp && git clone --depth 1 https://github.com/BurntSushi/ripgrep.git +cd ripgrep && dev index +dev map --depth 2 # Should show Rust hot paths +dev refs "main" # Should find callers/callees +dev search "grep pattern" # Should find search code +``` + +- [ ] Go repo indexes without crash +- [ ] Go repo has callees in `dev refs` output +- [ ] Go repo `dev map` shows hot paths (not all empty) +- [ ] Rust repo indexes without crash +- [ ] Rust repo `dev refs` shows callers/callees +- [ ] Rust repo `dev map` shows hot paths + +--- + +## Commit strategy + +``` +1. feat(core): add Go callee extraction and pattern rules +2. feat(core): add Rust scanner with full extraction +3. feat(core): wire Go and Rust into pattern matcher and pipeline +4. docs: update language lists and add changelog +``` + +--- + +## Known Limitations (v1) + +Intentional scope limits, documented for transparency. Not bugs — tracked for v2. + +1. **`//!` inner doc comments not extracted.** Module-level docs (`//!` at top of `lib.rs`) are skipped. Only `///` outer doc comments (item-level) are captured. +2. **`/** */` block doc comments not extracted.** Rare in practice — `///` covers 95%+ of real Rust code. +3. **Trait default method bodies not extracted.** `trait Foo { fn bar(&self) { ... } }` — the default implementation body and its callees are not captured. Only the trait definition itself is indexed. +4. **`#[cfg(test)]` inline test modules not detected.** `isTestFile` only checks path patterns (`tests/`, `_test.rs`). Functions inside `#[cfg(test)] mod tests { ... }` are not flagged as test code. +5. **Macro invocations not in callees.** `println!`, `vec!`, `format!` etc. are intentionally excluded. They're `macro_invocation` nodes, not function calls. + +--- + +## Dependencies + +- Phase 4 (Python support) — merged +- `tree-sitter-wasms@0.1.13` — already bundled, includes `tree-sitter-rust.wasm` +- No new npm dependencies diff --git a/.claude/da-plans/mcp/phase-2-composite-tools/2.1-review-adapter.md b/.claude/da-plans/mcp/phase-2-composite-tools/2.1-review-adapter.md new file mode 100644 index 0000000..a91c578 --- /dev/null +++ b/.claude/da-plans/mcp/phase-2-composite-tools/2.1-review-adapter.md @@ -0,0 +1,218 @@ +# Part 2.1: Review Analysis Service + dev_review MCP Adapter + +See [overview.md](overview.md) for architecture context. + +## Goal + +Build the shared review analysis service and the `dev_review` MCP adapter. +The service contains pure analysis logic. The adapter is a thin MCP wrapper. + +## What changes + +### `packages/core/src/services/review-analysis.ts` (new) + +~200-250 lines. Pure functions, no MCP or CLI dependencies. + +```typescript +export interface ReviewAnalysis { + target: string[]; + impact: ImpactAnalysis; + patterns?: PatternComparison; + structure?: StructuralContext; + similar?: SimilarCode[]; + warnings: string[]; // partial failure notes +} + +export interface ImpactAnalysis { + callers: Array<{ name: string; file: string; line: number }>; + callees: Array<{ name: string; file: string; line: number }>; + hotPathRank?: number; + hotPathScore?: number; + affectedPackages: string[]; + dependentFileCount: number; +} + +export interface PatternComparison { + conventions: Array<{ + pattern: string; // e.g., "error handling" + fileApproach: string; // what this file does + codebaseNorm: string; // what most files do + consistent: boolean; + }>; + similarFiles: Array<{ file: string; overlapScore: number }>; +} + +export interface StructuralContext { + subsystem: string; + subsystemFileCount: number; + totalComponents: number; + componentCount: number; + changeFrequency?: { last30Days: number; last90Days: number }; +} + +export async function analyzeForReview( + target: string[], + indexer: RepositoryIndexer, + searchService: SearchService, + graphPath?: string, + options?: { depth: 'quick' | 'standard' | 'deep' } +): Promise { + const depth = options?.depth ?? 'standard'; + const warnings: string[] = []; + + // Always run impact analysis + let impact: ImpactAnalysis; + try { + impact = await analyzeImpact(target, indexer, graphPath); + } catch (err) { + warnings.push('Impact analysis failed'); + impact = { callers: [], callees: [], affectedPackages: [], dependentFileCount: 0 }; + } + + if (depth === 'quick') { + return { target, impact, warnings }; + } + + // Standard: add patterns + structure in parallel + const [patterns, structure] = await Promise.all([ + comparePatterns(target, searchService).catch(() => { + warnings.push('Pattern comparison unavailable'); + return undefined; + }), + getStructuralContext(target, indexer, graphPath).catch(() => { + warnings.push('Structural context unavailable'); + return undefined; + }), + ]); + + let similar: SimilarCode[] | undefined; + if (depth === 'deep') { + similar = await findSimilarCode(target, searchService).catch(() => { + warnings.push('Similar code search unavailable'); + return undefined; + }); + } + + return { target, impact, patterns, structure, similar, warnings }; +} +``` + +Each specialist function (`analyzeImpact`, `comparePatterns`, etc.) is a +separate exported function — independently testable. + +### `packages/mcp-server/src/adapters/built-in/review-adapter.ts` (new) + +~150-200 lines. Thin wrapper around `analyzeForReview`. + +```typescript +export class ReviewAdapter extends ToolAdapter { + getToolDefinition(): ToolDefinition { + return { + name: 'dev_review', + description: + 'Prepare review context for code changes. Use BEFORE reviewing a PR, ' + + 'commit, or file change. Returns impact analysis (what depends on this ' + + 'code), pattern comparison (does it follow conventions), and structural ' + + 'context (where it fits). Replaces calling dev_refs + dev_patterns + ' + + 'dev_map separately.', + inputSchema: { ... }, + }; + } + + async execute(args, context): Promise { + const files = await this.resolveTarget(args.target); + const analysis = await analyzeForReview( + files, this.indexer, this.searchService, + this.graphPath, { depth: args.depth } + ); + return { + success: true, + data: formatReviewMarkdown(analysis, args.focus), + metadata: { tokens: estimateTokens(analysis, args.depth), ... }, + }; + } +} +``` + +**Tool description is key.** Must clearly signal: +- WHEN to use: "before reviewing a PR, commit, or file change" +- WHAT it replaces: "replaces calling dev_refs + dev_patterns + dev_map separately" +- HOW it differs from dev_patterns: dev_patterns = one file's patterns. dev_review = full change context. + +### `packages/mcp-server/src/schemas/index.ts` + +Add `ReviewArgsSchema`: + +```typescript +export const ReviewArgsSchema = z.object({ + target: z.string().min(1, 'Target must be a non-empty path or git range'), + focus: z.enum(['security', 'quality', 'performance', 'all']).default('all'), + depth: z.enum(['quick', 'standard', 'deep']).default('standard'), +}).strict(); +``` + +### Target resolution + +```typescript +async function resolveTarget(target: string): Promise { + // Git diff range: starts with HEAD, contains .., or looks like a range + if (target.startsWith('HEAD') || target.includes('..')) { + const result = execSync(`git diff --name-only ${target}`, { encoding: 'utf-8' }); + const files = result.trim().split('\n').filter(Boolean); + if (files.length === 0) throw new Error(`No files changed in range: ${target}`); + return files; + } + + // File or directory + const stat = await fs.stat(target); + if (stat.isDirectory()) { + // Get indexed files in this directory + return globby([`${target}/**`], { gitignore: true }); + } + + return [target]; +} +``` + +Edge cases: +- Invalid git range → error message: "Invalid git range: {target}" +- Nonexistent file → error message: "File not found: {target}" +- Empty diff → error message: "No files changed in range: {target}" +- Binary files in diff → filter to indexable extensions only + +## Tests + +### Service tests (packages/core) + +| Test | What it verifies | +|------|-----------------| +| analyzeImpact returns callers/callees | refs queried for target files | +| analyzeImpact returns hot path rank | graph loaded, PageRank computed | +| analyzeImpact returns affected packages | cross-package deps identified | +| comparePatterns returns conventions | pattern analysis for target | +| comparePatterns returns similar files | search finds similar code | +| getStructuralContext returns subsystem | map queried for target location | +| analyzeForReview quick depth | only impact, no patterns/structure | +| analyzeForReview standard depth | impact + patterns + structure | +| analyzeForReview deep depth | includes similar code | +| analyzeForReview partial failure | one specialist fails → partial report + warning | +| analyzeForReview all fail | returns empty sections + warnings, no crash | + +### Adapter tests (packages/mcp-server) + +| Test | What it verifies | +|------|-----------------| +| ReviewAdapter returns formatted markdown | service output → markdown | +| ReviewAdapter file target | resolveTarget for file path | +| ReviewAdapter directory target | resolveTarget for directory | +| ReviewAdapter git diff target | resolveTarget for HEAD~1 | +| ReviewAdapter invalid target | graceful error | +| ReviewAdapter empty diff | graceful error | +| ReviewAdapter token estimation | estimateTokens by depth | + +## Commit + +``` +feat(core): add review analysis service +feat(mcp): add dev_review composite adapter +``` diff --git a/.claude/da-plans/mcp/phase-2-composite-tools/2.2-research-adapter.md b/.claude/da-plans/mcp/phase-2-composite-tools/2.2-research-adapter.md new file mode 100644 index 0000000..f8611d1 --- /dev/null +++ b/.claude/da-plans/mcp/phase-2-composite-tools/2.2-research-adapter.md @@ -0,0 +1,238 @@ +# Part 2.2: Research Analysis Service + dev_research MCP Adapter + +See [overview.md](overview.md) for architecture context. + +## Goal + +Build the shared research analysis service and the `dev_research` MCP adapter. +The service gathers and enriches internal codebase knowledge. The AI assistant +combines this with its own external research to produce synthesis. + +## The value proposition + +An AI assistant calling `dev_search` gets search results. Then it decides which +results to trace with `dev_refs`, which to analyze with `dev_patterns`. It spends +tokens on this decision-making, and it may miss cross-package dependencies or +pattern inconsistencies. + +`dev_research` always: +- Traces call graphs for the top results (the AI might skip this) +- Checks pattern consistency across all relevant files (the AI might only check one) +- Maps the architectural context (the AI might not think to ask) +- Returns it all in one structured report + +The AI assistant is better at *judgment*. We're better at *thoroughness*. + +## What changes + +### `packages/core/src/services/research-analysis.ts` (new) + +~200-250 lines. + +```typescript +export interface ResearchAnalysis { + query: string; + relevantCode: Array<{ + name: string; + file: string; + line: number; + type: string; + signature?: string; + snippet?: string; + score: number; + }>; + callGraphs?: Array<{ + file: string; + callers: Array<{ name: string; file: string }>; + callees: Array<{ name: string; file: string }>; + }>; + patterns?: Array<{ + file: string; + conventions: Array<{ pattern: string; description: string }>; + }>; + architecture?: { + subsystems: Array<{ name: string; fileCount: number }>; + hotPaths: Array<{ file: string; rank: number; refs: number }>; + packagesInvolved: string[]; + }; + testCoverage?: Array<{ + sourceFile: string; + testFile?: string; + testCount?: number; + }>; + warnings: string[]; +} + +export async function analyzeForResearch( + query: string, + indexer: RepositoryIndexer, + searchService: SearchService, + graphPath?: string, + options?: { scope: string; depth: string } +): Promise { + const depth = options?.depth ?? 'standard'; + const scope = options?.scope ?? 'implementation'; + const warnings: string[] = []; + + // Step 1: Semantic search + const limit = depth === 'quick' ? 5 : depth === 'standard' ? 10 : 20; + const searchResults = await searchService.search(query, { limit }); + + if (searchResults.length === 0) { + return { query, relevantCode: [], warnings: ['No code found'] }; + } + + const relevantCode = formatSearchResults(searchResults); + const topFiles = extractUniqueFiles(searchResults, depth === 'quick' ? 3 : 5); + + // Step 2: Enrich ALL results — scope controls emphasis, not data collection + const [callGraphs, patterns, architecture] = await Promise.all([ + traceCallGraphs(topFiles, indexer, graphPath).catch(() => { + warnings.push('Call graph tracing unavailable'); + return undefined; + }), + analyzeFilePatterns(topFiles, searchService).catch(() => { + warnings.push('Pattern analysis unavailable'); + return undefined; + }), + getArchitecturalContext(topFiles, indexer, graphPath).catch(() => { + warnings.push('Architectural context unavailable'); + return undefined; + }), + ]); + + // Step 3: Test coverage (deep only) + let testCoverage; + if (depth === 'deep') { + testCoverage = await findTestCoverage(topFiles).catch(() => { + warnings.push('Test coverage analysis unavailable'); + return undefined; + }); + } + + return { + query, relevantCode, callGraphs, patterns, + architecture, testCoverage, warnings, + }; +} +``` + +### Scope behavior (emphasis, not exclusion) + +All enrichments run regardless of scope. Scope controls how the output is +formatted and which sections are highlighted: + +| Scope | Emphasis | Sections highlighted | +|-------|----------|---------------------| +| `architecture` | Structure and dependencies | Architecture + call graphs first | +| `implementation` | Code details and patterns | Relevant code + patterns first | +| `usage` | How the concept is consumed | Call graphs (callers) + architecture first | + +This avoids the counterintuitive "usage skips call graphs" problem from the +first draft. + +### `packages/mcp-server/src/adapters/built-in/research-adapter.ts` (new) + +~150-200 lines. Thin wrapper. + +```typescript +export class ResearchAdapter extends ToolAdapter { + getToolDefinition(): ToolDefinition { + return { + name: 'dev_research', + description: + 'Research a topic across the codebase. Finds relevant code, traces call ' + + 'graphs, compares patterns, and maps the architecture around a concept. ' + + 'Use for understanding HOW something works in this codebase, not for ' + + 'finding a specific symbol (use dev_refs for that). The AI assistant ' + + 'should combine this internal context with its own external research ' + + '(web search, docs) for the complete picture.', + inputSchema: { ... }, + }; + } +} +``` + +**Tool description signals:** +- WHEN: "for understanding HOW something works" +- NOT WHEN: "not for finding a specific symbol (use dev_refs)" +- HOW TO USE: "combine this internal context with external research" + +This explicitly tells the AI assistant that `dev_research` gives the internal half +and the assistant should add the external half. + +### `packages/mcp-server/src/schemas/index.ts` + +```typescript +export const ResearchArgsSchema = z.object({ + query: z.string().min(1, 'Query must be a non-empty string'), + scope: z.enum(['architecture', 'implementation', 'usage']).default('implementation'), + depth: z.enum(['quick', 'standard', 'deep']).default('standard'), +}).strict(); +``` + +## Output format + +Scope controls section ordering and emphasis, not data inclusion: + +```markdown +## Research: {query} + +### Relevant Code (ranked) +1. **{name}** ({type}) — {file}:{line} + `{signature}` + > {snippet preview — first 2 lines} + +### Call Graph +**{file}:** + Callers: {caller1}, {caller2} + Callees: {callee1}, {callee2} + +### Patterns +- **Error handling:** {description across files} +- **Naming:** {conventions observed} +- **Types:** {shared interfaces and types} + +### Architecture +- **Subsystem:** {name} ({count} files) +- **Packages:** {list} +- **Hot paths:** {files with high PageRank involvement} + +### Test Coverage (deep only) +- {source} → {test file} ({count} tests) +- {source} → no test file found +``` + +## Tests + +### Service tests (packages/core) + +| Test | What it verifies | +|------|-----------------| +| Search returns ranked results | semantic search works | +| Call graphs traced for top files | refs queried per file | +| Patterns analyzed for top files | pattern analysis works | +| Architecture context returned | map + graph queried | +| Test coverage found (deep) | test file detection works | +| Empty search returns helpful message | graceful empty state | +| Partial failure → partial report | one enrichment fails → warning, not crash | +| Quick depth limits results | 5 results, 3 files enriched | +| Deep depth expands results | 20 results, 5 files, test coverage | +| Scope changes section ordering | architecture scope → architecture first | + +### Adapter tests (packages/mcp-server) + +| Test | What it verifies | +|------|-----------------| +| ResearchAdapter returns formatted markdown | service output → markdown | +| ResearchAdapter scope parameter | all three scopes produce different ordering | +| ResearchAdapter depth parameter | quick < standard < deep output | +| ResearchAdapter empty results | helpful message | +| ResearchAdapter token estimation | estimateTokens by depth | + +## Commit + +``` +feat(core): add research analysis service +feat(mcp): add dev_research composite adapter +``` diff --git a/.claude/da-plans/mcp/phase-2-composite-tools/2.3-cli-commands.md b/.claude/da-plans/mcp/phase-2-composite-tools/2.3-cli-commands.md new file mode 100644 index 0000000..4c4b18c --- /dev/null +++ b/.claude/da-plans/mcp/phase-2-composite-tools/2.3-cli-commands.md @@ -0,0 +1,122 @@ +# Part 2.3: CLI Commands — dev review and dev research + +See [overview.md](overview.md) for architecture context. + +## Goal + +Expose `dev_review` and `dev_research` as CLI commands. Same output as MCP +tools, readable directly in the terminal. + +## What changes + +### `packages/cli/src/commands/review.ts` (new) + +```typescript +export const reviewCommand = new Command('review') + .description('Get a complete review context for files or changes') + .argument('[target]', 'File, directory, or git range (default: staged changes)') + .addOption( + new Option('--focus ', 'Review focus area') + .choices(['security', 'quality', 'performance', 'all']) + .default('all') + ) + .addOption( + new Option('--depth ', 'Analysis depth') + .choices(['quick', 'standard', 'deep']) + .default('standard') + ) + .option('--diff ', 'Git diff range (e.g., HEAD~1, main...HEAD)') + .option('--pr ', 'GitHub PR number to review') + .option('--json', 'Output as JSON', false) + .action(async (target, options) => { ... }); +``` + +**Target resolution priority:** +1. `--pr 31` → fetch changed files via `gh api repos/{owner}/{repo}/pulls/31/files` +2. `--diff HEAD~1` → parse `git diff --name-only HEAD~1` +3. `target` argument → file or directory path +4. No argument → staged changes (`git diff --cached --name-only`) + +**Implementation:** Creates `RepositoryIndexer`, runs the same analysis as +`ReviewAdapter.execute()`, but directly (not through MCP). Shares the analysis +functions via a shared module. + +### `packages/cli/src/commands/research.ts` (new) + +```typescript +export const researchCommand = new Command('research') + .description('Research a topic across the codebase') + .argument('', 'What to research (e.g., "authentication flow", "error handling patterns")') + .addOption( + new Option('--scope ', 'Research scope') + .choices(['architecture', 'implementation', 'usage']) + .default('implementation') + ) + .addOption( + new Option('--depth ', 'Research depth') + .choices(['quick', 'standard', 'deep']) + .default('standard') + ) + .option('--json', 'Output as JSON', false) + .action(async (query, options) => { ... }); +``` + +### Shared analysis module + +To avoid duplicating logic between CLI and MCP adapters: + +``` +packages/core/src/services/ + review-analysis.ts ← analysis logic (impact, patterns, structure) + research-analysis.ts ← analysis logic (search, enrich, format) +``` + +Both CLI commands and MCP adapters call these services. The services depend +on `RepositoryIndexer` and `SearchService` — no MCP-specific code. + +### `packages/cli/src/cli.ts` + +Register both commands: + +```typescript +import { reviewCommand } from './commands/review.js'; +import { researchCommand } from './commands/research.js'; + +program.addCommand(reviewCommand); +program.addCommand(researchCommand); +``` + +### Output formatting + +CLI output uses chalk for color in terminal, plain markdown for pipes: + +```bash +# Terminal (colored) +dev review packages/core/src/scanner/rust.ts + +# Pipe to file (plain markdown) +dev review packages/core/src/scanner/rust.ts > review.md + +# JSON for scripting +dev review --json packages/core/src/scanner/rust.ts | jq '.impact' +``` + +Detect TTY: if stdout is a TTY, use chalk. If piped, output plain markdown. + +## Tests + +| Test | What it verifies | +|------|-----------------| +| `dev review ` produces report | file target works | +| `dev review --diff HEAD~1` | git diff integration | +| `dev review` with no args uses staged | default to staged changes | +| `dev research ` produces report | query works | +| `dev research --scope architecture` | scope parameter respected | +| `dev review --json` outputs valid JSON | JSON mode works | +| Pipe detection (no chalk when piped) | TTY detection works | + +## Commit + +``` +feat(cli): add dev review and dev research commands +``` diff --git a/.claude/da-plans/mcp/phase-2-composite-tools/2.4-docs-and-agents.md b/.claude/da-plans/mcp/phase-2-composite-tools/2.4-docs-and-agents.md new file mode 100644 index 0000000..2c53f99 --- /dev/null +++ b/.claude/da-plans/mcp/phase-2-composite-tools/2.4-docs-and-agents.md @@ -0,0 +1,123 @@ +# Part 2.4: Docs, Agents, and Changelog + +See [overview.md](overview.md) for architecture context. + +## Goal + +Update all documentation, agent definitions, and release materials for the +new composite tools. + +## What changes + +### `CLAUDE.md` + +Update MCP tools section: + +```markdown +## MCP tools — conserve context + +### 5 low-level adapters (existing) +| Tool | Purpose | +|------|---------| +| dev_search | Hybrid code search | +| dev_refs | Callers/callees + dependency chains | +| dev_map | Codebase structure + hot paths | +| dev_patterns | File pattern analysis | +| dev_status | Index health | + +### 2 composite adapters (new) +| Tool | Purpose | +|------|---------| +| dev_review | Complete review context in one call — impact, patterns, structure | +| dev_research | Codebase research in one call — relevant code, call graph, architecture | + +Composite tools run low-level tools internally and return workflow-ready context. +One call replaces 5-10 low-level tool calls. +``` + +Update CLI commands section to include `dev review` and `dev research`. + +Update Agent → MCP Tool Matrix to show which agents benefit from composite tools. + +### Agent updates — specific frontmatter changes + +**code-reviewer.md** — Add `mcp__dev-agent__dev_review` to `tools:` frontmatter. +Use `dev_review` in Phase 1 to gather impact/patterns/structure in one call +instead of calling `dev_refs` + `dev_map` + `dev_patterns` separately. + +```yaml +# Current tools: +tools: Read, Grep, Glob, Bash, mcp__dev-agent__dev_search, mcp__dev-agent__dev_refs, mcp__dev-agent__dev_map, mcp__dev-agent__dev_patterns +# Add: +tools: ..., mcp__dev-agent__dev_review +``` + +**research-planner.md** — Add `mcp__dev-agent__dev_research` to `tools:` frontmatter. +Use `dev_research` in Phase 1 to map the territory in one call. + +```yaml +# Current tools: +tools: Read, Grep, Glob, Bash, mcp__dev-agent__dev_search, mcp__dev-agent__dev_refs, mcp__dev-agent__dev_map, mcp__dev-agent__dev_patterns +# Add: +tools: ..., mcp__dev-agent__dev_research +``` + +**bug-investigator.md** — Add `mcp__dev-agent__dev_research` to `tools:` frontmatter. +Use `dev_research "error in "` to quickly gather all relevant context. + +```yaml +# Current tools: +tools: Read, Edit, Write, Glob, Grep, Bash, mcp__dev-agent__dev_search, mcp__dev-agent__dev_refs, mcp__dev-agent__dev_map +# Add: +tools: ..., mcp__dev-agent__dev_research +``` + +**plan-reviewer.md** — Add `mcp__dev-agent__dev_review` to `tools:` frontmatter. +Use `dev_review` to verify that plan claims about file impact and conventions +match actual codebase state. + +```yaml +# Current tools: +tools: Read, Grep, Glob, Bash, mcp__dev-agent__dev_refs, mcp__dev-agent__dev_map, mcp__dev-agent__dev_patterns +# Add: +tools: ..., mcp__dev-agent__dev_review +``` + +### `website/content/updates/index.mdx` + +Add release entry. + +### `website/content/latest-version.ts` + +Update version. + +### Changeset + +```yaml +--- +'@prosdevlab/dev-agent': minor +--- + +Composite MCP tools — dev_review and dev_research + +- dev_review: complete review context in one call — impact analysis, pattern comparison, + structural context, and similar code detection +- dev_research: codebase research in one call — relevant code, call graphs, patterns, + and architecture for any conceptual query +- CLI commands: `dev review` and `dev research` for terminal use +- Depth levels (quick/standard/deep) control token cost and latency +``` + +## Tests + +| Test | What it verifies | +|------|-----------------| +| CLAUDE.md mentions dev_review and dev_research | docs updated | +| Agent definitions reference composite tools | agents updated | +| Changeset valid | `pnpm changeset status` passes | + +## Commit + +``` +docs: add composite tools to CLAUDE.md, agents, and doc site +``` diff --git a/.claude/da-plans/mcp/phase-2-composite-tools/overview.md b/.claude/da-plans/mcp/phase-2-composite-tools/overview.md new file mode 100644 index 0000000..b47d001 --- /dev/null +++ b/.claude/da-plans/mcp/phase-2-composite-tools/overview.md @@ -0,0 +1,265 @@ +# MCP Phase 2: Composite Tools — dev_review and dev_research + +**Status:** Draft (revised after AI agent architect review) + +## Context + +MCP Phase 1 delivered 5 low-level tools: `dev_search`, `dev_refs`, `dev_map`, +`dev_patterns`, `dev_status`. Each returns focused context for a specific query. + +AI assistants using these tools must orchestrate them manually — call `dev_search` +to find relevant code, `dev_refs` to trace call chains, `dev_patterns` to compare +conventions, `dev_map` for structural context. This works, but: + +1. **Round-trip latency** — 5 sequential tool calls take 5x the time of one +2. **Planning overhead** — the AI spends tokens deciding which tool to call next +3. **Incomplete coverage** — the AI may skip cross-package impact analysis or + pattern comparison because it doesn't know to ask + +### The insight + +Dev-agent is an MCP server. The AI assistant (Claude Code, Cursor) IS the LLM. +We don't need to bring our own. Our job is to give the assistant the best possible +context so it can do what it does best — reason, research, and synthesize. + +**Composite tools deliver workflow-ready context in a single call.** The assistant +reads it and applies judgment. We handle the data; it handles the intelligence. + +``` +┌──────────────────────────────────────────────────────────┐ +│ The Partnership │ +│ │ +│ dev-agent (MCP) │ AI assistant (LLM) │ +│ ───────────────── │ ───────────────── │ +│ Internal knowledge: │ External research: │ +│ What does our code do? │ What should it do? │ +│ Who calls what? │ What do best practices │ +│ What patterns exist? │ say? │ +│ Where does it fit? │ How do popular projects │ +│ │ handle this? │ +│ │ │ +│ dev_review returns: │ AI assistant adds: │ +│ impact analysis │ security concerns │ +│ pattern comparison │ logic issues │ +│ structural context │ recommendations │ +│ similar code │ evidence from web/docs │ +│ │ │ +│ dev_research returns: │ AI assistant adds: │ +│ relevant code │ external comparisons │ +│ call graphs │ best practice research │ +│ architecture │ synthesis + plan │ +└──────────────────────────────────────────────────────────┘ + +CLI is the degraded experience: structured data, no synthesis. +Still useful — developers can read the report. But the magic +happens in the AI assistant. +``` + +### What this phase delivers + +Two composite MCP tools that run low-level tools in parallel and return +enriched, structured context: + +- **`dev_review`** — Everything an AI needs to review a file or change +- **`dev_research`** — Everything an AI needs to understand a concept in the codebase + +Plus CLI commands (`dev review`, `dev research`) for standalone terminal use. + +--- + +## Architecture + +### Shared analysis services (not adapter DI) + +The AI agent architect review identified a key problem: existing adapters contain +inline analysis logic that would be duplicated by composites. The `refs` CLI command +already duplicates RefsAdapter's caller resolution algorithm. + +Fix: extract analysis logic into shared services in `packages/core/src/services/`. +Both MCP adapters and CLI commands call these services. + +``` +┌──────────────────────────────────────────────────────────┐ +│ packages/core/src/services/ │ +│ │ +│ review-analysis.ts ← impact, patterns, structure │ +│ research-analysis.ts ← search, enrich, format │ +│ │ +│ Pure functions. Take indexer + search service + graph. │ +│ Return structured data. No MCP, no CLI, no formatting. │ +└──────────────────────────────────────────────────────────┘ + ▲ ▲ + │ │ +┌────────┴─────────┐ ┌─────────┴──────────┐ +│ MCP Adapters │ │ CLI Commands │ +│ review-adapter │ │ dev review │ +│ research-adapter│ │ dev research │ +│ │ │ │ +│ Format for MCP │ │ Format for terminal│ +│ (markdown) │ │ (chalk/plain) │ +└──────────────────┘ └────────────────────┘ +``` + +This also sets up the path to refactor existing adapters (refs, map, etc.) to +use shared services — eliminating the CLI duplication problem. That refactor is +out of scope for Phase 2 but enabled by this architecture. + +### Parallel query composition + +Composites run analysis functions in parallel via `Promise.all`. This is not a +"swarm" or multi-agent system — it's parallel query composition with structured +formatting. The AI assistant provides the intelligence; we provide the data. + +```typescript +// review-analysis.ts — pure functions, no adapter dependencies +export async function analyzeForReview( + target: string[], + indexer: RepositoryIndexer, + searchService: SearchService, + graphPath?: string, + options?: { depth: 'quick' | 'standard' | 'deep' } +): Promise { + + const [impact, patterns, structure] = await Promise.all([ + analyzeImpact(target, indexer, graphPath), + comparePatterns(target, searchService), + getStructuralContext(target, indexer, graphPath), + ]); + + return { impact, patterns, structure }; +} +``` + +### Depth levels + +| Depth | What it runs | Latency | Best for | +|-------|-------------|---------|----------| +| `quick` | impact only (refs + graph) | <2s | Small changes, quick check | +| `standard` | impact + patterns + structure | <5s | Normal PRs, most reviews | +| `deep` | all + similar code + change frequency | <10s | Major refactors, new features | + +### Partial failure behavior + +When one specialist fails (e.g., patterns times out but refs succeeds): +- Return a partial report with available sections +- Add a warning note: "Pattern analysis unavailable — index may need refresh" +- Never crash. Always return something useful. + +--- + +## Parts + +| Part | Description | Risk | +|------|-------------|------| +| [2.1](./2.1-review-adapter.md) | Shared review analysis service + `dev_review` MCP adapter | Medium | +| [2.2](./2.2-research-adapter.md) | Shared research analysis service + `dev_research` MCP adapter | Medium | +| [2.3](./2.3-cli-commands.md) | `dev review` and `dev research` CLI commands (using shared services) | Low | +| [2.4](./2.4-docs-and-agents.md) | Update CLAUDE.md, agents, doc site, changelog | Low | + +--- + +## Decisions + +| Decision | Rationale | Alternatives | +|----------|-----------|-------------| +| No LLM in composite tools | AI assistant IS the LLM. We provide context, it provides judgment. No API key, no cost, no extra dependency. | LLM in tool: competes with a superior model, adds complexity | +| Shared services in core, not adapter DI | Avoids duplication between CLI and MCP. Pure functions, testable. Adapters are thin wrappers. | Adapter DI: creates coupling between MCP adapters. Service bypass: duplicates logic. | +| Parallel query composition, not "swarm" | Honest naming. It's Promise.all on analysis functions. No dynamic routing, no agent autonomy. | "Swarm" label: misleading, creates confusion with subagents package | +| Partial failure → partial report | Always return something useful. A review with impact but no patterns is better than an error. | Fail entirely: wastes the successful queries. Retry: adds latency. | +| MCP is primary, CLI is secondary | Our users are AI assistant users. MCP delivers the full experience. CLI is the degraded-but-functional fallback. | CLI-first: would need its own LLM, adds cost and complexity | +| dev_review description triggers on "review a PR/change" | Disambiguates from dev_patterns (single file analysis) and dev_search (conceptual query). | Generic description: AI won't know when to use it | +| dev_research enriches ALL results by default | Scope controls emphasis in output, not which data is collected. Avoids the counterintuitive "usage skips call graphs" problem. | Scope skips enrichments: confusing mapping, loses useful data | + +--- + +## Risk register + +| Risk | Likelihood | Impact | Mitigation | +|------|-----------|--------|------------| +| AI assistant doesn't call composite tools | Medium | High | Tool descriptions must clearly signal WHEN to use them. Test with Claude Code and Cursor. | +| Composite too slow (>10s) | Medium | Medium | Parallel execution + depth levels. Quick mode for fast feedback. | +| Output too large for AI context | Medium | Medium | Depth levels cap output. Quick: ~500, Standard: ~1,500, Deep: ~3,000 tokens. | +| AI makes worse decisions with composite than manual tools | Low | High | Test: compare review quality with composite vs 5 manual tool calls. If worse, the composite is failing. | +| Shared services create unwanted coupling | Low | Medium | Services are pure functions with minimal interface. No state, no side effects. | +| CLI experience too degraded without LLM | Medium | Low | CLI returns structured markdown that developers can read. Add LLM synthesis in Phase 3. | + +--- + +## Test strategy + +| Test | Priority | What it verifies | +|------|----------|-----------------| +| analyzeImpact returns callers/callees/rank | P0 | Core impact analysis works | +| comparePatterns returns convention comparison | P0 | Pattern analysis works | +| getStructuralContext returns subsystem + hot paths | P0 | Map integration works | +| ReviewAdapter returns combined report | P0 | MCP tool composes correctly | +| ReviewAdapter depth levels | P0 | Quick returns less than deep | +| ReviewAdapter partial failure | P0 | One specialist fails → partial report, not crash | +| ReviewAdapter file target | P0 | Single file resolved | +| ReviewAdapter git diff target | P1 | git range parsed, files extracted | +| ReviewAdapter invalid target | P0 | Graceful error for nonexistent file/bad git range | +| ResearchAdapter returns relevant code + enrichment | P0 | Search + enrichment pipeline works | +| ResearchAdapter scope controls output emphasis | P0 | All data collected, scope affects formatting | +| ResearchAdapter empty results | P0 | Helpful message, not crash | +| Tool descriptions disambiguate from low-level tools | P1 | AI chooses correctly in test scenarios | +| CLI `dev review ` produces report | P0 | Standalone CLI works | +| CLI `dev research ` produces report | P0 | Standalone CLI works | +| Shared services called by both MCP and CLI | P1 | No logic duplication | + +--- + +## Verification checklist + +### Automated (CI) +- [ ] Review analysis service tests pass +- [ ] Research analysis service tests pass +- [ ] MCP adapter tests pass +- [ ] CLI command tests pass +- [ ] `pnpm build && pnpm test` passes +- [ ] `pnpm typecheck` clean + +### Manual +- [ ] `dev_review` in Claude Code: AI uses the report to produce a quality review +- [ ] `dev_review` in Cursor: same experience +- [ ] `dev_research` in Claude Code: AI combines our context with its own web research +- [ ] AI chooses `dev_review` over `dev_search` + `dev_refs` when reviewing a PR +- [ ] CLI `dev review` produces readable terminal output +- [ ] CLI `dev research` produces readable terminal output + +--- + +## Commit strategy + +``` +1. feat(core): add review analysis service +2. feat(mcp): add dev_review composite adapter +3. feat(core): add research analysis service +4. feat(mcp): add dev_research composite adapter +5. feat(cli): add dev review and dev research commands +6. docs: update CLAUDE.md, agents, and doc site for composite tools +``` + +--- + +## Dependencies + +- MCP Phase 1 (5 low-level tools) — merged +- Core Phase 3 (cached dependency graph) — merged +- No new npm dependencies +- No LLM API key required + +--- + +## Future work (Phase 3) + +- **LLM synthesis for CLI** — Optional `ANTHROPIC_API_KEY` or Ollama integration + adds AI-powered synthesis to CLI output. MCP users already get this from their + AI assistant. +- **Cached external research** — `dev research --index ripgrep` clones, indexes, + and caches an external repo for comparison. Reusable across sessions. +- **Custom review profiles** — `.dev-agent/review.yml` for team-specific review + focuses. +- **PR integration** — `dev review --pr 31 --comment` posts report as GitHub PR + comment. +- **Refactor existing adapters** — Migrate refs, map, etc. to shared services + pattern, eliminating CLI duplication. diff --git a/.claude/scratchpad.md b/.claude/scratchpad.md index 511499d..67504d1 100644 --- a/.claude/scratchpad.md +++ b/.claude/scratchpad.md @@ -11,6 +11,7 @@ ## Future Work +- **Storage management** — Must ship before external repo indexing (MCP Phase 3). Indexes grow unbounded. Need: TTL-based eviction for external repos (30d default), max storage cap with LRU eviction, `dev storage --auto-cleanup`, config in `.dev-agent/config.yml` (`maxSize`, `externalTTL`, `keepPrimary`). Primary repo index never auto-deleted. Check storage health on every `dev index` / `dev status` call. - Antfly SDK: server-side path filter for `getDocsByFilePath` (eliminates 5k cap) - Betweenness centrality — identifies bridge files between subsystems. Worth adding if agents need refactoring guidance. graphology (MIT, 1.6k stars) is the upgrade path if we need more than 3 hand-rolled algorithms. - **Connected components hub filtering** — widely-shared utility files (e.g., logger.ts imported by 50+ files) merge separate subsystems into one component. Filter out hub nodes (high in-degree) before computing components for better subsystem identification. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f882a17..5eafd95 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,127 +1,145 @@ # Contributing to Dev-Agent -Thank you for considering contributing to dev-agent! This document outlines the process for contributing and the standards we follow. +## Prerequisites -## 🎯 **Core Values** +- **Node.js >= 22** (LTS) +- **pnpm** — `npm install -g pnpm` +- **Claude Code** — [install](https://claude.com/claude-code). We use Claude Code + dev-agent for development. +- **dev-agent** — `npm install -g @prosdevlab/dev-agent` +- **Antfly** — local search backend. Run `dev setup` after install. -1. **Testability First** - If it's hard to test, refactor it -2. **Modularity** - Small, focused, reusable modules -3. **100% Coverage on Utilities** - Pure functions should be fully tested -4. **Atomic Commits** - Each commit should build and test independently +## Getting started -## Development Process +```bash +# Clone and install +git clone https://github.com/prosdevlab/dev-agent.git +cd dev-agent +pnpm install + +# Start Antfly (one-time setup) +dev setup -1. **Fork and clone** the repository -2. **Install dependencies**: `pnpm install` -3. **Create a branch**: `git checkout -b feature/my-feature` -4. **Make your changes** -5. **Test your changes**: `pnpm test` -6. **Ensure code quality**: `pnpm lint && pnpm typecheck` +# Index the codebase +dev index -## Commit Message Convention +# Install MCP server for Claude Code +dev mcp install -We follow [Conventional Commits](https://www.conventionalcommits.org/) for commit messages. This is enforced using commitlint. +# Build and test +pnpm build +pnpm test +``` -Format: `type(scope): subject` +## Development workflow -Types: -- `feat`: A new feature -- `fix`: A bug fix -- `docs`: Documentation changes -- `style`: Changes that don't affect code meaning (formatting, etc.) -- `refactor`: Code changes that neither fix bugs nor add features -- `perf`: Performance improvements -- `test`: Adding or fixing tests -- `chore`: Changes to build process or auxiliary tools +We use Claude Code with dev-agent's MCP tools for development. The tools +provide semantic code search, call graph tracing, pattern analysis, and +codebase structure — saving significant context window usage. -Example: -``` -feat(core): add new API method for authentication +### Before you code + +1. **Index the repo:** `dev index` (run after pulling changes) +2. **Understand the area:** `dev search "your topic"`, `dev map --focus packages/core` +3. **Plan non-trivial features:** Write a plan in `.claude/da-plans/` and run the `plan-reviewer` agent before implementation + +### While you code + +```bash +pnpm dev # Watch mode +pnpm test # Run tests (from root, NOT turbo test) +pnpm lint # Biome lint +pnpm typecheck # Type check (AFTER pnpm build) ``` -## Pull Request Process +### Before you PR -1. Update the README.md if needed with details of changes to the interface. -2. Add a changeset to document your changes: `pnpm changeset` -3. Create a pull request to the `main` branch. -4. The PR will be reviewed and merged if it meets our standards. +1. Run the `code-reviewer` agent on your branch diff +2. Address any CRITICAL or WARNING findings +3. Add a changeset: `pnpm changeset` (only for `@prosdevlab/dev-agent` or `@prosdevlab/kero`) +4. Update release notes in `website/content/updates/index.mdx` and `website/content/latest-version.ts` -## Adding New Packages +## Commit convention -1. Create a new directory in the `packages` folder. -2. Create a `package.json`, `tsconfig.json`, and source files. -3. Add the package to relevant workspace configurations. -4. Update path mappings in the root `tsconfig.json`. +[Conventional Commits](https://www.conventionalcommits.org/) enforced by commitlint. -## Testing & Testability +``` +type(scope): description -### 📖 **Read First:** [TESTABILITY.md](./docs/TESTABILITY.md) +feat(mcp): add health check adapter +fix(core): resolve vector search timeout +docs: update CLAUDE.md +chore: update dependencies +``` -Our comprehensive testability guide covers: -- When and how to extract utilities -- Organization patterns -- Coverage targets -- Real-world examples +Types: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `chore`, `ci` -### **Quick Rules:** +## Pull request process -1. **Extract Pure Functions** to `utils/` modules - - ✅ DO: `utils/formatting.ts` with `formatDocument(doc: Document)` - - ❌ DON'T: Private methods in 500-line classes +1. Create a branch: `git checkout -b feat/my-feature` +2. Make your changes with tests +3. Run the full validation suite: `pnpm build && pnpm test && pnpm typecheck && pnpm lint` +4. Run `code-reviewer` agent on the diff +5. Add a changeset if the change affects published packages +6. Push and create a PR to `main` -2. **Aim for 100% on Utilities** - - Pure functions are easy to test - - No mocks needed - - Foundation for everything else +## CLI tools (available after `dev index`) -3. **No Non-Null Assertions (`!`)** - - Use guard clauses or optional chaining - - Makes code safer and more testable +```bash +dev search "authentication" # Semantic code search +dev refs "functionName" # Find callers/callees +dev refs "fn" --depends-on "src/db.ts" # Trace dependency chain +dev map # Codebase structure overview +dev map --focus packages/core --depth 3 # Focused map +``` -4. **Organize by Domain** - - ✅ `utils/strings.ts`, `utils/dates.ts`, `utils/validation.ts` - - ❌ `utils.ts` (500 lines of everything) +## Agent system (`.claude/agents/`) -### **Coverage Targets:** +We use Claude Code agents for code review, research, and planning: -| Code Type | Target | Example | -|-----------|--------|---------| -| **Pure Utilities** | 100% | `formatDocument()`, `calculateCoverage()` | -| **Integration** | >80% | `RepositoryIndexer`, `ExplorerAgent` | -| **CLI/UI** | >60% | Command handlers, spinners | +| Agent | Purpose | +|-------|---------| +| `code-reviewer` | Orchestrates security, logic, and quality review in parallel | +| `research-planner` | Maps internal code + delegates external research to sub-agents | +| `plan-reviewer` | Two-pass plan review (engineer + SDET) | +| `bug-investigator` | Systematic root cause analysis | +| `quick-scout` | Fast codebase exploration | -### **Before Submitting:** +Agents use dev-agent's MCP tools (`dev_search`, `dev_refs`, `dev_map`, `dev_patterns`) +to understand the codebase without reading every file. -```bash -# Run tests with coverage -pnpm vitest run --coverage +## Architecture -# Check specific package -pnpm vitest run packages/core/src/indexer --coverage -``` +See [CLAUDE.md](./CLAUDE.md) for the full monorepo structure, build order, +MCP tools reference, and non-negotiables. -- Write tests for all new features and bug fixes -- Run existing tests to ensure your changes don't break existing functionality -- See [TESTABILITY.md](./docs/TESTABILITY.md) for detailed guidelines +## Testing -## Code Style +- **Tests run from root only:** `pnpm test` +- **Build before typecheck:** `pnpm build` then `pnpm typecheck` +- **Biome for linting:** `pnpm lint` (not ESLint) -We use Biome for linting and formatting: +| Code type | Coverage target | +|-----------|----------------| +| Pure utilities | 100% | +| Integration | >80% | +| CLI/UI | >60% | -- Run `pnpm lint` to check code quality. -- Run `pnpm format` to format the code. +See [TESTABILITY.md](./docs/TESTABILITY.md) for detailed guidelines. -All code must pass linting and typechecking before being merged. +## Changesets -## Versioning +We use [Changesets](https://github.com/changesets/changesets) for versioning. +Only `@prosdevlab/dev-agent` and `@prosdevlab/kero` are published — all other +packages are private and bundled. -We use [Changesets](https://github.com/changesets/changesets) to manage versions and generate changelogs. +```bash +pnpm changeset # Create a changeset +``` -After making changes: -1. Run `pnpm changeset` -2. Follow the prompts to describe your changes -3. Commit the generated changeset file +When adding a changeset, also update: +1. `website/content/updates/index.mdx` — release notes +2. `website/content/latest-version.ts` — latest version callout ## Questions? -If you have any questions, please open an issue or discussion in the repository. \ No newline at end of file +Open an issue or discussion in the repository.