Skip to content

Commit 12d4845

Browse files
prosdevclaude
andcommitted
fix(core,mcp): address graph architect review findings
- Rename traceTo → dependsOn (makes call direction unambiguous) - Cache dependency graph on RefsAdapter (60s TTL, avoids rebuilding per request) - Log warning when 10k doc limit is hit in both dev_map and dev_refs - Remove overly strict shortestPath early return for unknown source nodes - Update docs for dependsOn parameter name - Track hub filtering, 10k scaling, and perf concerns in scratchpad Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent d40b7fe commit 12d4845

11 files changed

Lines changed: 595 additions & 20 deletions

File tree

.claude/da-plans/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ Implementation deviations are logged at the bottom of each plan file.
99

1010
| Track | Description | Status |
1111
|-------|-------------|--------|
12-
| [Core](core/) | Scanner, vector storage, services, indexer | Phase 1: Merged, Phase 2: Merged (indexing rethink) |
12+
| [Core](core/) | Scanner, vector storage, services, indexer | Phase 1: Merged, Phase 2: Merged, Phase 3: Draft (graph cache) |
1313
| [CLI](cli/) | Command-line interface | Not started |
1414
| [MCP Server](mcp/) | Model Context Protocol server + adapters | Phase 1: Merged (tools improvement) |
1515
| [Subagents](subagents/) | Coordinator, explorer, planner, GitHub agents | Not started |
Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Part 3.1: Build and Save Dependency Graph at Index Time
2+
3+
See [overview.md](overview.md) for architecture context.
4+
5+
## Goal
6+
7+
After `linearMerge` (full index) or `batchUpsertAndDelete` (incremental), build the
8+
dependency graph from the scan results and save it as JSON.
9+
10+
## What changes
11+
12+
### `packages/core/src/storage/path.ts`
13+
14+
Add `dependencyGraph` to `getStorageFilePaths`:
15+
16+
```typescript
17+
export function getStorageFilePaths(storagePath: string): {
18+
vectors: string;
19+
metadata: string;
20+
watcherSnapshot: string;
21+
dependencyGraph: string; // NEW
22+
// ... deprecated paths
23+
} {
24+
return {
25+
// ... existing
26+
dependencyGraph: path.join(storagePath, 'dependency-graph.json'),
27+
};
28+
}
29+
```
30+
31+
### `packages/core/src/map/graph.ts`
32+
33+
Add serialization/deserialization:
34+
35+
```typescript
36+
export interface CachedGraph {
37+
version: 1;
38+
generatedAt: string;
39+
nodeCount: number;
40+
edgeCount: number;
41+
graph: Record<string, WeightedEdge[]>;
42+
}
43+
44+
export function serializeGraph(graph: Map<string, WeightedEdge[]>): string {
45+
let edgeCount = 0;
46+
const obj: Record<string, WeightedEdge[]> = {};
47+
for (const [key, edges] of graph) {
48+
obj[key] = edges;
49+
edgeCount += edges.length;
50+
}
51+
return JSON.stringify({
52+
version: 1,
53+
generatedAt: new Date().toISOString(),
54+
nodeCount: graph.size,
55+
edgeCount,
56+
graph: obj,
57+
});
58+
}
59+
60+
export function deserializeGraph(json: string): Map<string, WeightedEdge[]> | null {
61+
try {
62+
const data = JSON.parse(json);
63+
if (data.version !== 1) return null;
64+
const graph = new Map<string, WeightedEdge[]>();
65+
for (const [key, edges] of Object.entries(data.graph)) {
66+
graph.set(key, edges as WeightedEdge[]);
67+
}
68+
return graph;
69+
} catch {
70+
return null;
71+
}
72+
}
73+
```
74+
75+
### `packages/core/src/indexer/index.ts`
76+
77+
After `linearMerge` completes in `index()`, build and save the graph:
78+
79+
```typescript
80+
// After linearMerge (line ~180)
81+
const documents = prepareDocumentsForEmbedding(scanResult.documents);
82+
// ... linearMerge call ...
83+
84+
// Build and cache dependency graph
85+
const graph = buildDependencyGraph(
86+
documents.map(d => ({ id: d.id, score: 0, metadata: d.metadata }))
87+
);
88+
const graphJson = serializeGraph(graph);
89+
await fs.writeFile(filePaths.dependencyGraph, graphJson, 'utf-8');
90+
```
91+
92+
## Tests
93+
94+
| Test | What it verifies |
95+
|------|-----------------|
96+
| `serializeGraph` round-trips correctly | Serialize → deserialize → same graph |
97+
| `deserializeGraph` returns null for invalid JSON | Error handling |
98+
| `deserializeGraph` returns null for wrong version | Schema evolution |
99+
| `getStorageFilePaths` includes `dependencyGraph` | Path registration |
100+
| After `index()`, graph file exists | Integration |
101+
102+
## Commit
103+
104+
```
105+
feat(core): build and save dependency graph at index time
106+
```
Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# Part 3.2: Load Cached Graph in dev_map and dev_refs
2+
3+
See [overview.md](overview.md) for architecture context.
4+
5+
## Goal
6+
7+
Replace `getAll(limit: 10000)``buildDependencyGraph()` in `dev_map` and `dev_refs`
8+
with loading the cached graph from disk. Falls back to current approach if graph file
9+
is missing or corrupted.
10+
11+
## What changes
12+
13+
### `packages/core/src/map/graph.ts`
14+
15+
Add a loader that reads from disk with fallback:
16+
17+
```typescript
18+
import * as fs from 'node:fs/promises';
19+
20+
/**
21+
* Load dependency graph from cache, or build from docs as fallback.
22+
*/
23+
export async function loadOrBuildGraph(
24+
graphPath: string | undefined,
25+
fallbackDocs: () => Promise<SearchResult[]>
26+
): Promise<Map<string, WeightedEdge[]>> {
27+
// Try cached graph first
28+
if (graphPath) {
29+
try {
30+
const json = await fs.readFile(graphPath, 'utf-8');
31+
const graph = deserializeGraph(json);
32+
if (graph) return graph;
33+
} catch {
34+
// File missing or unreadable — fall through to build
35+
}
36+
}
37+
38+
// Fallback: build from docs (current approach)
39+
const docs = await fallbackDocs();
40+
return buildDependencyGraph(docs);
41+
}
42+
```
43+
44+
### `packages/core/src/map/index.ts`
45+
46+
Replace the graph build in `generateCodebaseMap`:
47+
48+
```typescript
49+
// Before (current):
50+
const graph = buildDependencyGraph(allDocs);
51+
52+
// After:
53+
const graph = await loadOrBuildGraph(
54+
context.graphPath, // new optional field on MapGenerationContext
55+
async () => allDocs // fallback uses already-fetched docs
56+
);
57+
```
58+
59+
Add `graphPath` to `MapGenerationContext`:
60+
61+
```typescript
62+
export interface MapGenerationContext {
63+
indexer: RepositoryIndexer;
64+
gitExtractor?: LocalGitExtractor;
65+
logger?: Logger;
66+
graphPath?: string; // NEW — path to cached dependency-graph.json
67+
}
68+
```
69+
70+
### `packages/mcp-server/src/adapters/built-in/refs-adapter.ts`
71+
72+
Replace the `getDependencyGraph` method:
73+
74+
```typescript
75+
private async getDependencyGraph() {
76+
const CACHE_TTL_MS = 60_000;
77+
if (this.cachedGraph && Date.now() - this.cachedGraphTime < CACHE_TTL_MS) {
78+
return this.cachedGraph;
79+
}
80+
81+
// Try loading from disk first (no getAll needed)
82+
this.cachedGraph = await loadOrBuildGraph(
83+
this.graphPath,
84+
async () => this.indexer!.getAll({ limit: 50000 }) // raised limit as fallback
85+
);
86+
this.cachedGraphTime = Date.now();
87+
return this.cachedGraph;
88+
}
89+
```
90+
91+
### `packages/mcp-server/bin/dev-agent-mcp.ts`
92+
93+
Pass `graphPath` to both MapAdapter and RefsAdapter from `getStorageFilePaths`.
94+
95+
## Tests
96+
97+
| Test | What it verifies |
98+
|------|-----------------|
99+
| `loadOrBuildGraph` with valid cached file | Loads from disk, doesn't call fallback |
100+
| `loadOrBuildGraph` with missing file | Calls fallback, builds from docs |
101+
| `loadOrBuildGraph` with corrupted file | Calls fallback, doesn't crash |
102+
| `generateCodebaseMap` uses cached graph when available | Integration |
103+
| `dev_refs dependsOn` uses cached graph | Integration |
104+
105+
## Commit
106+
107+
```
108+
feat(core,mcp): load cached dependency graph in dev_map and dev_refs
109+
```
Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Part 3.3: Incremental Graph Updates via File Watcher
2+
3+
See [overview.md](overview.md) for architecture context.
4+
5+
## Goal
6+
7+
When the file watcher detects changes and calls `applyIncremental`, update the
8+
cached dependency graph without a full rebuild. This keeps the graph fresh as
9+
files are edited.
10+
11+
## What changes
12+
13+
### `packages/core/src/map/graph.ts`
14+
15+
Add an incremental update function:
16+
17+
```typescript
18+
/**
19+
* Update a dependency graph incrementally.
20+
*
21+
* For changed/new files: remove old edges from those files, add new edges.
22+
* For deleted files: remove all edges from those files.
23+
* Pure function — returns a new graph.
24+
*/
25+
export function updateGraphIncremental(
26+
existing: Map<string, WeightedEdge[]>,
27+
changedDocs: SearchResult[],
28+
deletedFiles: string[]
29+
): Map<string, WeightedEdge[]> {
30+
const updated = new Map(existing);
31+
32+
// Remove edges for deleted files
33+
for (const file of deletedFiles) {
34+
updated.delete(file);
35+
}
36+
37+
// Remove old edges for changed files, then add new ones
38+
const changedGraph = buildDependencyGraph(changedDocs);
39+
for (const file of changedGraph.keys()) {
40+
// Remove old edges (the file was re-scanned)
41+
updated.delete(file);
42+
}
43+
for (const [file, edges] of changedGraph) {
44+
updated.set(file, edges);
45+
}
46+
47+
return updated;
48+
}
49+
```
50+
51+
### `packages/core/src/indexer/index.ts`
52+
53+
In `applyIncremental`, update the cached graph:
54+
55+
```typescript
56+
async applyIncremental(upserts: EmbeddingDocument[], deleteIds: string[]): Promise<void> {
57+
await this.vectorStorage.batchUpsertAndDelete(upserts, deleteIds);
58+
59+
// Update cached dependency graph
60+
const graphPath = getStorageFilePaths(this.config.vectorStorePath).dependencyGraph;
61+
try {
62+
const existing = await loadGraphFromDisk(graphPath);
63+
if (existing) {
64+
const deletedFiles = extractFilesFromDeleteIds(deleteIds);
65+
const changedDocs = upserts.map(d => ({ id: d.id, score: 0, metadata: d.metadata }));
66+
const updated = updateGraphIncremental(existing, changedDocs, deletedFiles);
67+
await fs.writeFile(graphPath, serializeGraph(updated), 'utf-8');
68+
}
69+
} catch {
70+
// Graph update failed — next full index will fix it
71+
}
72+
}
73+
```
74+
75+
## Tests
76+
77+
| Test | What it verifies |
78+
|------|-----------------|
79+
| `updateGraphIncremental` adds edges for new files | New file → new edges appear |
80+
| `updateGraphIncremental` removes edges for deleted files | Deleted file → edges gone |
81+
| `updateGraphIncremental` replaces edges for changed files | Changed file → old edges removed, new edges added |
82+
| `updateGraphIncremental` with empty existing graph | Handles first incremental gracefully |
83+
| Incremental update failure doesn't crash indexer | Error resilience |
84+
85+
## Commit
86+
87+
```
88+
feat(core): incremental dependency graph updates via file watcher
89+
```

0 commit comments

Comments
 (0)