Draft
Conversation
… stats bugs Group A of SchemaAnalyzer refactor: - Fix A1: array element stats overwrite bug (isNewTypeEntry) - Fix A2: probability >100% for array-embedded objects (x-documentsInspected) - Rename folder: src/utils/json/mongo/ → src/utils/json/data-api/ - Rename enum: MongoBSONTypes → BSONTypes - Rename file: MongoValueFormatters → ValueFormatters - Add 9 new tests for array stats and probability
Group B of SchemaAnalyzer refactor: - B1: SchemaAnalyzer class with addDocument(), getSchema(), reset(), getDocumentCount() - B2: clone() method using structuredClone for schema branching - B3: addDocuments() batch convenience method - B4: static fromDocument()/fromDocuments() factories (replaces getSchemaFromDocument) - B5: Migrate ClusterSession to use SchemaAnalyzer instance - B6-B7: Remove old free functions (updateSchemaWithDocument, getSchemaFromDocument) - Keep getPropertyNamesAtLevel, getSchemaAtPath, buildFullPaths as standalone exports
…x properties type Group C of SchemaAnalyzer refactor: - C1: Add typed x-minValue, x-maxValue, x-minLength, x-maxLength, x-minDate, x-maxDate, x-trueCount, x-falseCount, x-minItems, x-maxItems, x-minProperties, x-maxProperties to JSONSchema interface - C2: Fix properties type: properties?: JSONSchema → properties?: JSONSchemaMap - C3: Fix downstream type errors in SchemaAnalyzer.test.ts (JSONSchemaRef casts)
…temBsonType Group D of SchemaAnalyzer refactor: - D1: Add bsonType to FieldEntry (dominant BSON type from x-bsonType) - D2: Add bsonTypes[] for polymorphic fields (2+ distinct types) - D3: Add isOptional flag (x-occurrence < parent x-documentsInspected) - D4: Add arrayItemBsonType for array fields (dominant element BSON type) - D5: Sort results: _id first, then alphabetical by path - D6: Verified generateMongoFindJsonSchema still works (additive changes) - G4: Add 7 getKnownFields tests covering all new fields
… toFieldCompletionItems) Group E of SchemaAnalyzer refactor: - E1: generateDescriptions() — post-processor adding human-readable description strings with type info, occurrence percentage, and min/max stats - E2: toTypeScriptDefinition() — generates TypeScript interface strings from JSONSchema for shell addExtraLib() integration - E3: toFieldCompletionItems() — converts FieldEntry[] to CompletionItemProvider- ready FieldCompletionData[] with insert text escaping and $ references Also: - Rename isOptional → isSparse in FieldEntry and FieldCompletionData (all fields are implicitly optional in MongoDB API / DocumentDB API; isSparse is a statistical observation, not a constraint) - Fix lint errors (inline type specifiers) - 18 new tests for transformers + updated existing tests
- Add 5 tests for clone(), reset(), fromDocument(), fromDocuments(), addDocuments() - Mark all checklist items A-G as complete, F1-F2 as deferred - Add Manual Test Plan section (§14) with 5 end-to-end test scenarios - Document clone() limitation with BSON Binary types (structuredClone)
- Add monotonic version counter to SchemaAnalyzer (incremented on mutations) - Cache getKnownFields() with version-based staleness check - Add ClusterSession.getKnownFields() accessor (delegates to cached analyzer) - Wire collectionViewRouter to use session.getKnownFields() instead of standalone function - Add ext.outputChannel.trace for schema accumulation and reset events
Co-authored-by: tnaum-ms <171359267+tnaum-ms@users.noreply.github.com>
…ypeScript definitions and completion items
Move SchemaAnalyzer, JSONSchema types, BSONTypes, ValueFormatters, and getKnownFields into packages/schema-analyzer as @vscode-documentdb/schema-analyzer. - Set up npm workspaces (packages/*) and TS project references - Update all extension-side imports to use the new package - Configure Jest multi-project for both extension and package tests - Remove @vscode/l10n dependency from core (replaced with plain Error) - Fix strict-mode type issues (localeCompare bug, index signatures) - Update .gitignore to include root packages/ directory - Add packages/ to prettier glob
…itions The bsonToTypeScriptMap emits non-built-in type names (ObjectId, Binary, Timestamp, etc.) without corresponding import statements or declare stubs. Currently harmless since the output is for display/hover only, but should be addressed if the TS definition is ever consumed by a real TS language service. Addresses PR #506 review comment from copilot.
…ion names - Prefix with _ when PascalCase result starts with a digit (e.g. '123abc' → '_123abcDocument') - Fall back to 'CollectionDocument' when name is empty or only separators - Filter empty segments from split result - Add tests for edge cases Addresses PR #506 review comment from copilot.
Add comment explaining why the cast to JSONSchema is safe: our SchemaAnalyzer never produces boolean schema refs. Notes that a typeof guard should be added if the function is ever reused with externally-sourced schemas. Addresses PR #506 review comment from copilot.
…lashes - Replace SPECIAL_CHARS_PATTERN with JS_IDENTIFIER_PATTERN for proper identifier validity check (catches dashes, brackets, digits, quotes, etc.) - Escape embedded double quotes and backslashes when quoting insertText - Add tests for all edge cases (dashes, brackets, digits, quotes, backslashes) - Mark future-work item #1 as resolved; item #2 (referenceText/$getField) remains open for aggregation completion provider phase Addresses PR #506 review comment from copilot.
…PI alongside MongoDB API
- Create workerTypes.ts with typed IPC message protocol (MainToWorkerMessage, WorkerToMainMessage) - Create scratchpadWorker.ts with init/eval/shutdown/tokenRequest handlers - Add scratchpadWorker webpack entry point to webpack.config.ext.js - Worker lazy-imports @mongosh/* packages (same pattern as current evaluator) - Supports both SCRAM and Entra ID auth (OIDC via IPC token callback) - Worker logs lifecycle events to main thread via 'log' messages Step 6.2 WI-1, Phase 1 of 3.
Rewrite ScratchpadEvaluator to route all execution through the worker thread: - ScratchpadEvaluator now manages worker lifecycle (spawn, kill, shutdown, dispose) - Worker state machine: idle → spawning → ready → executing → ready (or terminated) - Request/response correlation via requestId UUID map - Timeout enforced via worker.terminate() — actually stops infinite loops - Help command stays in main thread (static text, no @MongoSH needed) - Cluster switch detection: kills worker and respawns with new credentials - Entra ID OIDC token requests delegated via IPC to main thread - Worker logging routed to ext.outputChannel - executeScratchpadCode.ts: cancellable progress notification In-process eval path is fully replaced — no feature flag. Step 6.2 WI-1, Phase 2 of 3.
- Export disposeEvaluator() from executeScratchpadCode.ts for clean worker shutdown - Wire evaluator disposal into extension deactivation via ext.context.subscriptions - Worker thread is properly terminated when the extension deactivates Completes the SCRAM auth + kill/respawn wiring (credential passthrough was already implemented in Phase 2's buildInitMessage). Step 6.2 WI-1, Phase 3 of 3.
SchemaStore integration: - Cap schema feeding at 100 documents (randomly sampled via Fisher-Yates) - Prevents unbounded IPC/memory usage for large result sets Connection state synchronization: - Shutdown worker when scratchpad connection is cleared (disconnect) - Shutdown worker when the last .documentdb editor tab closes - Worker respawns lazily on next Run Export shutdownEvaluator() for graceful worker cleanup. Step 6.2 WI-2, Phase 5.
…lysis
Replace JSON.parse with EJSON.parse when deserializing worker eval results.
This preserves BSON types (ObjectId, Date, Decimal128) so that SchemaAnalyzer
correctly identifies field types for autocompletion.
With JSON.parse, BSON types became plain objects ({'$oid': '...'}) causing
SchemaAnalyzer to create wrong field paths (e.g. '_id.$oid' instead of '_id')
and wrong types (object instead of objectid).
Benchmarked on 100 documents with ~100 fields each:
- JSON.parse: 2.2ms parse, broken types (wrong paths + types)
- EJSON.parse: 9.9ms parse, correct types (only Int32/Long→Double)
- Both are negligible vs actual query time (100-5000ms)
Int32 and Long are still collapsed to Double (JavaScript number) — this is
a fundamental EJSON limitation, not a serialization bug.
Show distinct progress messages during scratchpad execution:
- 'Initializing scratchpad runtime…' — worker thread being created
- 'Authenticating with {clusterName}…' — MongoClient connecting + auth
- 'Running query…' — user code being evaluated
On subsequent runs (worker already alive), only 'Running query…' is shown.
The progress notification remains cancellable (Cancel kills worker).
Added onProgress callback parameter to ScratchpadEvaluator.evaluate().
Log levels: - Worker init/shutdown → debug (lifecycle, not user-visible) - Eval start/end → trace (verbose diagnostic) - Errors/uncaught exceptions → error - MongoClient close failure → warn - Worker exit, connection clear, editors close → debug - Route worker IPC log messages to matching LogOutputChannel methods (trace/debug/info/warn/error) instead of appendLine for all Progress UX: - Title changed to 'DocumentDB Scratchpad' (static bold prefix) - Phase messages show as: 'Initializing…', 'Authenticating…', 'Running query…' - Removed cluster name from authenticating phase (redundant in context) Cancel handling: - Suppress error notification when user explicitly cancels execution - Track cancelled state to avoid showing 'Worker terminated' error panel
Fix regression: worker not shutting down when scratchpad editors close. - Switched from onDidCloseTextDocument to tabGroups.onDidChangeTabs - onDidCloseTextDocument fires before tab state updates (race condition) - onDidChangeTabs fires after tabs are removed, state is consistent Add 'Show Schema Store Stats' diagnostics command: - Shows collection count, document count, field count in output channel - Per-collection breakdown with key, doc count, field count - Available via Command Palette: 'DocumentDB: Show Schema Store Stats'
…e EJSON serialization CursorIterationResult from @MongoSH is an Array subclass with extra properties (cursorHasMore, documents). EJSON.serialize treats it as a plain object and includes those properties, producing: { cursorHasMore: true, documents: [...] } instead of just: [doc1, doc2, ...] This caused: 1. Output showed cursor wrapper object instead of document array 2. resultFormatter showed 'Result: Cursor' instead of 'N documents returned' 3. SchemaStore never received documents (no _id at top level of wrapper object) Fix: Array.from(shellResult.printable) before EJSON.stringify normalizes Array subclasses to plain Arrays, preserving correct serialization.
@MongoSH's CursorIterationResult extends ShellApiValueClass (not Array). Its asPrintable() returns { ...this } which produces: { cursorHasMore: true, documents: [doc1, doc2, ...] } This wrapper object was passed through as-is, causing: 1. Output showed { cursorHasMore, documents: [...] } instead of just [...] 2. Header showed 'Result: Cursor' instead of 'N documents returned' 3. SchemaStore never received documents (wrapper has no _id field) Fix: Add unwrapCursorResult() helper that extracts the documents array from the { documents: [...] } wrapper. Applied in both: - resultFormatter.ts — for display formatting and document count - executeScratchpadCode.ts — for SchemaStore feeding
Use @MongoSH's result type instead of guessing from array shape: - Cursor results: 'Result: Cursor (20 documents)' — type + batch count - Other typed results: 'Result: Document', 'Result: string', etc. - No type: no header line (plain JS values) Previous behavior tried to detect 'documents' by checking Array.isArray which was fragile and didn't communicate what kind of result it was.
…count) - .toArray() returns type=null with an Array: show 'N results' - .count() returns type=null with a number: no special header (value shown) - Cursor: 'Result: Cursor (N documents)' (unchanged) - Typed results: 'Result: Document', etc. (unchanged)
1. Untyped array results (e.g. .toArray()): 'Result: Array (5 elements)' 2. Worker eval log: include line count '(3 lines, 51 chars, db: demo_data)' 3. Schema stats: show 'db/collection' instead of internal clusterId::db::coll
- Connect instruction dialog is now modal with title/detail separation - Scratchpad template includes note: 'only the last result is displayed' - Help text Tips section includes same note - Wording differs between template and help (not identical)
Replace 'MongoClient' with 'client' or 'database client' in: - Worker log messages visible in the output channel - JSDoc comments describing worker behavior - Code comments in worker and evaluator Type references (MongoClient, MongoClientOptions) are unchanged — these are the actual driver API names. The extension is a DocumentDB tool using the MongoDB API wire protocol. User-facing text should not reference MongoDB implementation details.
Collaborator
Author
Step 6.2 — Persistent Worker Eval (Option F)PR #540 implements Step 6.2: persistent worker thread for scratchpad code evaluation. Key changes:
See #540 for full details. |
…state
If buildInitMessage() throws or the worker reports initResult { success: false },
spawnWorker() now calls terminateWorker() before rethrowing. This returns the
evaluator to 'idle' state so the next evaluate() call can respawn a fresh worker
instead of being stuck in the 'spawning' state indefinitely.
…numeric types Switch from relaxed to canonical EJSON (relaxed: false) for the worker-to-main IPC payload. Canonical EJSON preserves Int32, Long, Double, and Decimal128 type wrappers so that EJSON.parse on the main thread reconstructs actual BSON instances. This allows SchemaAnalyzer.inferType() to correctly distinguish numeric subtypes, which feeds into type-aware operator ranking in completions. Also drops the space/indent parameter from EJSON.stringify since the IPC payload is never displayed to users — reducing transfer size. Fixes the incorrect comment that claimed Int32/Long collapse was a fundamental EJSON limitation (it was caused by using relaxed mode).
Mark displayBatchSize in workerTypes.ts and ScratchpadEvaluator.ts with TODO(F11) comments noting the field is sent but not yet read by the worker. References future-work.md §F11 for the plan to wire documentDB.mongoShell.batchSize.
…aluator
Wrap the three error strings most likely to reach users in l10n.t():
- 'No credentials found for cluster {0}'
- 'Worker is not running'
- 'Execution timed out after {0} seconds'
These flow through to vscode.window.showErrorMessage via the catch in
executeScratchpadCode.ts, so non-English users now see translated details.
…improved telemetry
…nResult shape The unwrapCursorResult() check and feedResultToSchemaStore() unwrap now require both 'cursorHasMore' (boolean) and 'documents' (array) before unwrapping. Previously, any object with a 'documents' array field would be unwrapped, which could false-positive on user documents with a 'documents' field.
…quest timeout
The timeout in sendRequest() is used for init, eval, and shutdown. The previous
message 'Execution timed out' was misleading when init hangs. Changed to
'Operation timed out after {0} seconds' which is accurate for all callers.
…oded 0 Capture startTime before evaluate() and compute elapsed time on failure. The error output panel now shows the real duration instead of 'Executed in 0ms'.
Only perform count (100) swaps instead of shuffling the entire array. Same output distribution, simpler loop bounds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Shell Integration — DocumentDB Query Language & Autocomplete
Umbrella PR for the shell integration feature: a custom
documentdb-queryMonaco language with intelligent autocomplete, hover docs, and validation across all query editor surfaces (filter, project, sort, aggregation, scratchpad).Work is organized as incremental steps, each delivered via a dedicated sub-PR merged into
feature/shell-integration.Progress
SchemaAnalyzer(JSON Schema output, incremental merge, 24 BSON types)@vscode-documentdb/schema-analyzerpackage, enrichedFieldEntrywith BSON types, added schema transformers, introduced monorepo structuredocumentdb-constantsPackage · feat: add documentdb-constants package — operator metadata for autocomplete #513 — 308 operator entries (DocumentDB API query operators, update operators, stages, accumulators, BSON constructors, system variables) as static metadata for autocompletedocumentdb-querycustom language with JS Monarch tokenizer (no TS worker), validated via POC across 8 test criteriaCompletionItemProvider· feat: documentdb-query language — CompletionItemProvider, HoverProvider, acorn validation #518 —documentdb-querylanguage registration, per-editor model URIs, completion data store,CompletionItemProvider(filter/project/sort),HoverProvider,acornvalidation,$-prefix fix, query parser replacement (shell-bson-parser), type-aware operator sorting, legacy JSON Schema pipeline removalcompletions/folderconnectToClient,mongoConnectionStrings).documentdbfiles with JS syntax highlighting, CodeLens (connection status, Run All, per-block Run), in-process@mongosheval reusing existingMongoClient, result formatting with headers/code echo/error hintsSchemaStore) · feat: Shared Schema Cache (SchemaStore) — Step 6.1 #538 — SharedSchemaStoresingleton for cross-tab and scratchpad schema sharing, Quick Scan action for on-demand schema samplingMongoClientfor infinite-loop safety and eval isolationCompletionItemProvider· 🔄 WI-1 in progress Step 7 WI-1: Scratchpad IntelliSense — Shell API type definitions + TS Server Plugin #543 — VS Code extension-hostCompletionItemProviderfordb.chains, operators, schema fields via.d.ts+ custom providershow dbs,use db,help,it,exit; persistent eval context;CommandInterceptorCompletionItemProvider— Stage-aware pipeline autocomplete with per-stage operator setsKey Architecture Decisions
documentdb-querycustom language — JS Monarch tokenizer, no TS worker (~400-600 KB saved)CompletionItemProvider+ URI routing (documentdb://{editorType}/{sessionId})documentdb-constantsbundled at build time; field data pushed via tRPC subscriptionacorn.parseExpressionAt()for syntax errors;acorn-walk+documentdb-constantsfor identifier validationlanguage="json"with JSON Schema validationlanguage="documentdb-scratchpad"referencing built-in JS grammar; in-process eval with@mongoshpackages reusing existingMongoClientCommandInterceptor