feat: annotate automatic tools and exclude from fluency scoring#529
Merged
feat: annotate automatic tools and exclude from fluency scoring#529
Conversation
Add automaticTools.json with a curated list of tool IDs that Copilot calls autonomously (file reads, codebase searches, error checks, confirmations, etc.). These tools do not require user configuration and should not inflate fluency scores. Changes: - New vscode-extension/src/automaticTools.json: list of automatic tool IDs - maturityScoring.ts: compute nonAutoToolCount and use it for tuStage/agStage thresholds; evidence text now distinguishes intentional vs automatic tools - webview/usage/main.ts: show 'auto' badge on automatic tools in tool tables - cli/esbuild.js: include automaticTools.json in the bundled data files - docs/FLUENCY-LEVELS.md: document the automatic vs intentional tool distinction Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- maturityScoring.ts getFluencyLevelData(): update threshold text in Agentic and Tool Usage levels to say 'intentional tools' instead of 'unique tools', and explain that automatic tools are excluded (Stage 1 description, Stage 2 now lists examples of intentional tools) - .github/agents/tool-names.agent.md: add automaticTools.json as a key file, add Automatic vs. Intentional Tools guidance section, update checklist - sync-toolnames prompts (both copies): update constraints to also modify automaticTools.json and classify new tools as automatic or intentional Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The fluency score's Tool Usage and Agentic categories counted all tools Copilot uses — including tools it calls automatically on its own (reading files, searching the codebase, checking errors, etc.). This inflated scores for users who simply use agent mode, even if they haven't configured anything intentional like MCP servers or terminal access. Only tools the user explicitly enables or configures should signal AI fluency.
What changed
New:
vscode-extension/src/automaticTools.jsonA curated list of ~60 tool IDs that Copilot calls autonomously as background context gathering — file reads, directory listings, grep/search, error checks, confirmations, memory, terminal reads, and internal session tools. These are excluded from fluency scoring but still shown everywhere else.
Fluency scoring (
maturityScoring.ts)Both the fast-path (
calculateFluencyScoreForTeamMember) and the detail-path (calculateMaturityScores) now computenonAutoToolCountby filtering automatic tools from the tool map. This value drives:Evidence text now reads e.g. "3 intentional tools used (+ 8 automatic)" so users understand the distinction.
The
getFluencyLevelData()function (which powers the Fluency Level Viewer) also has its threshold descriptions updated to say "intentional tools" and explain automatic tools are excluded.Usage Analysis webview (
webview/usage/main.ts)All tools still appear in the tool-usage tables, but automatic tools now show an
autobadge with a tooltip explaining they don't count toward fluency scoring.CLI (
cli/esbuild.js)automaticTools.jsonadded to the data files copied and bundled at build time — the CLI sharesmaturityScoring.tsso it benefits automatically.Documentation & agent instructions
docs/FLUENCY-LEVELS.md— new Automatic vs. Intentional Tools section with examples of each category.github/agents/tool-names.agent.md—automaticTools.jsonlisted as a key file; added decision guide and checklist item for classifying new toolsautomaticTools.jsonwhen syncing new tool names