From e61cffe3a3776c03c78d099df7423edd59ed69d8 Mon Sep 17 00:00:00 2001 From: Oto Macenauer Date: Thu, 4 Jun 2026 11:07:17 +0200 Subject: [PATCH] docs: add agentic development tools documentation Add docs/tools.md as overview catalog and per-tool guides for Pi coding agent, CodeBurn token dashboard, and GitHub Copilot CLI. Each guide covers quickstart, key features, token-saving extensions, and troubleshooting. Update docs/README.md with tools section. Closes #10 Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/README.md | 9 ++ docs/tools/README.md | 17 ++++ docs/tools/codeburn.md | 207 ++++++++++++++++++++++++++++++++++++++ docs/tools/copilot-cli.md | 185 ++++++++++++++++++++++++++++++++++ docs/tools/pi.md | 123 ++++++++++++++++++++++ 5 files changed, 541 insertions(+) create mode 100644 docs/tools/README.md create mode 100644 docs/tools/codeburn.md create mode 100644 docs/tools/copilot-cli.md create mode 100644 docs/tools/pi.md diff --git a/docs/README.md b/docs/README.md index e60899a..da4fcc9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,6 +11,15 @@ Navigation hub for all guides in this repository. Browse by category below. | [Skill Testing](./skill-testing.md) | Eval creation, fixtures, regression loops, trigger and description optimization | | [Troubleshooting](./troubleshooting.md) | Setup fixes for install, activation, and proxy issues | +## Tools for Agentic Development + +| Guide | Description | +|-----------------------------|--------------------------------------------------------------------| +| [Tools Overview](./tools/) | Catalog of CLI-based AI coding agents and agentic development tools | +| [Pi Coding Agent](./tools/pi.md) | Minimal BYOK terminal agent -- quickstart, extensions, troubleshooting | +| [GitHub Copilot CLI](./tools/copilot-cli.md) | Multi-model agentic terminal agent with native GitHub integration | +| [CodeBurn](./tools/codeburn.md) | Token usage, cost, and performance dashboard for 25+ AI coding tools | + ## Skill Guides | Guide | Description | diff --git a/docs/tools/README.md b/docs/tools/README.md new file mode 100644 index 0000000..2dd2af0 --- /dev/null +++ b/docs/tools/README.md @@ -0,0 +1,17 @@ +# Tools for Agentic Development + +A curated catalog of tools that support AI-powered agentic programming workflows. Each tool has its own guide covering what it does, how to get started, recommended extensions, and common issues. + +## What Are Agentic Development Tools? + +Agentic development tools are CLI-based AI coding agents that operate directly in your terminal. They read, write, and edit code, run shell commands, and iterate autonomously toward a goal you describe in natural language. Unlike traditional IDE copilots that autocomplete single lines, agentic tools take ownership of multi-step tasks: implementing features, debugging across files, running tests, and fixing what breaks. + +## Tool Guides + +| Tool | Description | +|------|-------------| +| [Pi](./pi.md) | Minimal, open-source, BYOK terminal coding agent with a rich extension ecosystem | +| [GitHub Copilot CLI](./copilot-cli.md) | Multi-model agentic terminal agent with native GitHub integration | +| [CodeBurn](./codeburn.md) | CLI/TUI dashboard tracking AI coding token usage, cost, and performance across 25+ tools | + +> **Adding a new tool?** Create `docs/tools/.md` and add a row to the table above. diff --git a/docs/tools/codeburn.md b/docs/tools/codeburn.md new file mode 100644 index 0000000..2283dd3 --- /dev/null +++ b/docs/tools/codeburn.md @@ -0,0 +1,207 @@ +# [CodeBurn](https://codeburn.app/) + +CodeBurn is an open-source CLI and TUI dashboard that tracks AI coding token usage, cost, and performance across 25+ AI coding tools. It breaks down spending by task type, model, tool, project, and provider. Everything runs locally -- no wrapper, no proxy, no API keys. Built by [AgentSeal](https://github.com/getagentseal). + +## Why CodeBurn + +- **25+ providers** -- Claude Code, Cursor, Codex, Gemini CLI, Pi, Copilot, Cline, Roo Code, and many more. Auto-detected from session data on disk. +- **Zero overhead** -- reads existing session files directly, prices calls via LiteLLM. No proxy or API key needed. +- **Task classification** -- deterministic categorization (coding, debugging, feature dev, refactoring, testing, exploration, planning, delegation, git ops, build/deploy, brainstorming, conversation) from tool usage patterns, no LLM calls. +- **One-shot rate** -- tracks how often the AI gets edits right on the first try, per model and task category. +- **macOS menubar app** -- live spend widget that refreshes every 30 seconds. + +## Trade-offs + +- Menubar mode is macOS-only (Swift/SwiftUI). +- Cursor support degraded -- Cursor stopped writing per-call token counts to its local DB in early 2026, so parsing may report $0. +- Some providers (Kiro, Copilot) estimate tokens from content length since session data lacks explicit counts. + +--- + +## Quickstart + +```bash +brew tap getagentseal/codeburn +brew install codeburn +``` + +Or via npm: + +```bash +npm install -g codeburn +``` + +Or run without installing: + +```bash +npx codeburn +``` + +Launch the interactive TUI dashboard: + +```bash +codeburn +``` + +--- + +## Menubar Mode (macOS) + +Install and launch with a single command: + +```bash +codeburn menubar +``` + +Reinstall with `codeburn menubar --force`. The menubar icon shows spend for the selected period. Click to open a popover with agent tabs, trend/forecast/pulse/stats views, activity breakdowns, optimize findings, and CSV/JSON export. + +**Configure period from terminal:** + +```bash +defaults write org.agentseal.codeburn-menubar CodeBurnMenubarPeriod -string month +``` + +Allowed values: `today`, `week`, `month`, `sixMonths`. + +**Compact mode** (drops decimals): + +```bash +defaults write org.agentseal.codeburn-menubar CodeBurnMenubarCompact -bool true +``` + +--- + +## Commands + +### Dashboard & Reports + +| Command | Description | +|---------|-------------| +| `codeburn` | Interactive TUI dashboard (default: 7 days) | +| `codeburn today` | Today-only view | +| `codeburn month` | Current month view | +| `codeburn status` | Compact one-liner with today + month totals | +| `codeburn report` | Full report with custom period/format | + +Report supports filters: + +```bash +codeburn report -p 30days +codeburn report -p all +codeburn report --from 2026-04-01 --to 2026-04-10 +codeburn report --provider claude +codeburn report --project myapp +codeburn report --format json +codeburn report --refresh 60 # auto-refresh interval in seconds +``` + +### Analysis + +| Command | Description | +|---------|-------------| +| `codeburn optimize` | Find waste patterns, get copy-paste fixes (graded A-F) | +| `codeburn compare` | Side-by-side model comparison (1-shot rate, cost/edit, cache hit) | +| `codeburn yield` | Productive vs reverted vs abandoned spend (requires git repo) | +| `codeburn models` | Per-model token and cost table | + +```bash +codeburn optimize -p today +codeburn compare -p week --provider claude +codeburn yield -p 30days +codeburn models --by-task --top 10 +``` + +### Export + +```bash +codeburn export # CSV (today, 7d, 30d) +codeburn export -f json +codeburn export --project inventory +``` + +### Configuration + +| Command | Description | +|---------|-------------| +| `codeburn plan set claude-max` | Track against $200/month plan | +| `codeburn plan set claude-pro` | Track against $20/month plan | +| `codeburn plan set custom --monthly-usd 200 --provider codex` | Custom plan | +| `codeburn plan reset` | Remove plan config | +| `codeburn currency GBP` | Set display currency (162 ISO 4217 codes) | +| `codeburn currency --reset` | Back to USD | +| `codeburn model-alias "my-proxy" "claude-opus-4-6"` | Map proxy model names | + +### TUI Keyboard Shortcuts + +| Key | Action | +|-----|--------| +| `q` | Quit | +| `1`-`5` | Switch period (Today, 7D, 30D, Month, 6M) | +| `c` | Model comparison | +| `o` | Optimize view | +| `b` | Go back | +| `p` | Toggle providers | + +--- + +## Suggested Workflow + +1. Run `codeburn optimize` to find waste. Wait 48 hours, run again to see improvement. +2. After a week, use `codeburn compare` to decide if your default model fits your task mix. +3. In git repos, run `codeburn yield` monthly to see how much AI spend translates into landed commits. +4. Watch one-shot rate per category weekly -- this tracks your prompting skill, not just the model. + +### Reading the Dashboard + +| Signal | Likely meaning | +|--------|---------------| +| Cache hit < 80% | System prompt or context not stable | +| Lots of `Read` calls | Agent re-reading same files, missing context | +| Low 1-shot rate (~30%) | Agent stuck in retry loops | +| Opus dominating small turns | Overpowered model for simple tasks | +| Bash dominated by `git status`, `ls` | Agent exploring instead of executing | + +--- + +## Environment Variables + +| Variable | Description | +|----------|-------------| +| `CLAUDE_CONFIG_DIR` | Override Claude Code data directory | +| `CLAUDE_CONFIG_DIRS` | OS-delimited list of Claude directories to scan together | +| `CODEX_HOME` | Override Codex data directory | + +Multiple Claude configs can be merged: + +```bash +CLAUDE_CONFIG_DIRS=~/.claude-work:~/.claude-personal codeburn +``` + +--- + +## Troubleshooting + +### No data showing + +CodeBurn reads session files from disk. If a provider shows no data, verify the tool has been used and check that its data directory exists. Use environment variables to point to non-default locations. + +### Cursor shows $0 + +Cursor stopped writing per-call token counts to `state.vscdb` in early 2026. This is a Cursor-side change, not a CodeBurn bug. + +### First run is slow + +On large Cursor databases, the first scan can take up to a minute. Results are cached after. + +### JSON integration + +```bash +codeburn report --format json | jq '.projects' +codeburn today --format json | jq '.overview.cost' +``` + +--- + +## Links + +- [Website](https://codeburn.app/) -- [Documentation](https://codeburn.app/docs) -- [GitHub](https://github.com/getagentseal/codeburn) -- [npm](https://www.npmjs.com/package/codeburn) -- [Discord](https://discord.gg/w2sw8mCqep) diff --git a/docs/tools/copilot-cli.md b/docs/tools/copilot-cli.md new file mode 100644 index 0000000..4be4574 --- /dev/null +++ b/docs/tools/copilot-cli.md @@ -0,0 +1,185 @@ +# [GitHub Copilot CLI](https://github.com/features/copilot/cli) + +GitHub Copilot CLI is a fully agentic, terminal-native coding agent from GitHub. Not to be confused with the older `gh copilot suggest` shell helper -- this is a complete autonomous agent that edits files, runs commands, creates PRs, and iterates toward goals. It supports multiple LLM providers (Claude, GPT, Gemini) within the same tool and integrates natively with the GitHub ecosystem. + +## Why Copilot CLI + +- **Multi-model** -- switch between Claude Sonnet/Opus, GPT-5.x, and Gemini models mid-session via `/model`. +- **Plan + Autopilot modes** -- Plan mode builds a structured approach for your review; Autopilot executes autonomously. +- **Built-in GitHub integration** -- repos, issues, PRs, and code search through natural language. +- **MCP server support** -- GitHub's own MCP server built-in, plus custom MCP servers. +- **LSP support** -- go-to-definition, hover, and diagnostics for richer code understanding. +- **Custom agents** -- define reusable `.agent.md` files with specialized instructions and tools. +- **Subscription-based** -- uses your existing GitHub Copilot subscription (Free through Enterprise). + +## Trade-offs + +- Requires a GitHub Copilot subscription (usage-based AI Credits since June 2026). +- Opus models restricted to Pro+ and above. +- MCP tool schemas add token overhead -- 40 tools can cost 10-15KB per turn. +- Younger than Claude Code as an agentic tool; ecosystem still maturing. +- Some Windows-specific rendering issues (flickering, keyboard input). + +--- + +## Quickstart + +```bash +# npm +npm install -g @github/copilot + +# Homebrew +brew install github/copilot/copilot + +# WinGet (Windows) +winget install GitHub.CopilotCLI + +# Shell script +curl -fsSL https://cli.github.com/copilot/install.sh | bash +``` + +Authenticate: + +```bash +copilot auth +``` + +Requires an active GitHub Copilot subscription (Free, Pro, Pro+, Max, Business, or Enterprise). + +--- + +## Key Commands + +| Command | Purpose | +|---------|---------| +| `/model` | Switch AI model mid-session | +| `/compact` | Manually compress session context | +| `/clear` | Reset conversation context | +| `/session` | Show session metrics | +| `/usage` | Show token usage metrics | +| `/add-dir` | Grant access to specific directories | +| `/remote` | Control session from github.com or mobile | +| `/chronicle` | Review session history with personalized tips | +| `/every` | Schedule recurring prompt at interval | +| `/after` | Schedule one-shot prompt after delay | +| `/rubber-duck` | Adversarial feedback mode | +| `/experimental` | Toggle experimental features | + +Full reference: [CLI Command Reference](https://docs.github.com/en/copilot/reference/copilot-cli-reference/cli-command-reference) + +--- + +## Custom Instructions + +Copilot CLI uses `copilot-instructions.md` files (not `CLAUDE.md` or `AGENTS.md`): + +| Scope | Location | +|-------|----------| +| Global | `~/.copilot/copilot-instructions.md` | +| Repository | `.github/copilot-instructions.md` | +| Path-specific | `.github/instructions/NAME.instructions.md` (with path globs) | + +Custom agents use `.agent.md` files: + +| Scope | Location | +|-------|----------| +| Global | `~/.copilot/agents/NAME.agent.md` | +| Project | `.github/agents/NAME.agent.md` | + +Invoke with: `copilot --agent=refactor-agent --prompt "..."`. See [Creating Custom Agents](https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/create-custom-agents-for-cli). + +--- + +## Configuration + +Config lives in `~/.copilot/` (override with `COPILOT_HOME` env var). + +| File | Purpose | +|------|---------| +| `settings.json` | User preferences (JSONC) | +| `config.json` | Internal application state | +| `copilot-instructions.md` | Global custom instructions | + +Settings cascade: User -> Repository -> Local. CLI flags and env vars take highest precedence. + +**Key environment variables:** + +| Variable | Purpose | +|----------|---------| +| `COPILOT_HOME` | Redirect config/state storage | +| `COPILOT_GITHUB_TOKEN` | Auth token (also checks `GH_TOKEN`, `GITHUB_TOKEN`) | +| `COPILOT_CUSTOM_INSTRUCTIONS_DIRS` | Comma-separated list of instruction directories | + +--- + +## Token-Saving Extensions + +Each MCP tool registration costs 100-500 tokens per step. With many servers, overhead adds up fast. These extensions help keep token usage under control. + +### [caveman](https://github.com/JuliusBrussee/caveman) + +Ultra-compressed communication mode that cuts ~75% of output tokens while keeping full technical accuracy. Caveman ships rule files that auto-detect Copilot CLI during install. The agent speaks in terse, fragment-style responses -- all technical substance preserved, all filler removed. + +Install on Windows: + +```powershell +irm https://raw.githubusercontent.com/JuliusBrussee/caveman/main/install.ps1 | iex +``` + +Install on macOS/Linux: + +```bash +curl -fsSL https://raw.githubusercontent.com/JuliusBrussee/caveman/main/install.sh | bash +``` + +Caveman installs as always-on rule files for Copilot CLI (not a plugin/extension). Rules are placed in your `copilot-instructions.md`. + +### [context-mode](https://github.com/mksglu/context-mode) + +Context window optimization via MCP server. Sandboxes tool output so raw bytes never enter the context window -- up to 98% reduction. Tracks edits, git operations, tasks, and errors in SQLite with FTS5 search. On compaction, retrieves only BM25-ranked relevant content. + +Copilot CLI support since v1.0.0. Connects as a custom MCP server: + +```bash +npm install -g context-mode +``` + +Then add to your Copilot CLI MCP configuration. See [platform support docs](https://github.com/mksglu/context-mode/blob/main/docs/platform-support.md) for Copilot-specific setup. + +### [RTK (Rust Token Killer)](https://github.com/rtk-ai/rtk) + +CLI proxy that rewrites verbose shell commands into token-optimized equivalents (60-90% savings on dev operations). Two approaches for Copilot CLI: + +**Official RTK init:** + +```bash +rtk init -g --copilot +``` + +**Community approach** -- [rtk-for-copilot](https://github.com/Martin-Sciarrillo/rtk-for-copilot) uses `copilot-instructions.md` to teach the agent to route commands through RTK. No hooks needed; place instructions at `~/.copilot/copilot-instructions.md` (global) or `.github/copilot-instructions.md` (per-repo). + +--- + +## Troubleshooting + +### Authentication fails + +Run `copilot auth` to re-authenticate. Check that `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN` is set correctly if using env vars. Older clients may fail to communicate with servers -- keep the tool updated. + +### Rate limiting / credits exhausted + +Copilot CLI enforces session and weekly (7-day) limits based on token consumption with model multipliers. Opus models consume credits faster. Check remaining credits with `/usage`. Downgrade model with `/model` to stretch budget. + +### High token overhead from MCP tools + +Each registered MCP tool schema is included in every API call. 15 servers x 15 steps = ~265K tokens of overhead from unused tools. Audit with GitHub's `copilot-token-audit` extension. Remove MCP servers you don't actively use. + +### Windows rendering issues + +Known platform-specific bugs with terminal flickering, scrolling, and keyboard input. Keep the tool updated; many are fixed in recent releases. + +--- + +## Links + +- [Product Page](https://github.com/features/copilot/cli) -- [Documentation](https://docs.github.com/en/copilot/how-tos/copilot-cli/use-copilot-cli/overview) -- [GitHub](https://github.com/github/copilot-cli) -- [npm](https://www.npmjs.com/package/@github/copilot) -- [Plans & Pricing](https://github.com/features/copilot/plans) diff --git a/docs/tools/pi.md b/docs/tools/pi.md new file mode 100644 index 0000000..c5aa96d --- /dev/null +++ b/docs/tools/pi.md @@ -0,0 +1,123 @@ +# [Pi Coding Agent](https://pi.dev/) + +Pi is a minimal, open-source, BYOK (bring your own key) terminal-based AI coding agent created by Mario Zechner. It ships with only four built-in tools (read, write, edit, bash) and a ~300-word system prompt, making it one of the leanest coding agents available. Everything else is added through extensions. + +## Why Pi + +- **Multi-provider** -- supports 20+ LLM providers (Anthropic, OpenAI, Google, xAI, DeepSeek, Mistral, Groq, and more). +- **Subscription login** -- `/login` works with Claude Pro/Max, ChatGPT Plus/Pro, and GitHub Copilot subscriptions (no API key needed). +- **Extensible** -- skills, prompt templates, themes, and hooks are bundled as Pi packages and shared via npm or git. +- **Session tree** -- every message has an id and parentId; `/tree` navigates, `/fork` branches a new session from the current point. +- **Multiple modes** -- interactive, print/JSON, RPC, and SDK (for embedding Pi in other tools). +- **Auto-compaction** -- context is automatically compacted when approaching limits. +- **Project instructions** -- uses `AGENTS.md` files (in `~/.pi/agent/`, parent dirs, or CWD). + +## Trade-offs + +- Minimal core means you **need extensions** for features like sub-agents, permissions, or plan mode. +- Younger ecosystem -- fewer extensions and less community content than Claude Code or Cursor. +- Some rough edges remain: agent can hang on provider switches mid-session, `grep`/`diff` exit-code 1 is misreported as an error. + +--- + +## Quickstart + +```bash +brew install pi-coding-agent +``` + +Or via npm: + +```bash +npm install -g @earendil-works/pi-coding-agent +``` + +Verify the install: + +```bash +pi --version +pi --help +``` + +--- + +## Suggested Extensions + +### [context-mode](https://github.com/mksglu/context-mode) + +Context window optimization for AI coding agents. Sandboxes tool output so raw bytes never enter the context window -- reducing consumption by up to 98%. Tracks edits, git operations, tasks, errors, and decisions in SQLite with FTS5 full-text search. On compaction, only BM25-ranked relevant content is retrieved. + +```bash +pi install npm:context-mode +``` + +### [pi-caveman](https://github.com/v2nic/pi-caveman) + +Ultra-compressed communication mode that cuts ~75% of output tokens while preserving full technical accuracy. Based on [caveman by Julius Brussee](https://github.com/JuliusBrussee/caveman). Provides `/caveman`, `/caveman-commit`, `/caveman-review`, `/caveman-help`, and `/caveman:compress` commands. Supports multiple intensity levels and persists settings across sessions. + +```bash +pi install npm:pi-caveman +``` + +### [pi-subagents](https://github.com/tintinweb/pi-subagents) + +Claude Code-style autonomous sub-agents for Pi. Spawn specialized agents in isolated sessions, each with its own tools, system prompt, model, and thinking level. Run foreground or background, steer mid-run, resume completed sessions, and define custom agent types. Includes a live widget UI with spinners, tool activity, and token counts. + +```bash +pi install npm:@tintinweb/pi-subagents +``` + +### [pi-rtk-optimizer](https://github.com/MasuRii/pi-rtk-optimizer) + +Automatically rewrites bash tool commands to their [RTK](https://github.com/example/rtk) equivalents and compacts noisy tool output to reduce context window usage. Delegates rewrite decisions to the installed `rtk rewrite` command, keeping RTK as single source of truth. Includes a runtime guard when the `rtk` binary is unavailable. Slash commands: `/rtk show`, `/rtk verify`, `/rtk stats`, `/rtk reset`. + +```bash +pi install npm:pi-rtk-optimizer +``` + +### [pi-permission-system](https://github.com/MasuRii/pi-permission-system) + +Centralized, deterministic permission gates for tool, bash, MCP, skill, and special operations. Features tool filtering (hides disallowed tools before the agent starts), system prompt sanitization, runtime enforcement (block/ask/allow with UI confirmation), and bash command control. Supports project-local permission files. + +```bash +pi install npm:pi-permission-system +``` + +--- + +## Troubleshooting + +### Pi won't start after install + +**PATH not set.** The `pi` binary may not be on your PATH after `npm install -g`. Verify with `which pi` (macOS/Linux) or `where pi` (Windows). If missing, check `npm config get prefix` and add its `bin/` subdirectory to your PATH. + +### `pi install` fails with "not a valid git URL" + +**Relative path bug.** Running `pi install local-path` (without `./` prefix) makes Pi treat it as a git URL. Always use the explicit prefix: + +```bash +# Wrong +pi install my-extension + +# Correct +pi install ./my-extension +``` + +See [Issue #1426](https://github.com/badlogic/pi-mono/issues/1426). + +### Agent hangs or freezes + +Several known causes: + +- **Provider switch mid-session** -- switching LLM providers while a session is active can cause a hang. Restart the session. +- **Streaming never resolves** -- if the agent is stuck in a "working" state with no progress, interrupt with `Ctrl+C` and restart. +- **Offline/proxy issues** -- use `--offline` or `PI_OFFLINE=1` to disable startup network calls if behind a proxy. + +### Extension install issues + +- **Temp directory permissions** -- older versions used the OS temp directory for extension installs. Update Pi to a recent version which uses `~/.pi/agent/tmp/extensions` with `0700` permissions. +- **npm registry errors** -- make sure your npm registry is reachable. If behind a corporate proxy, configure npm with `npm config set proxy` / `npm config set https-proxy`. + +### grep/diff shows false errors + +Pi's bash tool treats `grep`/`diff` exit code 1 (meaning "no matches") as an error. This is a known issue ([#3051](https://github.com/earendil-works/pi/issues/3051)). The agent usually recovers, but may retry unnecessarily.