feat(02-02): scout agent — YC-OSS fetcher, weighted lead scorer, dedup

.planning/phases/02-core-pipeline-scout-through-writer/.continue-here.md

@@ -0,0 +1,60 @@
+---
+phase: 02-core-pipeline-scout-through-writer
+task: 5
+total_tasks: 7
+status: in_progress
+last_updated: 2026-02-26T10:41:11.427Z
+---
+
+<current_state>
+Phase 2, plan 02-05 complete. Branch `feature/phase-02-plan-02-scout`, PR #7 open at https://github.com/coder-ishan/Ingot/pull/7.
+Plans 02-01 (schemas + profile agent) and 02-05 (writer agent) are done. Plans 02-02, 02-03, 02-04, 02-06, 02-07 are NOT done — their summaries don't exist.
+</current_state>
+
+<completed_work>
+
+- Plan 02-01: `src/ingot/models/schemas.py` — all 6 Pydantic output schemas (UserProfile, IntelBriefPhase1, IntelBriefFull, MatchResult, MCQAnswers, EmailDraft). Committed in a prior session.
+- Plan 02-05: `src/ingot/agents/writer.py` — full Phase 2 rewrite. Two agent factories (create_mcq_agent, create_writer_agent), MCQ optional flow, tone routing (hr/cto/ceo/default), CAN-SPAM footer builder, run_writer() with Email+FollowUp SQLite persistence.
+- README: Added "Agent architecture" section documenting deps-injection pattern and factory pattern.
+- STATE.md updated to reflect 02-05 complete.
+- All committed and pushed to `feature/phase-02-plan-02-scout`.
+
+Key decision made this session: **factory functions over module-level singletons** for agents, so Orchestrator injects model string from ConfigManager at runtime. Also: **deps injection over @agent.tool** for data the Orchestrator pre-loads (IntelBriefFull, MatchResult, UserProfile) — no extra LLM round-trips.
+
+</completed_work>
+
+<remaining_work>
+
+- Plan 02-02: (wave 1) — check plan file for what it builds
+- Plan 02-03: (wave 2, checkpoint/autonomous=false) — check plan file
+- Plan 02-04: (wave 2) — check plan file
+- Plan 02-06: (wave 4, checkpoint/autonomous=false) — Orchestrator runtime wiring; will call create_writer_agent(config.agents["writer"].model) and create_mcq_agent(config.agents["writer"].model) using factory pattern established in 02-05
+- Plan 02-07: (wave 5) — Scout agent
+
+Note: 02-02 and 02-03/04 were skipped this session (user targeted plan-05 directly). They still need execution before phase verification.
+
+</remaining_work>
+
+<decisions_made>
+
+- Agent factories over module-level singletons: `create_mcq_agent(model)` / `create_writer_agent(model)` accept model string at call time from ConfigManager. This is the pattern Orchestrator (02-06) should use for ALL agents.
+- Deps injection over tool calls: `WriterDeps` carries pre-loaded data. `@agent.tool` is reserved for dynamic lookups (web search, unknown DB rows). This decision documented in README.
+- `defer_model_check=True` on all PydanticAI agents — codebase convention, allows import without API key set.
+- Phase 1 `WriterAgent` class + `register_agent()` pattern was intentionally replaced (it was a stub). The new writer does NOT use `ingot.agents.base.AgentDeps` or `ingot.agents.registry`.
+
+</decisions_made>
+
+<blockers>
+
+- Plans 02-02, 02-03, 02-04 not yet executed — 02-06 (Orchestrator) depends on 02-03 and 02-04. If you execute 02-06 next, those dependencies may cause issues.
+- Context was at 81% when pausing — start fresh session for remaining plans.
+
+</blockers>
+
+<context>
+The phase is building the core pipeline: Scout → Research → Matcher → Writer → Orchestrator. We've done schemas (02-01) and Writer (02-05). The natural next targets are the remaining wave 1 plan (02-02) and then wave 2 (02-03, 02-04) before tackling the Orchestrator (02-06) which wires everything together. The factory pattern established in writer.py should be applied consistently to all other agents in this phase.
+</context>
+
+<next_action>
+Start with: `/gsd:execute-phase 2 plan-02` — execute plan 02-02 (wave 1, the remaining unstarted plan alongside 02-01). Then proceed through 02-03, 02-04 in order before tackling 02-06 (Orchestrator). Check each plan file first to understand what it builds.
+</next_action>

README.md

@@ -36,6 +36,41 @@ INGOT orchestrates seven specialised agents in a pipeline:
 
 ---
 
+## Agent architecture
+
+### Data flow: dependency injection over tool calls
+
+Agents receive all data they need through a typed `Deps` dataclass injected before the run — they do not call tools to fetch data mid-generation.
+
+```
+Orchestrator
+  │
+  ├─ loads IntelBriefFull, MatchResult, UserProfile from DB
+  ├─ constructs WriterDeps(intel_brief=..., match_result=..., user_profile=...)
+  └─ calls run_writer(deps, model=config.agents["writer"].model)
+         │
+         └─ writer_agent.run(..., deps=deps)
+               system_prompt injection reads deps directly — no tool calls
+```
+
+This means an agent like Writer never calls `load_intel_brief(lead_id)` during generation. The Orchestrator loads it once and passes it in. Benefits:
+
+- **No wasted round-trips** — tool calls require an extra LLM inference step for data the Orchestrator already has
+- **Deterministic** — all context is present from the first token; the LLM cannot skip a fetch or hallucinate a wrong ID
+- **Tool calls reserved for dynamic lookups** — if an agent genuinely needs to search the web or query an unknown row, `@agent.tool` is the right pattern; for pre-loaded pipeline data it is not
+
+### Agent factories
+
+Agents are not module-level singletons. Each agent is constructed by a factory function that accepts a model string:
+
+```python
+agent = create_writer_agent(config.agents["writer"].model)
+```
+
+The Orchestrator reads the model from `~/.ingot/config.json` at runtime and passes it to the factory. This means you can switch any agent to a different LLM backend (Ollama, GPT-4o, Claude Haiku) without touching agent code — just update `config.json` or re-run `ingot setup`.
+
+---
+
 ## Prerequisites
 
 | Requirement | Notes |

pyproject.toml

@@ -24,6 +24,11 @@ dependencies = [
     "structlog>=25",
     "aiosmtplib>=3",
     "aioimaplib>=2",
+    "PyMuPDF>=1.24",
+    "python-docx>=1.1",
+    "beautifulsoup4>=4.12",
+    "scikit-learn>=1.5",
+    "lxml>=5.0",
 ]
 
 [project.optional-dependencies]