Centralize agent logging behind scoped callbacks and remove artifact_root tramp data

[luisorofino]

What does this PR do?

Reworks how the ddev/ai agent framework does run logging, replacing path-threading with a single, identity-aware, callback-driven logger. No user-facing behavior changes — this is an internal architecture refactor of the agent runtime.

Highlights:

• New AgentScope (agent/scope.py) — a small (owner_id, AgentRole) value carrying explicit agent identity, where AgentRole is one of PHASE, SUBAGENT, or GOAL_REVIEWER. This identity now travels with each event instead of being reconstructed from a path.
• Scoped callbacks (callbacks/callbacks.py) — agent-tier callbacks now carry an AgentScope. Added on_agent_start, renamed on_complete→on_agent_finish and on_error→on_agent_error, and added Callbacks.with_set(...) for composition.
• Run-wide AgentLogger (runtime/agent_log.py) — one long-lived observer, created once by the orchestrator, that demultiplexes the callback stream into one append-only JSONL file per agent keyed by AgentScope. It is the only thing that knows the log root.
• ReActProcessFactory (react/factory.py) — binds the runtime builder and the run-wide Callbacks once and exposes create(scope, agent_config, system_prompt). It injects itself into the runtime it builds so a nested spawn_subagent can create properly-scoped children. Replaces the loose runtime_builder + callbacks primitives that callers previously had to re-assemble.
• ReActProcess (react/process.py) — now requires a scope, fires on_agent_start, stamps every callback with its scope, and gained run_once(...) for single-turn calls so phases no longer hand-fire callbacks for memory steps.
• Simplified spawn_subagent — no more manual path/log orchestration; it just derives a child AgentScope and asks the injected factory for a scoped child process.
• Deleted artifact_root everywhere and removed the old per-agent tools/agents/agent_logger.py.

Motivation

The previous design had a few related smells:

1. artifact_root tramp data. A filesystem path was threaded through the orchestrator → resources → phases → agent build → tool registry → spawn_subagent, purely so each layer could reconstruct a per-agent logging directory. Most layers didn't use the path; they just passed it deeper. Classic tramp data.
2. Manual, per-agent logging. Each agent/subagent built its own logger and path, and phases/tools fired log writes by hand. Logging concerns leaked into business logic — spawn_subagent orchestrated directories and log lines, and phases manually logged memory steps.
3. Identity-blind callbacks. Callbacks didn't carry which agent an event belonged to, so a single run-wide observer couldn't attribute events. That's what forced per-agent loggers and the path threading in the first place.
4. Loose wiring. Layers received runtime_builder and callbacks as separate primitives and had to re-derive how to construct child processes, duplicating setup.

This PR pushes identity into the event stream (AgentScope) and centralizes logging behind callbacks, so the path lives in exactly one place (the orchestrator) and everything downstream is identity-aware and path-free.

Notes / follow-ups

The code is intentionally not perfect yet — this is a step, not the destination.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[datadog-datadog-prod-us1-2]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 97.28%
• Overall Coverage: 88.10% (+0.03%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0484d7a | Docs | Datadog PR Page | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 480793d456

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Report run_once send failures to scoped callbacks

When the single-turn path is used for the phase memory step, an agent.send() failure here propagates without firing fire_agent_error, unlike start(). In that scenario the run-wide AgentLogger never writes the error/failed-finish records for the phase, so the centralized agent log shows the last successful ReAct loop but omits the failing memory LLM call that aborted the phase; wrap this send in the same BaseException notification pattern used by start().

Useful? React with 👍 / 👎.

[luisorofino]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Add max_timeout to PhaseOrchestrator's arguments #24010
• Add context flags to TaskConfig #23992
• Centralize agent logging behind scoped callbacks and remove artifact_root tramp data #23935 👈 (View in Graphite)
• Improve readability of run_tasks in AgenticPhase #23919
• Move AI orchestration runtime out of phases #23916
• Decouple AI agent runtime construction #23912
• Replace extra_init_kwargs with polymorphic Phase.build() factory and ResourceProvider #23889
• loa/openmetrics-ai-gen

This stack of pull requests is managed by Graphite. Learn more about stacking.

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run