refactor: decompose large monolithic modules into focused packages

[dimakis]

Summary

• Decompose 8 oversized Python modules (architect.py, data_scientist.py, llm.py, mlflow.py, specialist_tools.py, openai_compat, ha_registry, agents routes) into well-organized packages with single-responsibility modules
• Extract UI components into dedicated directories (diagnostics tabs, architecture panels, dashboard cards, inline-assistant, chat hooks)
• Resolve all mypy type errors introduced by decomposition (92 -> 0) with proper type annotations, generics, and facade re-exports for backward compatibility

Motivation

Several core modules had grown beyond 800-1200 lines, making navigation, testing, and code review difficult. This refactor improves maintainability without changing any behavior.

Type of Change

• refactor — Code restructuring (no feature or fix)

Changes

Backend decomposition (8 modules -> packages):

• src/agents/architect.py -> src/agents/architect/ (agent, workflow, proposals, review, tools, entity_context, stream_event)
• src/agents/data_scientist.py -> src/agents/data_scientist/ (agent, collectors, constants, prompts, suggestions, workflow)
• src/llm.py -> src/llm/ (factory, resilient, circuit_breaker, usage)
• src/tracing/mlflow.py -> src/tracing/ (init, tracer, spans, runs, logging, feedback)
• src/tools/specialist_tools.py -> split into routing, runners, strategies, consult modules
• src/api/routes/openai_compat.py -> package (handlers, streaming_filter, schemas, utils)
• src/api/routes/ha_registry.py -> package (automations, scenes, scripts, services, summary, sync, helpers)
• src/api/routes/agents.py -> package (core, config_versions, prompt_versions, schemas, seed, serializers)

Additional extractions:

• Analyst mixins (config, discussion, persistence) from base_analyst
• Graph state split into submodules (base, conversation, analysis, dashboard, discovery, enums, etc.)
• HA automation schema split into triggers, conditions, actions

UI decomposition:

• Diagnostics page -> 5 tab components
• Architecture page -> graph + panel components
• Dashboard -> card components
• InlineAssistant -> hook + input + messages + types
• Chat -> header + hooks

Backward compatibility:

• All original module paths preserved as facade re-exports (__init__.py files re-export public symbols)
• No import changes required by consumers

Type safety:

• Fixed 92 mypy errors across 25 files introduced by decomposition
• Added return type to _safe_import_mlflow (eliminated 23 cascade errors)
• Added type stubs to analyst mixins
• Unified StreamEvent via canonical re-export
• Proper generic type params, enum conversions, and casts throughout

Testing

• Unit tests added/updated
• make ci-local passes (ruff format, ruff check, mypy 0 errors/269 files, bandit, 2717 tests pass)

Security Considerations

• No security implications

This is a pure structural refactor. No behavioral changes, no new endpoints, no auth/crypto changes.

Checklist

• My code follows the project's style (ruff, mypy pass)
• I've written tests that prove my fix/feature works
• I've updated documentation if needed
• I've added conventional commit messages
• Coverage has not decreased
• CI passed locally (make ci-local)
• Branch commits squashed into a single conventional commit
• Ready for rebase-merge

In general, the fix is to avoid sending raw exception messages to the client. Instead, log the detailed error server-side and return a generic, sanitized message over SSE that does not leak internal details. This keeps observability for developers while protecting users from seeing stack traces or sensitive provider/backend messages.

Concretely, the best change with minimal behavioral impact is:

• Keep the existing _format_sse_error helper and the SSE error shape.
• Change _stream_chat_completion’s except block so it:
  • Logs the full exception (including stack trace) using logging.exception.
  • Sends a generic error string (e.g. "An internal error occurred during streaming.") to _format_sse_error instead of str(e).

This only affects:

• src/api/routes/openai_compat/handlers.py: update the except Exception as e: block at the end of _stream_chat_completion and add an import for logging.
  No changes are needed in routes.py or utils.py beyond this, because _format_sse_error will no longer receive sensitive messages.

To implement:

• Add import logging alongside the existing imports in handlers.py.
• Change the except block:

    except Exception as e:
        import logging

        logging.exception("Unhandled error in _stream_chat_completion")
        yield _format_sse_error("An internal error occurred during streaming.")

This preserves the SSE structure (data: { "error": { "message": ..., "type": "api_error" } }) and the client contract, but replaces the sensitive content with a safe, generic message.