Skip to content

feat(ui): workspace-level skill-builder LLM config (closes #92)#96

Merged
initializ-mk merged 3 commits into
mainfrom
fix/issue-92-workspace-llm-config
Jun 3, 2026
Merged

feat(ui): workspace-level skill-builder LLM config (closes #92)#96
initializ-mk merged 3 commits into
mainfrom
fix/issue-92-workspace-llm-config

Conversation

@initializ-mk
Copy link
Copy Markdown
Contributor

Summary

Closes #92.

Pre-#92 the forge ui skill builder borrowed credentials from whichever agent the operator clicked into. It read the picked agent's forge.yaml + .env, called os.Setenv to push those values into the UI process env, applied a hardcoded model upgrade (gpt-4.1 for openai, claude-opus-4-6 for anthropic), and used them to call the LLM in-process. That model produced four problems:

  1. Empty workspaces had no answer. The skill builder needed an agent context to borrow credentials from.
  2. Cross-agent skill development was arbitrary. Picking one of several agents to "borrow from" determined billing/model/endpoint.
  3. The hardcoded codegen upgrade ignored OPENAI_BASE_URL. Any agent pointed at a custom OpenAI-compatible endpoint (OpenRouter, vLLM, litellm, self-hosted Kimi/Llama) got a request for gpt-4.1 against an endpoint that didn't host it.
  4. The UI process env was stomped when switching agents. Agent A's OPENAI_API_KEY would leak into the UI process, then agent B's would overwrite it.

This PR decouples the skill-builder LLM from any specific agent.

What landed

New forge-ui/uiconfig package (the foundation)

  • SkillBuilderConfig (persisted yaml/json shape) + SkillBuilderLLM (resolved runtime view).
  • LoadSkillBuilderLLM(workspaceDir, agentDir, envLookup) with three-tier precedence:
    1. <workspace>/.forge/ui.yaml — primary, per-workspace.
    2. ~/.forge/ui.yaml — fallback, operator's machine-wide.
    3. The picked agent's forge.yaml + .envdeprecated; emits a deprecation warning the UI surfaces.
  • SaveSkillBuilderLLM(workspaceDir, cfg) — persists to <workspace>/.forge/ui.yaml.
  • ValidateSkillBuilderConfig(cfg) — same rules apply to file load + HTTP write.
  • EnvLookupForWorkspace(workspaceDir) — file-first env-lookup layer that consults <workspace>/.forge/.env, falling back to OS env. Zero process-env mutation.
  • SetEnvFileValue(workspaceDir, name, value) — atomic writer (temp + rename) at mode 0600. Edits in place (no duplicate lines on rewrite), removes the key on empty value, auto-generates <workspace>/.forge/.gitignore containing .env, preserves operator-added gitignore lines.

Inline API key storage (Option 2 from the design discussion)

The Settings modal accepts the API key value via a password field. On save, the key is written to <workspace>/.forge/.env under the operator's api_key_env name (or the provider default). An auto-generated <workspace>/.forge/.gitignore protects the file from accidental commits. The key value never appears in ui.yaml and never appears in the GET response — only the persisted env-var name comes back. Submitting an empty api_key leaves the saved value untouched (for non-key field rotation).

Handler refactor

  • handleSkillBuilderProvider and handleSkillBuilderChat call the loader instead of reading per-agent .env or encrypted secrets. Zero os.Setenv calls remain. Credentials are threaded as request-scoped values via LLMStreamOptions.LLM.
  • SkillBuilderCodegenModel reduced to a Deprecated: no-op shim that returns the configured model verbatim. The operator's chosen model is used — no hardcoded gpt-4.1 / claude-opus-4-6 override.
  • forge-cli/cmd/ui.go llmStreamFunc simplified from ~120 lines of env-reading + ResolveModelConfig + OAuth + codegen-upgrade logic to ~30 lines that consume opts.LLM directly.

Settings HTTP API

  • GET /api/settings/skill-builder — current config + metadata for the form (source, has_key, providers list).
  • PUT /api/settings/skill-builder — validates + persists ui.yaml; persists api_key to .forge/.env if provided. The PUT body uses a wrapping skillBuilderSettingsRequest type that keeps api_key off the persisted YAML struct, so the value can never leak into ui.yaml.
  • GET /api/skill-builder/provider (path-less) for first-run detection in empty workspaces.

Frontend

  • New SkillBuilderSettingsModal component (preact/htm). Provider picker, model field, optional base URL (openai-only), api_key_env override, password field for the API key value with show/hide toggle.
  • Status banner surfaces the resolution source (workspace / user / agent_fallback / unset) and the deprecation warning when the fallback shim resolves.
  • First-run state shows "Workspace skill-builder LLM is not configured" → Configure button directly in the chat banner.

Regression tests

forge-ui/uiconfig — 18 tests:

  • Load + Save round-trip
  • workspace → user → agent_fallback precedence
  • ollama HasCredentials without a key
  • api_key_env override
  • envfile create + 0600 perm + auto-gitignore
  • in-place update (no duplicate lines)
  • empty-value removes the key
  • comments + other-keys preservation on rewrite
  • file-precedes-OS env precedence
  • missing-file fallback to OS env
  • gitignore idempotence
  • operator-gitignore line preservation

forge-ui — 19 settings + skill-builder tests:

  • PUT persists + 0600 perm + auto-gitignore + no leak into ui.yaml
  • PUT honors custom api_key_env name
  • PUT omitting api_key preserves existing key
  • PUT validation rejects bad provider / missing model
  • GET after PUT returns source: workspace
  • Agent-fallback returns deprecation warning + preserves configured model verbatim (no codegen upgrade)
  • Env-stomp regression — switching between two agents with different OPENAI_API_KEY values in their .env files does NOT leak either key into the UI process's env. Pre-feat(ui): workspace-level LLM config for skill builder (decouple build-time codegen from per-agent runtime credentials) #92 this would have failed.

Gates

  • go test -race -count=1 ./forge-ui/... ./forge-cli/cmd/... ./forge-cli/runtime/... — green
  • golangci-lint run — 0 issues across both modules
  • gofmt -l forge-ui/ forge-cli/ — clean
  • node -c forge-ui/static/app.js — clean

Decoupling rules the implementation pins

These are guaranteed by the tests above:

  • The skill builder LLM is independent of any agent's runtime LLM.
  • The skill builder never calls os.Setenv on the forge ui process.
  • The previous SkillBuilderCodegenModel mapping is gone — the operator's chosen model is used verbatim.
  • The API key value lives in <workspace>/.forge/.env (mode 0600, gitignored) and never in ui.yaml.

Acceptance criteria

  • Operator can mkdir empty && forge ui --dir empty and use the skill builder before scaffolding any agent.
  • Skill-builder LLM is independent of any agent's runtime LLM; changes don't require restarting forge ui.
  • Agent pointed at a custom OpenAI-compatible endpoint (provider: openai + OPENAI_BASE_URL) doesn't cause the skill builder to issue requests for gpt-4.1 against that endpoint.
  • Switching the selected agent in the UI doesn't modify the forge ui process's env (TestSkillBuilderProvider_DoesNotStompProcessEnvBetweenAgents pins this).
  • Status banner reflects the workspace-level config + resolution source.

Out of scope (deliberately deferred)

  • CLI helper forge ui set-llm — listed as optional in the issue body. Backend has the full API + persistence; CLI helper is a fast follow-up.
  • Dedicated first-run wizard route — the current implementation surfaces the configure prompt directly in the chat banner (simpler UX). A separate one-screen wizard is what the issue spec specs; recommend deferring until UX feedback indicates the inline prompt isn't enough.
  • Encrypted secret store integration — the ~/.forge/secrets.enc AES-256-GCM keyring (gated by FORGE_PASSPHRASE) is the obvious next step for operators who don't trust plain .env. The current implementation matches forge init's .env pattern; encrypted-store integration is a fast follow-up.

Test plan

  • go test -race -count=1 ./forge-ui/... ./forge-cli/cmd/... ./forge-cli/runtime/... green
  • golangci-lint run clean
  • gofmt -l clean
  • Manual: empty workspace → first-run banner → Configure → save → chat works (covered by TestPutSkillBuilderSettings_PersistsAndEchoes + TestPutSkillBuilderSettings_PersistsAPIKeyToEnvFile)
  • Manual: switch between agents A and B → UI process env doesn't absorb either agent's OPENAI_API_KEY (pinned by env-stomp regression test)
  • Manual: agent points at custom OpenAI-compatible endpoint → skill builder uses workspace model verbatim, not gpt-4.1 (pinned by TestSkillBuilderProvider_AgentFallback_PreservesConfiguredModel)
  • Operator-side smoke against a real workspace with a real key (only you can do this with real credentials)

Pre-#92 the forge ui skill builder borrowed credentials from whichever
agent the operator clicked into: it read the picked agent's forge.yaml +
.env, called os.Setenv to push those values into the UI process env,
applied a hardcoded model upgrade (gpt-4.1 for openai, claude-opus-4-6
for anthropic), and used them to call the LLM in-process. That model
produced four problems documented in the issue:

  1. Empty workspaces had no answer — the skill builder needed an agent
     context to borrow credentials from.
  2. Cross-agent skill development was arbitrary — picking one of several
     agents to borrow from determined billing/model/endpoint.
  3. The hardcoded codegen upgrade ignored OPENAI_BASE_URL — any agent
     pointed at a custom OpenAI-compatible endpoint (OpenRouter, vLLM,
     litellm, self-hosted Kimi/Llama) got a request for gpt-4.1 against
     an endpoint that didn't host it.
  4. The UI process env was stomped when switching agents — agent A's
     OPENAI_API_KEY would leak into the process, then agent B's would
     overwrite it.

This fix decouples the skill-builder LLM from any specific agent.

What landed:

  * NEW package forge-ui/uiconfig — workspace-level config loader with
    a three-tier resolution precedence:
      1. <workspace>/.forge/ui.yaml (primary, per-workspace)
      2. ~/.forge/ui.yaml (fallback, operator's machine-wide)
      3. The picked agent's forge.yaml + .env (deprecated; emits a
         deprecation warning the UI surfaces)
    Plus a forward-compat-ready Save function and a Validate function
    used by both the file loader and the HTTP write handler.

  * NEW EnvLookupForWorkspace + SetEnvFileValue in uiconfig — file-first
    env lookup layer (with OS env as fallback) and atomic writer for
    <workspace>/.forge/.env. SetEnvFileValue writes with mode 0600 via
    temp-file-and-rename, edits in place (no duplicate lines on
    rewrite), removes the key on empty value, and auto-generates a
    <workspace>/.forge/.gitignore containing ".env" so the secret file
    is auto-protected. Preserves operator-added gitignore lines.

  * Settings HTTP API at /api/settings/skill-builder (GET + PUT). PUT
    accepts an optional api_key field on the request body kept off the
    persisted YAML struct via a wrapping skillBuilderSettingsRequest
    type — the key value can never leak into ui.yaml. When present, the
    key is persisted to <workspace>/.forge/.env under the operator's
    api_key_env name (or the provider default). Path-less
    GET /api/skill-builder/provider added for first-run detection in
    empty workspaces (no agent picked yet).

  * forge-ui handler refactor — handleSkillBuilderProvider and
    handleSkillBuilderChat call the loader instead of reading per-agent
    .env / encrypted secrets. ZERO os.Setenv calls remain. Credentials
    are threaded as request-scoped values through LLMStreamOptions.LLM.

  * SkillBuilderCodegenModel reduced to a Deprecated: no-op shim that
    returns the configured model verbatim. The operator's chosen model
    is used — no hardcoded gpt-4.1 / claude-opus-4-6 override.

  * forge-cli/cmd/ui.go llmStreamFunc simplified from ~120 lines of
    env-reading + ResolveModelConfig + OAuth + codegen-upgrade logic to
    ~30 lines that consume opts.LLM directly. No more
    coreruntime.ResolveModelConfig coupling. OAuth path removed from
    the skill-builder code path — workspace config requires an explicit
    API key (the issue #83 OAuth-precedence guardrail still applies).

  * Frontend (forge-ui/static/app.js) — new SkillBuilderSettingsModal
    component (preact/htm). Provider picker, model field, optional base
    URL (openai-only), api_key_env override, password field for the API
    key value with show/hide toggle. Shows "(saved; leave blank to
    keep)" when a key is already persisted — submitting an empty key
    leaves the saved value untouched. Status banner surfaces the
    resolution source (workspace / user / agent_fallback / unset) and
    the deprecation warning when the fallback shim resolves.

  * docs/ui/skill-builder-llm.md — full configuration reference, three
    setup recipes (UI / file / API), status banner semantics table,
    trust-boundary explanation for why ui.yaml + .env are split.

Regression tests:

  forge-ui/uiconfig (18 tests):
    Load + Save round-trip; workspace > user > agent_fallback
    precedence; ollama-no-key-needed; api_key_env override; envfile
    create + 0600 perm + auto-gitignore; in-place update; empty-value
    removal; comments/other-keys preservation; file-precedes-OS
    precedence; missing-file fallback to OS env; gitignore idempotence;
    operator-gitignore preservation.

  forge-ui (19 settings + skill-builder tests):
    PUT persists + 0600 perm + auto-gitignore + no leak into ui.yaml;
    PUT honors custom api_key_env name; PUT omitting api_key preserves
    existing key; PUT validation rejects bad provider / missing model;
    GET after PUT returns workspace source; agent-fallback returns
    deprecation warning + preserves configured model verbatim (no
    codegen upgrade); env-stomp regression — switching between two
    agents with different OPENAI_API_KEY values in their .env files
    does NOT leak either key into the UI process's env.

go test -race ./forge-ui/... ./forge-cli/cmd/... ./forge-cli/runtime/...
golangci-lint and gofmt clean across both modules.
Cleans up stale per-agent skill-builder doc references after #92 and
applies the requested README title/lede update.

  docs/reference/web-dashboard.md  Rewrote the Skill Builder "How It
                                   Works" section: no longer claims the
                                   builder uses the agent's own LLM; no
                                   longer describes the hardcoded
                                   gpt-4.1 / claude-opus-4-6 codegen
                                   upgrade. Now points operators at
                                   docs/ui/skill-builder-llm.md for the
                                   workspace-LLM model + 3-tier
                                   resolution. API Endpoints table adds
                                   GET /api/skill-builder/provider
                                   (path-less, for first-run detection)
                                   and GET / PUT /api/settings/skill-
                                   builder with explicit notes on the
                                   inline-api_key flow and the
                                   <workspace>/.forge/.env destination.
                                   Architecture file listing adds
                                   handlers_settings.go and the
                                   uiconfig/ package.

  docs/skills/skills-cli.md        Replaced "uses the agent's own LLM
                                   provider" with a cross-link to the
                                   workspace-LLM page.

  README.md                        Title -> "Forge — An Open, Secure,
                                   Portable AI Agent Runtime for the
                                   Enterprise". Lede rewritten around
                                   Anthropic's Agent Skills standard and
                                   the SKILL.md -> portable agent ->
                                   deploy anywhere narrative.

Broken-link check on the edited pages and the workspace LLM doc
(README + 3 docs + ../ui/skill-builder-llm.md) passed.
…ation (#83)

PR #93 (issue #83) made `forge init` normalize `provider: custom` ->
`provider: openai` + OPENAI_BASE_URL / OPENAI_API_KEY at scaffold time
for OpenAI-compatible endpoints (OpenRouter, vLLM, litellm, self-hosted
Kimi/Llama). The generated forge.yaml has never carried provider: custom
since that landed, but the reference docs still listed "custom" as a
forge.yaml model.provider value, which is misleading.

  docs/reference/forge-yaml-schema.md  Drop "custom" from the
                                       model.provider option list.
                                       Add a short note explaining the
                                       OpenAI-compatible endpoint
                                       recipe and that the Custom
                                       wizard option normalizes to
                                       this shape.

  docs/reference/cli-reference.md      Keep "custom" as a valid
                                       --model-provider flag value
                                       (the CLI input still accepts
                                       it; the normalizer rewrites
                                       at scaffold time) but clarify
                                       inline that it's an alias for
                                       the OpenAI-compatible
                                       endpoint scaffold.

Both edits surfaced from the sync-docs pass on PR #96 as adjacent
doc-debt from #83.
@initializ-mk initializ-mk merged commit 7a66ce3 into main Jun 3, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(ui): workspace-level LLM config for skill builder (decouple build-time codegen from per-agent runtime credentials)

1 participant