feat: add runtime/templates package for declarative connectors (template-connectors v2, PR 1)#9325
Draft
royendo wants to merge 6 commits into
Draft
feat: add runtime/templates package for declarative connectors (template-connectors v2, PR 1)#9325royendo wants to merge 6 commits into
runtime/templates package for declarative connectors (template-connectors v2, PR 1)#9325royendo wants to merge 6 commits into
Conversation
Captures scope, splittable changes (EnvVarName/headers/icons), and sequenced PR plan derived from #8981. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
HTTPS headers ConfigProperty is form metadata only — should ship with the HTTPS template that uses it. ClickHouse url()/headers() SQL is inside runtime/templates/funcmap.go and headers.go, not a separate driver change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop the PR-A and PR-D split notes; land the HTTPS `headers` ConfigProperty and ClickHouse `url()`/`headers()` SQL helpers as part of PR 1 instead. Remove the DuckLake out-of-scope bullet since DuckLake never used template schemas. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ates Introduces a new package that loads connector and OLAP templates from embedded JSON definitions, validates them against a JSON Schema, and renders YAML output via Go `text/template` with `[[ ]]` delimiters. This replaces the hand-written TypeScript schemas and the Go-side template code used by the legacy add-data flow; the runtime becomes the single source of truth for connector form metadata, secret extraction, and YAML generation. Includes: - `registry.go` — loads `definitions/*/*.json` at process start (strict embed glob), preserves JSON property order for stable form rendering. - `render.go` — schema-driven property processing: `x-secret` extraction to `.env`, `x-omit-if-default` suppression, `x-step` splitting between connector and source outputs, header map handling. - `funcmap.go` — template helpers: `renderProps`, `duckdbSQL`, Azure URI decomposition, `s3ToHTTPS` / `gcsToHTTPS`, `clickhouseFormat`, `clickhouseURLSuffix`. `default` only supports positional syntax. - `env.go` — `ResolveEnvVarNameForKey` with `_1`/`_2` suffix-on-conflict policy and `.env` key reader. - `headers.go` — sensitive header detection, auth-scheme prefix split, env-var name derivation for HTTPS `headers` properties. - 36 JSON definitions: 6 OLAPs, 18 DuckDB-targeted source connectors, 12 ClickHouse-targeted source connectors. - `definitions/README.md` — documents the JSON Schema `x-*` extension dictionary and template helpers. No consumers yet; the new `ListTemplates` and `GenerateFile` RPCs land in a follow-up PR. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Working notes; not intended to ship. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… databricks source
Catches the templates package up to current `main`, which now ships
OLAP support for these drivers and a databricks-as-source path that
postdate the original v1 PR.
- `olap/databricks.json`, `olap/bigquery.json`, `olap/snowflake.json` —
OLAP connector forms backed by `runtime/drivers/{driver}/olap.go`.
- `olap/ducklake.json` — driver `duckdb` with the phase-1 single-field
ATTACH-clause form from `web-common/.../schemas/ducklake.ts`.
- `duckdb-models/databricks-duckdb.json` — Databricks as a DuckDB source
(per `feat: databricks as duckdb src` #9288).
- Removes `duckdb-models/clickhouse-duckdb.json`; ClickHouse is not a
source-into-DuckDB target.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This was referenced Apr 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First slice of the template-connectors restart (PR 1 of 4 from
PRD-template-connectors-v2.md). Lands the newruntime/templatespackage and the JSON template definitions, with no consumers — RPCs, frontend rewire, and TS deletions follow in subsequent PRs.runtime/templatespackage:Registry(loads embeddeddefinitions/*/*.json),Render(schema-driven property processing,x-secretenv-var extraction,x-stepsplitting, headers handling), and helpers infuncmap.goforduckdbSQL, Azure URI decomposition, S3/GCS HTTPS conversion, and ClickHouseurl()/headers()SQL.definitions/README.mddocuments thex-*extension dictionary and the template helper API so future connector authors don't have to reverse-engineer it.defaultValkeeps the v1 review hardening (positional-only) and the//go:embedglob is tightened todefinitions/*/*.jsonso editor swap files / docs can't slip in.Restart of #8981. The ported code came from
feat-template-connectorwith the embed-glob and README updates applied on top.Checklist:
Developed in collaboration with Claude Code