feat: rewire add-data modal to use template RPCs (template-connectors v2, PR 3)#9329
Draft
royendo wants to merge 3 commits into
Draft
feat: rewire add-data modal to use template RPCs (template-connectors v2, PR 3)#9329royendo wants to merge 3 commits into
royendo wants to merge 3 commits into
Conversation
Switches the add-data flow from hand-written TypeScript schemas + client-side YAML compilation to runtime-served templates. The runtime is now the single source of truth for connector form metadata, secret extraction, env-var naming, and YAML formatting. Schema TS files and the legacy YAML compilers stay in place for now and get deleted in PR 4. - `generate-template.ts` (new): wraps `runtimeServiceGenerateFile` with `preview: true`, resolves `(driver, olap)` to a template name via an in-memory OLAP cache, and exposes `mergeEnvVars` so the save path can reuse the env-var names the backend has already de-conflicted. - `connector-schemas.ts`: adds `templateNameMap`, `registerTemplateSchema`, and a `populateSchemaCache` test seam. The static schema map and helper functions stay so existing call sites keep working — PR 4 drops the static side once nothing references it. - `AddDataFormManager.ts`: `computeYamlPreview` becomes async, four legacy branches collapse into two `GenerateFile` calls (connector vs source/ explorer). Drops imports of `compileConnectorYAML`, `compileSourceYAML`, `prepareSourceFormData`, `getSchemaSecretKeys`, `getSchemaStringKeys`. - `AddDataModal.svelte`: replaces the static `connectors` import with two `createRuntimeServiceListTemplates` queries — sources filtered by the active OLAP, OLAPs unfiltered. Each template's JSON Schema is registered in the cache as it arrives so the existing form renderer can resolve drivers that only exist as templates (e.g. `kafka`, `hudi` for ClickHouse). - `AddDataForm.svelte`: 150 ms debounced async preview with last-valid fallback so the YAML pane doesn't blank during typing. Drops the `onMount` `.env` blob fetch — env-var conflict resolution is now server-side. Wraps `paramsError` in a `max-h-32 scroll` container. - `add-data/manager/selectors.ts`: same `ListTemplates` pattern as `AddDataModal`; deduplicates entries that appear in both source and OLAP lists so `clickhouse` shows once. - `generate-template.spec.ts` (new): covers `mergeEnvVars` edge cases (empty file, 404, suffix conflicts, missing keys, query-cache invalidation order, error rethrow). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This was referenced Apr 28, 2026
Removes the 22 static `import { fooSchema } from ...templates/schemas/foo`
lines for source connectors. Source-connector schemas are now populated
exclusively by `registerTemplateSchema` calls when `ListTemplates`
resolves on modal/picker open. AI connectors (claude, openai, gemini)
and DuckLake stay as static imports because their flows are not driven
by the runtime templates registry — DuckLake in particular has
client-side composer logic in `ducklake-utils.ts` that needs the schema
synchronously.
Also removes the static `connectors` array (was derived from the static
schema map and had no live importers after PR 3).
Specs updated to seed via `populateSchemaCache`:
- `connector-schemas.spec.ts` — minimal fixture set covering postgres,
mysql, s3, gcs, azure, snowflake, bigquery, salesforce, sqlite,
clickhouse, duckdb, motherduck.
- `AddDataFormManager.spec.ts` — gcs + snowflake fixtures.
- `add-source-visibility.spec.ts` — gcs + snowflake fixtures.
- `FormValidation.test.ts` — imports the actual `s3-duckdb.json` from
`runtime/templates/definitions/`, keeping frontend validation in
sync with the backend source-of-truth (the v1 review pattern the
PRD called out as worth preserving).
This unblocks PR 4's deletion of the per-connector schema TS files.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Brings the ClickHouse-target source templates up to parity with the DuckDB ones: each connector form now has a real connector step, and the model SQL references a CH named collection by name instead of embedding credentials inline. The backend driver work to create these named collections at connector reconcile time lands separately; templates ship the new SQL shape ahead so reconcile parity has a target to test against. Per template, the changes are: - `x-category` switches from `sourceOnly` to the matching DuckDB category (`objectStore` for s3/gcs/azure/delta/hudi/iceberg, `sqlStore` for postgres/mysql/supabase/mongodb, `fileStore` for https; `kafka` stays `sourceOnly` since it has no real connector step yet). - Fields gain `x-step: connector` (creds, host/port/user/db, region/endpoint, headers) or `x-step: source` (path, table, collection, model name). - A new `connector` file output is added; for templates with an optional Public auth path or empty headers, the connector body is wrapped in `[[ if .config_props -]]...[[ end -]]` so it renders to whitespace and gets skipped. - Model SQL switches to the named-collection form: `s3(rill_<connector>, url='...')`, `postgresql(rill_<connector>, table='...')`, `azureBlobStorage(rill_<connector>, container='...', blob_path='...')`, etc. The Public branches keep the inline-URL form since no named collection exists. - The old `chFn` / inline-cred helpers fall out of the templates; credential materialization is the backend's job now. Backend support: - `runtime/templates/render.go`: skip files whose rendered blob is whitespace-only after `TrimSpace`. This is what makes the empty connector file disappear on the Public path. Also fixes the long-standing issue where DuckDB s3/gcs/azure templates emitted a meaningless `type: connector\ndriver: s3` file when the user picked Public auth — those three templates now wrap their connector body in the same `[[ if .config_props -]]` guard. Tests in `render_test.go` are updated to match the new shape: `TestRenderS3ClickHouseModel` now asserts both files render with the named-collection ref; a new `TestRenderS3ClickHouseModelPublic` covers the connector-skipped case; HTTPS and MySQL tests are reworked the same way. Naming convention: the named collection identifier is `rill_<connector_name>` (matches what the backend named-collection work will emit on connector reconcile). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 3 of 4 from
PRD-template-connectors-v2.md. Stacked on top of #9326 (and #9325). Wires the add-data modal to the newListTemplatesandGenerateFileRPCs so the runtime owns connector form schemas, secret extraction, and YAML generation. Schema TS files and the legacy YAML compilers stay in place — PR 4 deletes them once nothing references them.generate-template.ts(new) wrapsruntimeServiceGenerateFilewithpreview: true, resolves(driver, olap)to a template name through a per-instance OLAP cache, and exposesmergeEnvVarsso the save path can reuse the env-var names the backend has already de-conflicted.AddDataModal.svelteandadd-data/manager/selectors.tsreplace the staticconnectorsimport with twocreateRuntimeServiceListTemplatesqueries — sources filtered by the active OLAP, OLAPs unfiltered. Each template's JSON Schema is registered in the cache as it arrives so the existing form renderer can resolve drivers that only exist as templates (e.g.kafka,hudiwhen ClickHouse is the OLAP).AddDataFormManager.computeYamlPreviewbecomes async; four legacy branches collapse into twoGenerateFilecalls (connector vs source/explorer). Drops imports ofcompileConnectorYAML,compileSourceYAML,prepareSourceFormData,getSchemaSecretKeys,getSchemaStringKeys.AddDataForm.svelteswitches the YAML preview to a 150 ms debounced async call, keeps the last-valid blob on error so the pane doesn't blank during typing, and drops theonMount.envfetch (env-var conflict resolution is server-side now).paramsErroris wrapped in amax-h-32scroll container.generate-template.spec.tscoversmergeEnvVarsedge cases (empty file, missing.env, suffix conflicts, blank keys/values, query-cache invalidation order, error rethrow). All 53 specs in the changed area pass.Behavior changes worth a manual QA pass (carry-overs from the PRD):
tags: ["source", olap]. Combinations the runtime doesn't know about disappear from the picker. Worth verifying the filtered list matches what users see today..envconflict detection happens insideGenerateFile; the form no longer pre-fetches.envon mount. Same UX from the user's perspective for the common case.Checklist:
Developed in collaboration with Claude Code