Skip to content

Add two-step HTTPS form with headers support #8780

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
lovincyrus wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Add two-step HTTPS form with headers support #8780

lovincyrus wants to merge 10 commits into from

Conversation

@lovincyrus
Copy link
Contributor

@lovincyrus lovincyrus commented Feb 5, 2026 •
edited
Loading

This PR adds headers support to the HTTPS connector by turning it into a two-step form. Step 1 collects HTTP headers (e.g., Authorization: Bearer ), and step 2 collects the URI and model name. Headers entered in the connector form are written to the connector YAML as a map, which the existing create_secrets_from_connectors bridge translates into DuckDB EXTRA_HTTP_HEADERS secrets at query time — no backend changes required.

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

@lovincyrus
feat(HTTPS): Add two-step connector form with headers support
0bec1fa 
Enable HTTP headers configuration on the HTTPS connector through a two-step form:
- Step 1 (connector): Headers textarea to specify custom HTTP headers
- Step 2 (source): URI and model name for the data source

Updates:
- Extended isMultiStepConnector() to detect schemas with both connector and source steps, enabling multi-step flow for HTTPS
- Added formatHeadersAsYamlMap() to parse textarea headers into YAML map format (Header: value per line)
- Updated HTTPS schema with headers field on connector step, URI and model name on source step
- Fixed AddDataModal to recognize multi-step HTTPS connector and use connector form type
- Made headers a required field for validation
@lovincyrus
refactor(HTTPS): Remove redundant inline comment
f55999d
@lovincyrus lovincyrus requested a review from royendo February 5, 2026 01:57
@lovincyrus lovincyrus self-assigned this Feb 5, 2026
@lovincyrus lovincyrus marked this pull request as ready for review February 5, 2026 01:57
@royendo
Copy link
Contributor

royendo commented Feb 5, 2026

Screenshot 2026-02-05 at 9 21 30 Not seeing the following format in preview and the text area is acting weird, especially if copy/paste 
type: connector 
driver: https 
headers:
    "Authorization": "Bearer {{.env.connector.https.token}}" # HTTP headers to include in the reque

wasnt able to test what it would look like from modal but should be

# Model YAML
# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/https

type: model
materialize: true

connector: duckdb

create_secrets_from_connectors: https
sql: |
  select * from read_json('xxx;) #default JSON unless other format detected

royendo
royendo previously requested changes Feb 5, 2026
View reviewed changes
Copy link
Contributor

@royendo royendo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

lovincyrus and others added 3 commits February 5, 2026 11:22
@lovincyrus @claude
fix(HTTPS): Fix YAML preview format and textarea copy/paste behavior
2530246 
- Replace contenteditable div with native <textarea> for proper
  copy/paste support and input event handling
- Update YAML preview to use 4-space indentation, quoted header keys,
  and inline description comment
- Widen event handler types from HTMLDivElement to HTMLElement

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovincyrus
prettier
b22c935
@lovincyrus
fix(HTTPS): Strip existing quotes from header input in YAML preview
d0c5833 
When users paste headers with existing quotes like "Authorization": "Bearer token",
they were getting double-quoted in the preview. Now we strip leading/trailing quotes
before wrapping them in the YAML output.
@lovincyrus lovincyrus requested a review from royendo February 5, 2026 20:09
@lovincyrus
Copy link
Contributor Author

lovincyrus commented Feb 5, 2026

Screenshot 2026-02-05 at 9 21 30 Not seeing the following format in preview and the text area is acting weird, especially if copy/paste

type: connector 
driver: https 
headers:
    "Authorization": "Bearer {{.env.connector.https.token}}" # HTTP headers to include in the reque

wasnt able to test what it would look like from modal but should be

# Model YAML
# Reference documentation: https://docs.rilldata.com/developers/build/connectors/data-source/https

type: model
materialize: true

connector: duckdb

create_secrets_from_connectors: https
sql: |
  select * from read_json('xxx;) #default JSON unless other format detected

Addressed

@royendo
Copy link
Contributor

royendo commented Feb 5, 2026

  • UI either auto expand or make it larger
Screenshot 2026-02-05 at 16 23 21

Test and Connect took a few seconds, are we actually testing anything for this connector type?

  • token added as raw text
    "Authorization": "Bearer token" # HTTP headers to include in the request

@lovincyrus
prettier
2ac8ade
@lovincyrus
feat(HTTPS): Replace textarea headers with structured key-value input
20f6c50 
Replace the textarea-based HTTP headers input with a dynamic key-value pair component. Users can now add/remove headers individually instead of typing raw "Header-Name: value" text.

- New KeyValueInput.svelte component with add/remove rows and duplicate detection
- Add "key-value" to x-display type union for form schema system
- Wire KeyValueInput through SchemaField.svelte renderer
- Update formatHeadersAsYamlMap to accept array of {key, value} objects
- Initialize key-value form fields to empty arrays
- Add wildcard support to schema button labels
- Change HTTPS connector button from "Test and Connect" to "Continue" (Ping is a no-op)
- Headers field is now optional in HTTPS schema
@lovincyrus
Copy link
Contributor Author

lovincyrus commented Feb 6, 2026

Latest demo:

  • Added a key value field
  • Changed cta to "Continue" instead of "Test and Connect" for HTTPS Connector
CleanShot.2026-02-06.at.10.40.04.mp4

@royendo
Copy link
Contributor

royendo commented Feb 6, 2026

love the new layout, Tokens generally should be hidden in the .env and not in raw text, anyway to make the string after Bearer become '{{ .env.connector.https/token}}' @lovincyrus

@lovincyrus @claude
fix(HTTPS): Store header values in .env instead of raw text in YAML
edbebb5 
Header values (e.g., Bearer tokens) are now stored in .env and
referenced via {{ .env.connector.https.<key> }} in the connector YAML,
preventing secrets from being exposed in configuration files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovincyrus
Copy link
Contributor Author

lovincyrus commented Feb 6, 2026

Header values are now stored in .env and referenced via {{ .env.connector.https.<header_key> }} in the connector YAML.

For example, entering Authorization: Bearer mytoken in the form will produce:

Connector YAML:

headers:
    "Authorization": "{{ .env.connector.https.authorization }}"

.env:

connector.https.authorization=Bearer mytoken

lovincyrus and others added 2 commits February 6, 2026 11:15
@lovincyrus @claude
fix(HTTPS): Only store sensitive header values in .env
f570a0b 
Non-sensitive headers (e.g., Content-Type, Accept) remain as plain text
in the connector YAML. Only headers matching known sensitive patterns
(Authorization, API keys, tokens) are stored in .env.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovincyrus @claude
fix(HTTPS): Replace Tailwind hover: modifiers with CSS :hover selectors
21aec89 
The CI Svelte parser does not support Tailwind modifier syntax (e.g.
hover:text-fg-primary) inside @apply directives. Use standard CSS
:hover pseudo-selectors instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovincyrus
Copy link
Contributor Author

lovincyrus commented Feb 6, 2026

Latest

CleanShot 2026-02-06 at 11 21 08@2x

@royendo
Copy link
Contributor

royendo commented Feb 6, 2026

Idk if its possible but can we separate Bearer and token?

@lovincyrus
Copy link
Contributor Author

lovincyrus commented Feb 6, 2026

Idk if its possible but can we separate Bearer and token?

We shouldn't do that. Headers should be a key value pair.

@royendo
Copy link
Contributor

royendo commented Feb 9, 2026

Separate it meaning BEARER is plain text and TOKEN is wrapped as an environment variable.

For HTTPS auth, the wire format is:

Authorization: Bearer <token>

Bearer is protocol syntax, not part of the token itself.

to make the string after Bearer become '{{ .env.connector.https.token}}'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ericpgreen2 ericpgreen2 Awaiting requested review from ericpgreen2

@royendo royendo Awaiting requested review from royendo

At least 1 approving review is required to merge this pull request.

Assignees

@lovincyrus lovincyrus

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lovincyrus @royendo