SQL: Auto union Iceberg and Redpanda topic by kbatuigas · Pull Request #575 · redpanda-data/cloud-docs

kbatuigas · 2026-05-04T22:59:07Z

Description

This pull request adds comprehensive documentation for Redpanda SQL's new support for Iceberg catalogs and bridge queries, enabling users to query both live Redpanda topics and their Iceberg-committed history. It introduces reference and how-to content for creating, altering, and dropping Iceberg catalogs, details the new USING CATALOG clause for Redpanda catalogs, and provides a step-by-step guide for querying Iceberg-enabled topics.

New SQL statement documentation:

Added reference pages for CREATE ICEBERG CATALOG, ALTER ICEBERG CATALOG, and DROP ICEBERG CATALOG, including syntax, options (covering authentication and TLS), and usage examples. [1] [2] [3]
Updated navigation to include the new Iceberg catalog statement references.

Enhancements to Redpanda catalog documentation:

Documented the new USING CATALOG clause for CREATE REDPANDA CATALOG, which links a Redpanda catalog to an Iceberg catalog for bridge queries.
Added the pandaproxy_url option, required when using USING CATALOG, and provided an example of creating a linked catalog. [1] [2]

How-to guide for querying Iceberg-enabled topics:

Added a step-by-step guide describing how to set up storage, Iceberg, and Redpanda catalogs, map topics as SQL tables, and run bridge queries that span live and historical data. The guide also explains prerequisites and links to related reference content.

Resolves https://github.com/redpanda-data/documentation-private/issues/
Review deadline: 18 May

Page previews

Redpanda SQL > Query Data > Query Iceberg topics
Reference > Redpanda SQL Reference > Statements
CREATE ICEBERG CATALOG
ALTER ICEBERG CATALOG
DROP ICEBERG CATALOG
CREATE REDPANDA CATALOG > Create catalog linked to Iceberg catalog

Checks

New feature
Content gap
Support Follow-up
Small fix (typos, links, copyedits, etc)

netlify · 2026-05-04T22:59:11Z

✅ Deploy Preview for rp-cloud ready!

Name	Link
🔨 Latest commit	`6999a6c`
🔍 Latest deploy log	https://app.netlify.com/projects/rp-cloud/deploys/6a0bd9422dcba900083302a2
😎 Deploy Preview	https://deploy-preview-575--rp-cloud.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

coderabbitai · 2026-05-04T22:59:13Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e4121c13-f7b1-47bd-b67c-fc370f8f61e3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch DOC-2006-document-feature-auto-union-iceberg-and-redpanda-topic

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

kbatuigas · 2026-05-13T18:51:51Z

+    warehouse = 's3://lakehouse-data/',
+    auth_type = 'oauth2',
+    oauth2_client_id = '<client-id>',
+    oauth2_client_secret = '<client-secret>',


Would we tell users to create and reference secrets here in the same way we do for RP Cloud secrets such as for catalog credentials in the cluster config, like so? https://deploy-preview-575--rp-cloud.netlify.app/redpanda-cloud/manage/iceberg/use-iceberg-catalogs/#use-a-secret-in-cluster-configuration

Greketrotny · 2026-05-18T08:46:18Z

+
+// TODO: SME — confirm when REFRESH must be run on the linked Iceberg table. Source shows that if the Iceberg table's schema isn't refreshed, the query fails at planning time with: `Schema not found for Iceberg table '<table>'. Run: REFRESH <catalog>=><table>`. Confirm:
+//   - Is REFRESH required only the first time, or every time the Iceberg schema changes?
+//   - Is REFRESH required when new records are added to the Iceberg table (no schema change), or only on schema change?


REFRESH pertains to the schema/shape of the table only. The whole point of the bridge queries is to fetch all data without refreshing anything.

Greketrotny · 2026-05-18T08:48:51Z

+== Query live and historical records together
+
+// TODO: SME — confirm when REFRESH must be run on the linked Iceberg table. Source shows that if the Iceberg table's schema isn't refreshed, the query fails at planning time with: `Schema not found for Iceberg table '<table>'. Run: REFRESH <catalog>=><table>`. Confirm:
+//   - Is REFRESH required only the first time, or every time the Iceberg schema changes?


Currently it's advised to run REFRESH at any shape change from any side, as that's the view Oxla has and considers during preparing the query. However, I believe @Bixkog is working on performing the REFRESH on all tables automatically on creation of the catalogs, so the tables will be visible immediately after creating the catalogs. The refresh is needed on any subsequent shape change, though, anyway.

kbatuigas · 2026-05-19T05:09:38Z

+
+== Set up the Iceberg query catalogs
+
+You create three objects, in this order: a storage connection, an Iceberg catalog, and a Redpanda catalog that links to the Iceberg catalog. The storage and Iceberg-catalog options must match the cluster's xref:manage:iceberg/rest-catalog/index.adoc[REST catalog configuration] (endpoint, credentials, region).


@Greketrotny if you wanted to continue with default_redpanda_catalog, are you still required to execute CREATE STORAGE, and CREATE ICEBERG CATALOG? Or could you go straight to CREATE TABLE default_redpanda_catalog=>orders (and the storage connection and iceberg catalog are auto-linked under the hood) for an Iceberg-enabled topic?

kbatuigas force-pushed the DOC-2006-document-feature-auto-union-iceberg-and-redpanda-topic branch 2 times, most recently from 03d3421 to 3fd1c5a Compare May 11, 2026 23:06

kbatuigas force-pushed the rp-sql branch from 248d62d to 3c582b2 Compare May 13, 2026 18:10

kbatuigas force-pushed the DOC-2006-document-feature-auto-union-iceberg-and-redpanda-topic branch from 44f89d0 to 72c11f6 Compare May 13, 2026 18:49

kbatuigas commented May 13, 2026

View reviewed changes

kbatuigas requested a review from Greketrotny May 13, 2026 18:53

kbatuigas marked this pull request as ready for review May 13, 2026 18:53

kbatuigas requested a review from a team as a code owner May 13, 2026 18:53

kbatuigas requested a review from mattschumpert May 13, 2026 18:54

Greketrotny approved these changes May 14, 2026

View reviewed changes

Comment thread modules/reference/pages/sql/sql-statements/drop-iceberg-catalog.adoc

Greketrotny approved these changes May 15, 2026

View reviewed changes

Greketrotny reviewed May 18, 2026

View reviewed changes

Comment thread modules/reference/pages/sql/sql-statements/create-redpanda-catalog.adoc Outdated

Greketrotny reviewed May 18, 2026

View reviewed changes

kbatuigas force-pushed the rp-sql branch from e051360 to 1b9d587 Compare May 19, 2026 03:26

kbatuigas added 12 commits May 18, 2026 20:27

Start reference update for SQL / Iceberg catalog

e06ea19

Draft bridge queries doc

57157d0

Update query-iceberg-topics per SME feedback

7b410a5

Add new iceberg catalog statements

7e6f4e6

Clarify Iceberg benefit of querying data aged out of topic retention

41cc35c

Links to Iceberg catalog docs

4c29076

Update example names

15a37f3

Update based on new-features list

3e917bb

Sync options from source

29cc2c3

Apply suggestions from SME feedback

6406d13

Apply suggestions

14831ec

Note about default schema

6999a6c

kbatuigas force-pushed the DOC-2006-document-feature-auto-union-iceberg-and-redpanda-topic branch from 95b585a to 6999a6c Compare May 19, 2026 03:30

kbatuigas commented May 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SQL: Auto union Iceberg and Redpanda topic#575

SQL: Auto union Iceberg and Redpanda topic#575
kbatuigas wants to merge 12 commits into
rp-sqlfrom
DOC-2006-document-feature-auto-union-iceberg-and-redpanda-topic

kbatuigas commented May 4, 2026 •

edited

Loading

Uh oh!

netlify Bot commented May 4, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented May 4, 2026 •

edited

Loading

Review skipped

Uh oh!

kbatuigas May 13, 2026

Uh oh!

Uh oh!

Uh oh!

Greketrotny May 18, 2026

Uh oh!

Greketrotny May 18, 2026

Uh oh!

kbatuigas May 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		== Set up the Iceberg query catalogs

		You create three objects, in this order: a storage connection, an Iceberg catalog, and a Redpanda catalog that links to the Iceberg catalog. The storage and Iceberg-catalog options must match the cluster's xref:manage:iceberg/rest-catalog/index.adoc[REST catalog configuration] (endpoint, credentials, region).

Conversation

kbatuigas commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Page previews

Checks

Uh oh!

netlify Bot commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for rp-cloud ready!

Uh oh!

coderabbitai Bot commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

kbatuigas May 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Greketrotny May 18, 2026

Choose a reason for hiding this comment

Uh oh!

Greketrotny May 18, 2026

Choose a reason for hiding this comment

Uh oh!

kbatuigas May 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kbatuigas commented May 4, 2026 •

edited

Loading

netlify Bot commented May 4, 2026 •

edited

Loading

coderabbitai Bot commented May 4, 2026 •

edited

Loading