Skip to content

feat(plugins): add replicate plugin to pull external data into the internal DB (#72)#234

Open
trongtruong110-ux wants to merge 1 commit into
outerbase:mainfrom
trongtruong110-ux:feat/external-source-replication-plugin
Open

feat(plugins): add replicate plugin to pull external data into the internal DB (#72)#234
trongtruong110-ux wants to merge 1 commit into
outerbase:mainfrom
trongtruong110-ux:feat/external-source-replication-plugin

Conversation

@trongtruong110-ux
Copy link
Copy Markdown

Problem

#72 asks for a way to pull data from an external data source (e.g. a Postgres on Supabase) into the internal Durable Object SQLite, so a StarbaseDB instance can act as a close-to-edge read replica.

Changes

Adds a new ReplicatePlugin (plugins/replicate/) — a pull-based replicator built to the requirements in the issue:

  • Per-table config — choose which tables to replicate, which columns to pull, and a cursorColumn for incremental polling. With no tables configured it discovers and full-syncs every base table from the external source.
  • Incremental, append-only polling — tables with a cursorColumn only pull rows newer than the last replicated value, tracked per-table in a tmp_replicate_cursors meta table.
  • Keyset pagination (O(n)) — incremental batches use WHERE cursor > ? ORDER BY cursor ASC LIMIT n, with the cursor advancing every batch, so large tables are not re-scanned. LIMIT/OFFSET re-walks every skipped row (O(n²)). Tables without a cursor column fall back to LIMIT/OFFSET full re-sync.
  • Bounded memory — rows are pulled one configurable batch at a time (batchSize, default 1000).
  • Schema mirroring — destination tables are created in the internal DB with column types inferred across the batch (so a NULL in the first row doesn't mistype a column).
  • RoutesPOST /replicate/run triggers a cycle; GET /replicate/status reports per-table cursor + last-run info. Both admin-only.
  • Interval pullingrunReplication() is public so it can be driven on a schedule from a CronPlugin callback.

Registered in dist/plugins.ts alongside the existing plugins.

Tests

plugins/replicate/index.test.ts covers incremental keyset pagination (asserting the cursor query advances and OFFSET is not used), full-resync OFFSET paging, value coercion (boolean / date / null), and the dataSource guards.

/claim #72

…ternal DB (outerbase#72)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant