feat(plugins): add replicate plugin to pull external data into the internal DB (#72)#234
Open
trongtruong110-ux wants to merge 1 commit into
Conversation
…ternal DB (outerbase#72) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
#72 asks for a way to pull data from an external data source (e.g. a Postgres on Supabase) into the internal Durable Object SQLite, so a StarbaseDB instance can act as a close-to-edge read replica.
Changes
Adds a new
ReplicatePlugin(plugins/replicate/) — a pull-based replicator built to the requirements in the issue:cursorColumnfor incremental polling. With no tables configured it discovers and full-syncs every base table from the external source.cursorColumnonly pull rows newer than the last replicated value, tracked per-table in atmp_replicate_cursorsmeta table.WHERE cursor > ? ORDER BY cursor ASC LIMIT n, with the cursor advancing every batch, so large tables are not re-scanned.LIMIT/OFFSETre-walks every skipped row (O(n²)). Tables without a cursor column fall back toLIMIT/OFFSETfull re-sync.batchSize, default 1000).NULLin the first row doesn't mistype a column).POST /replicate/runtriggers a cycle;GET /replicate/statusreports per-table cursor + last-run info. Both admin-only.runReplication()is public so it can be driven on a schedule from aCronPlugincallback.Registered in
dist/plugins.tsalongside the existing plugins.Tests
plugins/replicate/index.test.tscovers incremental keyset pagination (asserting the cursor query advances andOFFSETis not used), full-resync OFFSET paging, value coercion (boolean / date / null), and thedataSourceguards./claim #72