diff --git a/.codex/agents/document-reviewer.toml b/.codex/agents/document-reviewer.toml new file mode 100644 index 00000000..131904df --- /dev/null +++ b/.codex/agents/document-reviewer.toml @@ -0,0 +1,93 @@ +description = "Reviews PowerSync documentation against writing standards, technical accuracy, and Mintlify formatting. Use when asked to review, audit, or quality-check documentation pages or PRs." +developer_instructions = """ +# PowerSync Documentation Reviewer + +You are an experienced, pragmatic technical writer reviewing PowerSync documentation. Your job is to help maintain accuracy, clarity, and consistency — not to approve content unconditionally. + +**Rule #1**: If you want an exception to any rule, stop and get explicit permission. Breaking the letter or spirit of the rules is failure. + +## Working Relationship + +- Push back on content that violates standards — cite specific rules when you do +- Never rubber-stamp content. Give honest technical judgment +- Ask for clarification rather than assuming intent +- Never lie, guess, or make up information about the product +- If you're unsure whether something is technically accurate, say so + +## Review Checklist + +For each piece of content, check the following areas: + +### 1. Frontmatter + +- `title` is present and uses Title Case +- `description` is present, concise, and does not duplicate the opening paragraph +- No other required fields are missing + +### 2. Writing Standards + +- Second person ("you") throughout +- Active voice, present tense +- No promotional or marketing language ("breathtaking," "robust," "stands as a testament," etc.) +- No editorializing ("it's important to note," "in conclusion") +- No filler words in titles or descriptions ("Comprehensive," "Complete," "Significant") +- No "local-first" or "offline-first" — replace with outcome language: "responds instantly," "stays fully functional in poor network conditions," "responsive" +- No excessive use of "moreover," "furthermore," "additionally" +- Bold used sparingly — only for terms being defined or critical distinctions +- No em-dash connectors between clauses — two sentences instead +- Concepts written out in full sentences, not comma-separated shorthand + +### 3. Headings + +- Title Case throughout +- Hierarchy starts at H2 (H1 is the page title) +- No verb-first headings unless a procedural step + +### 4. Terminology + +| Use | Avoid | +|-----|-------| +| sync | synchronization | +| Postgres | PostgreSQL | +| partial sync | dynamic partial replication | +| PowerSync Service | powersync service | +| Sync Rules | sync rules | +| Sync Streams | sync streams | + +### 5. Code Examples + +- Language tag present on every code block +- No aliases in SQL unless required (self-joins, ambiguous columns) +- Realistic data — not `foo`, `bar`, `example.com` +- No real API keys or secrets +- No filename shown on Cloud/dashboard examples (only on self-hosted) + +### 6. Mintlify Components + +- `` for sequential procedures +- `` for platform-specific content +- SDK tab order: JS → Dart → Kotlin → Swift → .NET → Rust +- `` wrapping all images +- Callouts used appropriately (``, ``, ``, ``, ``) + +### 7. Links + +- Internal links use relative paths (`/sync/streams/overview`, not absolute URLs) +- No unverified external links + +### 8. Sync Streams and Sync Rules + +- Sync Streams are the default — flag any new content that teaches or adds Sync Rules examples +- If existing content shows both side by side, verify the examples return the same data with no mismatched filters + +## Output Format + +For each issue found, state: +1. The specific rule violated +2. The problematic text (quoted) +3. A suggested fix + +Group by section. Be concise — one line per issue where possible. + +At the end, give an overall assessment: **Approve**, **Approve with minor fixes**, or **Needs revision**.""" +name = "document-reviewer" diff --git a/.github/vale/config/vocabularies/PowerSync/accept.txt b/.github/vale/config/vocabularies/PowerSync/accept.txt index 47081d17..6b15e3a5 100644 --- a/.github/vale/config/vocabularies/PowerSync/accept.txt +++ b/.github/vale/config/vocabularies/PowerSync/accept.txt @@ -236,6 +236,7 @@ CloudWatch Cocoapods Cognito Coolify +Convex cron CSQLite Cursor @@ -310,4 +311,4 @@ Yjs Zod # Package names -drift_sqlite_async \ No newline at end of file +drift_sqlite_async diff --git a/architecture/powersync-service.mdx b/architecture/powersync-service.mdx index 92c6a181..518baad6 100644 --- a/architecture/powersync-service.mdx +++ b/architecture/powersync-service.mdx @@ -3,7 +3,7 @@ title: "PowerSync Service" description: "Understand the PowerSync Service architecture and how it replicates data and delivers real-time sync." --- -When we say "PowerSync instance" we are referring to an instance of the [PowerSync Service](https://github.com/powersync-ja/powersync-service), which is the server-side component of the sync engine responsible for the _read path_ from the source database to client-side SQLite databases: The primary purposes of the PowerSync Service are (1) replicating data from your source database (Postgres, MongoDB, MySQL, SQL Server), and (2) streaming data to clients. Both of these happen based on your [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)). +When we say "PowerSync instance" we are referring to an instance of the [PowerSync Service](https://github.com/powersync-ja/powersync-service), which is the server-side component of the sync engine responsible for the _read path_ from the source database to client-side SQLite databases: The primary purposes of the PowerSync Service are (1) replicating data from your source database (Postgres, MongoDB, MySQL, SQL Server or Convex), and (2) streaming data to clients. Both of these happen based on your [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)). ## Bucket System @@ -62,7 +62,7 @@ When a change occurs in the source database that affects a certain bucket (based ## Bucket Storage -The PowerSync Service persists the bucket state in durable storage: there is a pluggable storage layer for bucket data, and MongoDB and Postgres are currently supported as _bucket storage_ databases. The _bucket storage_ database is separate from the connection to your _source database_ (Postgres, MongoDB, MySQL or SQL Server). Our cloud-hosting offering (PowerSync Cloud) uses MongoDB Atlas as the _bucket storage_ database. +The PowerSync Service persists the bucket state in durable storage: there is a pluggable storage layer for bucket data, and MongoDB and Postgres are currently supported as _bucket storage_ databases. The _bucket storage_ database is separate from the connection to your _source database_ (Postgres, MongoDB, MySQL, SQL Server or Convex). Our cloud-hosting offering (PowerSync Cloud) uses MongoDB Atlas as the _bucket storage_ database. Persisting the bucket state in a database is also part of how PowerSync achieves high scalability: it means that the PowerSync Service can have a low memory footprint even as you scale to very large volumes of synced data and users/clients. @@ -87,7 +87,7 @@ When the PowerSync Service replicates data from the source database, it: Whenever a new version of Sync Streams/Sync Rules is deployed, initial replication takes place by means of taking a snapshot of all tables/collections they reference. -After that, data is incrementally replicated using a change data capture stream (the specific mechanism depends on the source database type: Postgres logical replication, MongoDB change streams, the MySQL binlog, or SQL Server Change Data Capture). +After that, data is incrementally replicated using a change data capture stream. The specific mechanism depends on the source database type: Postgres logical replication, MongoDB change streams, the MySQL binlog, SQL Server Change Data Capture, or Convex document deltas. ## Streaming Sync @@ -111,4 +111,3 @@ For more details on exactly how streaming sync works, see [PowerSync Protocol](/ The repo for the PowerSync Service can be found here: - diff --git a/configuration/app-backend/client-side-integration.mdx b/configuration/app-backend/client-side-integration.mdx index 2c9c4186..46f65e8b 100644 --- a/configuration/app-backend/client-side-integration.mdx +++ b/configuration/app-backend/client-side-integration.mdx @@ -10,7 +10,7 @@ After you've [instantiated](/intro/setup-guide#instantiate-the-powersync-databas | Purpose | Description | |---------|-------------| -| **Uploading mutations to your backend:** | Mutations that are made to the client-side SQLite database are uploaded to your backend application, where you control how they're applied to your backend source database (Postgres, MongoDB, MySQL, or SQL Server). This is how PowerSync achieves bi-directional syncing of data: The [PowerSync Service](/architecture/powersync-service) provides the _server-to-client read path_ based on your [Sync Streams or Sync Rules (legacy)](/sync/overview), and the _client-to-server write path_ goes via your backend. | +| **Uploading mutations to your backend:** | Mutations that are made to the client-side SQLite database are uploaded to your backend application, where you control how they're applied to your backend source database (Postgres, MongoDB, MySQL, SQL Server, or Convex). This is how PowerSync achieves bi-directional syncing of data: The [PowerSync Service](/architecture/powersync-service) provides the _server-to-client read path_ based on your [Sync Streams or Sync Rules (legacy)](/sync/overview), and the _client-to-server write path_ goes via your backend. | | **Authentication integration:** (optional) | PowerSync uses JWTs for authentication between the Client SDK and PowerSync Service. Some [authentication providers](/configuration/auth/overview#common-authentication-providers) generate JWTs for users which PowerSync can verify directly. For others, some code must be [added to your application backend](/configuration/auth/custom) to generate the JWTs. | @@ -180,4 +180,3 @@ For an example implementation of a PowerSync 'backend connector', see the SDK gu ## More Examples For additional implementation examples, see the [Examples](/intro/examples) section. - diff --git a/configuration/app-backend/setup.mdx b/configuration/app-backend/setup.mdx index 6825690a..58838647 100644 --- a/configuration/app-backend/setup.mdx +++ b/configuration/app-backend/setup.mdx @@ -8,7 +8,7 @@ PowerSync generally assumes that you have some kind of "backend application" as When you integrate PowerSync into your app project, PowerSync relies on that "backend application" for a few potential purposes: -1. **Allowing client-side mutations to be uploaded** and [applied](/handling-writes/writing-client-changes) to the backend source database (Postgres, MongoDB, MySQL, or SQL Server). When you write to the client-side SQLite database provided by PowerSync, those mutations are also placed into an [upload queue](/architecture/client-architecture#writing-data-via-sqlite-database-and-upload-queue). The PowerSync Client SDK manages uploading of those mutations to your backend using the `uploadData()` function that you define in your [client-side](/configuration/app-backend/client-side-integration) _backend connector_ implementation. Your `uploadData()` implementation should call your backend application API to apply the mutations to your source database. The reason why we designed PowerSync this way is to give you full control over things like server-side data validation and authorization of mutations, while PowerSync itself requires minimal permissions. +1. **Allowing client-side mutations to be uploaded** and [applied](/handling-writes/writing-client-changes) to the backend source database (Postgres, MongoDB, MySQL, SQL Server, or Convex). When you write to the client-side SQLite database provided by PowerSync, those mutations are also placed into an [upload queue](/architecture/client-architecture#writing-data-via-sqlite-database-and-upload-queue). The PowerSync Client SDK manages uploading of those mutations to your backend using the `uploadData()` function that you define in your [client-side](/configuration/app-backend/client-side-integration) _backend connector_ implementation. Your `uploadData()` implementation should call your backend application API to apply the mutations to your source database. The reason why we designed PowerSync this way is to give you full control over things like server-side data validation and authorization of mutations, while PowerSync itself requires minimal permissions. 2. **Authentication integration (optional):** _If_ you are implementing custom authentication (see below), your backend is responsible for securely generating the [JWTs](/configuration/auth/overview) used by the PowerSync Client SDK to authenticate with the [PowerSync Service](/architecture/powersync-service). If you will only use the backend for applying mutations and not for authentication, you can also use some kind of data API service or API platform (e.g. Hasura). @@ -56,4 +56,4 @@ You can use a serverless functions system like Azure Functions, AWS Lambda, Goog ### For MongoDB: PowerSync Hosted/Managed Option -For developers using MongoDB as a backend source database, an alternative option is to use CloudCode, a serverless cloud functions environment provided by a sibling product of PowerSync, that runs in the same cloud environment as PowerSync Cloud. We have a template that you can use as a turnkey starting point. See the [documentation](/configuration/app-backend/cloudcode). \ No newline at end of file +For developers using MongoDB as a backend source database, an alternative option is to use CloudCode, a serverless cloud functions environment provided by a sibling product of PowerSync, that runs in the same cloud environment as PowerSync Cloud. We have a template that you can use as a turnkey starting point. See the [documentation](/configuration/app-backend/cloudcode). diff --git a/configuration/source-db/connection.mdx b/configuration/source-db/connection.mdx index 287f844c..056e0143 100644 --- a/configuration/source-db/connection.mdx +++ b/configuration/source-db/connection.mdx @@ -1,13 +1,13 @@ --- title: "Source Database Connection" -description: "Connect a PowerSync Cloud instance to your Postgres, MongoDB, MySQL or SQL Server source database." +description: "Connect a PowerSync Cloud instance to your Postgres, MongoDB, MySQL, SQL Server or Convex source database." --- import SupabaseConnection from '/snippets/supabase-database-connection.mdx'; Each database provider has their quirks when it comes to specifying connection details, so we have documented database-specific and provider-specific instructions below: -Jump to: [Postgres](#postgres-provider-specifics) | [MongoDB](#mongodb-specifics) | [MySQL](#mysql-specifics) | [SQL Server](#sql-server-specifics) +Jump to: [Postgres](#postgres-provider-specifics) | [MongoDB](#mongodb-specifics) | [MySQL](#mysql-specifics) | [SQL Server](#sql-server-specifics) | [Convex](#convex-specifics) The below instructions are currently written for PowerSync Cloud. For self-hosted PowerSync instances, specify database connection details in the config file as documented [here](/configuration/powersync-service/self-hosted-instances#source-database-connections). @@ -279,4 +279,30 @@ PowerSync deploys and configures an isolated cloud environment for you, which ca Also see: -- [SQL Server Source Database Setup](/configuration/source-db/setup#sql-server) \ No newline at end of file +- [SQL Server Source Database Setup](/configuration/source-db/setup#sql-server) + + +## Convex Specifics + +Convex support is currently in an [Open Alpha release](/resources/feature-status). + +Before connecting PowerSync, add the required `powersync_checkpoints` table and mutation to your Convex deployment. See [Convex Source Database Setup](/configuration/source-db/setup#convex). + +1. In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance and go to **Database Connections**. +2. Click **Connect to Source Database** and ensure the **Convex** tab is selected. +3. Fill in your Convex connection details: + 1. **Name** can be any name for the connection. + 2. **Deployment URL** is your Convex deployment URL, for example `https://.convex.cloud`. + 3. **Deploy key** is the deploy key for the Convex deployment. In the Convex Dashboard, go to **Settings** → **General** and generate a deploy key. +4. Click **Test Connection** and fix any errors. +5. Click **Save Connection**. + +PowerSync deploys and configures an isolated cloud environment for you, which can take a few minutes to complete. + + + Convex deploy keys grant full read and write access to your Convex data. Use a deploy key for the correct environment, store it as a secret, and rotate it if it is exposed. + + +Also see: +- [Convex Source Database Setup](/configuration/source-db/setup#convex) +- [Type Mapping](/sync/types#convex-type-mapping) diff --git a/configuration/source-db/setup.mdx b/configuration/source-db/setup.mdx index 367c9707..a50db851 100644 --- a/configuration/source-db/setup.mdx +++ b/configuration/source-db/setup.mdx @@ -1,9 +1,9 @@ --- title: "Source Database Setup" -description: "Prepare your Postgres, MongoDB, MySQL or SQL Server database for PowerSync replication." +description: "Prepare your Postgres, MongoDB, MySQL, SQL Server or Convex database for PowerSync replication." --- -Jump to: [Postgres](#postgres) | [MongoDB](#mongodb) | [MySQL](#mysql) | [SQL Server](#sql-server) +Jump to: [Postgres](#postgres) | [MongoDB](#mongodb) | [MySQL](#mysql) | [SQL Server](#sql-server) | [Convex](#convex) import PostgresPowerSyncUser from '/snippets/postgres-powersync-user.mdx'; import PostgresPowerSyncPublication from '/snippets/postgres-powersync-publication.mdx'; @@ -371,7 +371,7 @@ To ensure that PowerSync can read the binary log, you need to configure your MyS - [`enforce_gtid_consistency`](https://dev.mysql.com/doc/refman/8.4/en/replication-options-gtids.html#sysvar_enforce_gtid_consistency): **`ON`**. Enforces GTID consistency. Default is **`OFF`**. - [`gtid_mode`](https://dev.mysql.com/doc/refman/8.4/en/replication-options-gtids.html#sysvar_gtid_mode): **`ON`**. Enables GTID based logging. Default is **`OFF`**. - [`binlog_format`](https://dev.mysql.com/doc/refman/8.4/en/replication-options-binary-log.html#sysvar_binlog_format): **`ROW`**. Sets the binary log format to row-based replication. This is required for PowerSync to correctly replicate changes. Default is **`ROW`**. -- [`binlog_row_image`](https://dev.mysql.com/doc/refman/8.4/en/replication-options-binary-log.html#sysvar_binlog_row_image): **`FULL`**. Captures the complete row data for each change. This is required for PowerSync to correctly replicate changes. Default is **`FULL`**. The `MINIMAL`/`NOBLOB` options will be supported in a future release. +- [`binlog_row_image`](https://dev.mysql.com/doc/refman/8.4/en/replication-options-binary-log.html#sysvar_binlog_row_image): **`FULL`**. Captures the complete row data for each change. This is required for PowerSync to correctly replicate changes. Default is **`FULL`**. The `MINIMAL`/`NOBLOB` options will be supported in a future release. These can be specified in a MySQL [option file](https://dev.mysql.com/doc/refman/8.4/en/option-files.html): @@ -440,7 +440,7 @@ binlog-ignore-db=user_db SQL Server support is currently in a [Beta release](/resources/feature-status). - **Version compatibility**: + **Version compatibility**: - PowerSync requires SQL Server 2019+ or Azure SQL Database. - SQL Server support was introduced in version 1.18.1 of the PowerSync Service. @@ -625,7 +625,7 @@ Recommended Capture Job settings: For Azure SQL Database, the CDC capture and cleanup jobs are managed automatically. Manual configuration is greatly limited. - See [Azure CDC Customization Limitations](https://learn.microsoft.com/en-us/azure/azure-sql/database/change-data-capture-overview?view=azuresql#cdc-customization). + See [Azure CDC Customization Limitations](https://learn.microsoft.com/en-us/azure/azure-sql/database/change-data-capture-overview?view=azuresql#cdc-customization). The main limitation is that the capture job polling interval cannot be modified and is fixed at 20 seconds. It is, however, still possible to [manually trigger](https://learn.microsoft.com/en-us/azure/azure-sql/database/change-data-capture-overview?view=azuresql#manual-cdc-control) the capture job on demand. @@ -649,6 +649,177 @@ Increasing this will increase throughput at the cost of increased memory usage. We are planning to expose these settings for SQL Server source database connections in the PowerSync Dashboard for PowerSync Cloud instances. +## Convex + + + Convex support is currently in an [Open Alpha release](/resources/feature-status). APIs, configuration, schema-change + handling, metrics, and replication behavior may change before this connector is considered stable. + + +PowerSync reads Convex data using the Convex Streaming Export API. Initial replication pins a single Convex snapshot cursor and snapshots each selected table at that cursor. Streaming replication then reads the global `document_deltas` stream and filters rows according to your Sync Streams. + +Convex does not support user-defined database schemas or namespaces in the same way as SQL databases. You do not need to qualify Convex table names in Sync Streams. If you do choose to qualify them, use the default `convex` schema. + +### Connection Requirements + +PowerSync can replicate from Convex Cloud or self-hosted Convex deployments. It requires: + +- A Convex deployment URL. +- A Convex deploy key. In the Convex Dashboard, go to **Settings** → **General** and generate a deploy key for the deployment PowerSync should replicate. +- The `powersync_checkpoints` table and `createCheckpoint` mutation described below. + + + Convex deploy keys grant full read and write access to your Convex data. Use a deploy key for the correct environment, + store it as a secret, and rotate it if it is exposed. + + +### Checkpoint Table + +PowerSync uses a small Convex table to generate write checkpoint markers. Convex table names cannot start with `_`, so the table is named `powersync_checkpoints`. + +Add the table to your Convex schema: + +```typescript convex/schema.ts +import { defineSchema, defineTable } from 'convex/server'; +import { v } from 'convex/values'; + +export default defineSchema({ + // ... your other tables + + powersync_checkpoints: defineTable({ + last_updated: v.float64() + }) +}); +``` + +### Checkpoint Mutation + +Deploy a Convex mutation named `powersync_checkpoints:createCheckpoint`. PowerSync calls this mutation after recording a write checkpoint so the Convex delta stream advances even when the app is otherwise idle. + +```typescript convex/powersync_checkpoints.ts +import { mutation } from './_generated/server'; + +export const createCheckpoint = mutation({ + args: {}, + handler: async (ctx) => { + const existing = await ctx.db.query('powersync_checkpoints').first(); + + if (existing) { + await ctx.db.patch(existing._id, { last_updated: Date.now() }); + } else { + await ctx.db.insert('powersync_checkpoints', { last_updated: Date.now() }); + } + } +}); +``` + +PowerSync excludes `powersync_checkpoints` from replicated source tables. The table exists only to advance the replication cursor for write checkpoint acknowledgements. + +### Client Writes + +PowerSync does not write application data directly to Convex. Your app still needs an upload path that takes queued client-side writes and applies them through Convex mutations. + +In most Convex apps, you already define one or more mutation functions for each writable table. Your PowerSync backend connector can call those same mutations from `uploadData()`. + +If you use Convex Auth tokens directly for PowerSync client authentication, configure PowerSync to accept the `convex` JWT audience. For self-hosted development this is configured in `client_auth.audience`; for PowerSync Cloud you can configure a custom audience in the instance settings. See [Custom Authentication](/configuration/auth/custom). + +### Schema Changes + +For Convex, most schema changes are document-shape changes rather than DDL events. PowerSync reads the JSON documents returned by Convex snapshots and document deltas, so added fields and type changes are replicated through normal Convex writes. + +If a Convex migration updates existing documents, those updates should appear in `document_deltas` and replicate like other writes. You do not need to redeploy your Sync Config merely because a field was added or changed type. + +If you remove a field from Convex documents, PowerSync stops replicating new values for that field. Previously synced values can remain on clients until the affected data is reprocessed. To completely remove a field from synced client data, update and redeploy your Sync Config. + +You also need to redeploy your Sync Config when you change which tables or fields are selected by Sync Streams, or when a Sync Streams deployment needs to include existing data that was not previously selected. + + + Dropping Convex tables has a known limitation. Deleting a table from the Convex dashboard does not emit per-document + delete rows in `document_deltas`, so previously synced rows for that table can remain on clients. Clear the table + before deleting it, or delete documents through Convex mutations that emit document delete deltas. Otherwise, treat the + table removal as a Sync Config deployment change and clear or re-replicate affected PowerSync state. + + +### Int64 and Bytes Values + +PowerSync syncs Convex `Int64` values as base-10 text values and Convex `Bytes` values as base64 text values. + +If you want a Convex `Int64` value to be synced as a SQLite integer, cast it to `INTEGER` in your Sync Streams. + +### Latency + +PowerSync polls the Convex `document_deltas` endpoint for changes. This means there is an inherent latency between a write being committed in Convex and that change being replicated into PowerSync. + +The polling interval defaults to once every 1000ms. When self-hosting PowerSync, you can configure this with the `polling_interval_ms` connection parameter: + +```yaml service.yaml +replication: + connections: + - type: convex + deployment_url: https://.convex.cloud + deploy_key: + polling_interval_ms: 1000 +``` + +Lowering this value can reduce replication latency, but it increases the number of requests made to Convex and the work performed by the PowerSync Service. + +### Mutation Transaction Atomicity + +Convex mutations are ACID transactions. When a mutation writes multiple documents, Convex exposes those writes in `document_deltas` with the same commit timestamp. PowerSync replicates all writes from the same mutation together as one batch, so clients do not observe a partial result from a single Convex mutation. + +### Sync Streams Examples + +You do not need to specify a schema when querying Convex tables in Sync Streams. + + + Convex document IDs are generated by Convex and are exposed as `_id`. Clients cannot create Convex IDs before + inserting documents into Convex. PowerSync clients need stable local IDs before writes are uploaded, so use a + client-generated UUID column as the synced `id` and keep `_id` as the Convex server-generated document ID. This is + similar to the pattern described in [Sequential ID Mapping](/client-sdks/advanced/sequential-id-mapping). + + +The client creates a UUID in its local `id` column before the write is uploaded. Your Convex mutation should store that +value in a separate `uuid` field on the Convex document. PowerSync then syncs `uuid AS id` back to the client, so the +client keeps the same stable local ID while Convex keeps its own server-generated `_id`. + +The example below uses one stream for a user's lists and todos: + +- `uuid` as the synced client-side `id` instead of the Convex `_id`. +- `list_uuid` as the synced relationship column instead of the Convex `list_id`. +- `CAST(an_int64_column AS INTEGER)` to sync a Convex `Int64` value as a SQLite integer. +- `substring(auth.user_id(), 1, 32)` to extract the Convex user ID from a Convex Auth JWT subject. Convex Auth subjects include the 32-character user ID followed by `|` and the user session ID. + +```yaml +config: + edition: 3 + +streams: + user_data: + with: + # Extract the Convex user ID from the JWT subject. + # Convex Auth subjects include `[32 character user ID]|[user session ID]`. + user_lists: | + SELECT uuid + FROM lists + WHERE archived != true + AND owner_id = substring(auth.user_id(), 1, 32) + auto_subscribe: true + queries: + - SELECT uuid AS id, name, owner_id FROM lists WHERE uuid IN user_lists + - | + SELECT + -- Use the client-generated todo UUID as the synced `id`. + uuid AS id, + description, + -- Map relationships that use Convex IDs, such as `list_id`, + -- to the related table's local UUID column. + list_uuid, + -- Cast Convex Int64 values to INTEGER when you want a SQLite integer. + CAST(an_int64_column AS INTEGER) AS an_int64_column + FROM todos + WHERE list_uuid IN user_lists +``` + ## Next Step Next, connect PowerSync to your database: diff --git a/handling-writes/writing-client-changes.mdx b/handling-writes/writing-client-changes.mdx index c79e8f62..60e70175 100644 --- a/handling-writes/writing-client-changes.mdx +++ b/handling-writes/writing-client-changes.mdx @@ -17,13 +17,13 @@ Since you get to define the client-side `uploadData()` function as you wish, you You can also use any API style you want — e.g. REST, GraphQL, gRPC, etc. -It's important that your API endpoint be blocking/synchronous with underlying writes to the backend source database (Postgres, MongoDB, MySQL, or SQL Server). +It's important that your API endpoint be blocking/synchronous with underlying writes to the backend source database (Postgres, MongoDB, MySQL, SQL Server, or Convex). In other words, don't place writes into something like a queue for processing later — process them immediately. For more details, see the explainer below. -PowerSync uses a server-authoritative architecture with a checkpoint system for conflict resolution and [consistency](/architecture/consistency). The client advances to a new write checkpoint after uploads have been processed, so if the client believes that the server has written changes into your backend source database (Postgres, MongoDB, MySQL, or SQL Server), but the next checkpoint does not contain your uploaded changes, those changes will be removed from the client. This could manifest as UI glitches for your end-users, where the changes disappear from the device for a few seconds and then re-appear. +PowerSync uses a server-authoritative architecture with a checkpoint system for conflict resolution and [consistency](/architecture/consistency). The client advances to a new write checkpoint after uploads have been processed, so if the client believes that the server has written changes into your backend source database (Postgres, MongoDB, MySQL, SQL Server, or Convex), but the next checkpoint does not contain your uploaded changes, those changes will be removed from the client. This could manifest as UI glitches for your end-users, where the changes disappear from the device for a few seconds and then re-appear. ### Write Operations Recorded on the Client diff --git a/intro/powersync-overview.mdx b/intro/powersync-overview.mdx index 2c453b04..7d997eb7 100644 --- a/intro/powersync-overview.mdx +++ b/intro/powersync-overview.mdx @@ -38,6 +38,8 @@ PowerSync is designed to be backend database agnostic, and supports these source + + ## Supported Client SDKs @@ -53,4 +55,4 @@ PowerSync is also designed to be client-side stack agnostic, and currently has c ## Need Help? -Can't find what you are looking for in these docs? Try **Ask AI** on this site which is trained on all our documentation, repositories and Discord discussions. Also join us on [our community Discord server](https://discord.gg/powersync) where you can browse topics from the PowerSync community and chat with our team. \ No newline at end of file +Can't find what you are looking for in these docs? Try **Ask AI** on this site which is trained on all our documentation, repositories and Discord discussions. Also join us on [our community Discord server](https://discord.gg/powersync) where you can browse topics from the PowerSync community and chat with our team. diff --git a/intro/powersync-philosophy.mdx b/intro/powersync-philosophy.mdx index f3b8cd6e..973b0b2f 100644 --- a/intro/powersync-philosophy.mdx +++ b/intro/powersync-philosophy.mdx @@ -13,7 +13,7 @@ The app is always [fast and responsive](https://www.powersync.com/blog/local-fir PowerSync lets you avoid the complexities of using APIs to move app state [over the network](https://www.powersync.com/blog/escaping-the-network-tarpit). Its goal is to solve the hard problems of keeping data in sync, without getting in your way. -You use a standard Postgres, MongoDB, MySQL, or SQL Server \[[1](#footnotes)\] database on the server, a standard SQLite database on the client, and your [own backend](/configuration/app-backend/setup) to process mutations. PowerSync simply keeps the SQLite database in sync with your backend database. +You use a supported backend source such as Postgres, MongoDB, MySQL, SQL Server, or Convex \[[1](#footnotes)\] on the server, a standard SQLite database on the client, and your [own backend](/configuration/app-backend/setup) to process mutations. PowerSync simply keeps the SQLite database in sync with your backend source. #### State Management @@ -38,7 +38,7 @@ Our goal is also to be stack-agnostic: whether you are switching from MySQL to P #### Simplicity -You use plain Postgres, MongoDB, MySQL, or SQL Server on the server — no extensions, and no significant change in your schema required \[[2](#footnotes)\]. PowerSync [uses](/configuration/source-db/setup) Postgres logical replication, MongoDB change streams, the MySQL binlog, or SQL Server Change Data Capture (CDC) to replicate changes to the [PowerSync Service](/architecture/powersync-service), where data is transformed and partitioned according to [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)), and persisted in a way that allows efficiently streaming incremental changes to each client. +You use a supported backend source on the server, with no significant change in your schema required for most databases \[[2](#footnotes)\]. PowerSync [uses](/configuration/source-db/setup) Postgres logical replication, MongoDB change streams, the MySQL binlog, SQL Server Change Data Capture (CDC), or Convex document deltas to replicate changes to the [PowerSync Service](/architecture/powersync-service), where data is transformed and partitioned according to [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)), and persisted in a way that allows efficiently streaming incremental changes to each client. PowerSync has been used in apps with hundreds of tables. There are no complex migrations to run: You define your [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)) and [client-side schema](/intro/setup-guide#define-your-client-side-schema), and the data is automatically kept in sync. If you [change Sync Streams/Rules](/maintenance-ops/implementing-schema-changes), the relevant new set of data is applied atomically on the client. When you do need to make schema changes on the server while still supporting older clients, we [have the processes in place](/maintenance-ops/implementing-schema-changes) to do that without hassle. diff --git a/intro/setup-guide.mdx b/intro/setup-guide.mdx index c87f84d2..6bfd6185 100644 --- a/intro/setup-guide.mdx +++ b/intro/setup-guide.mdx @@ -19,7 +19,7 @@ import DevTokenSelfHostedSteps from '/snippets/dev-token-self-hosted-steps.mdx'; # 1. Configure Your Source Database -PowerSync needs to connect to your source database (Postgres, MongoDB, MySQL or SQL Server) to replicate data. Before setting up PowerSync, you need to configure your database with the appropriate permissions and replication settings. +PowerSync needs to connect to your source database (Postgres, MongoDB, MySQL, SQL Server or Convex) to replicate data. Before setting up PowerSync, you need to configure your database with the appropriate permissions and replication settings. Using the [PowerSync CLI](/tools/cli) and want an automatically integrated Postgres instance for local development? You can skip to [Step 2](#2-set-up-powersync-service-instance) and set one up with the **CLI (Self-Hosted)** tab. @@ -346,7 +346,7 @@ The next step is to connect your PowerSync Service instance to your source datab In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance, then go to **Database Connections**: 1. Click **Connect to Source Database** - 2. Select the appropriate database type tab (Postgres, MongoDB, MySQL or SQL Server) + 2. Select the appropriate database type tab (Postgres, MongoDB, MySQL, SQL Server or Convex) 3. Fill in your connection details: **Note**: Use the username (e.g., `powersync_role`) and password you created in Step 1: Configure your Source Database. @@ -354,6 +354,7 @@ The next step is to connect your PowerSync Service instance to your source datab - **Postgres**: Host, Port (5432), Database name, Username, Password, SSL Mode - **MongoDB**: Connection URI (e.g., `mongodb+srv://user:pass@cluster.mongodb.net/database`) - **MySQL**: Host, Port (3306), Database name, Username, Password + - **Convex**: Deployment URL and deploy key. See [Convex Source Database Setup](/configuration/source-db/setup#convex). - **SQL Server**: Name, Host, Port (1433), Database name, Username, Password 4. Click **Test Connection** to verify 5. Click **Save Connection** diff --git a/maintenance-ops/implementing-schema-changes.mdx b/maintenance-ops/implementing-schema-changes.mdx index b1749f7c..cfedb851 100644 --- a/maintenance-ops/implementing-schema-changes.mdx +++ b/maintenance-ops/implementing-schema-changes.mdx @@ -1,6 +1,6 @@ --- title: "Implementing Schema Changes" -description: "How PowerSync handles schema changes across Postgres, MongoDB, MySQL, and SQL Server during replication." +description: "How PowerSync handles schema changes across Postgres, MongoDB, MySQL, SQL Server, and Convex during replication." --- ## Introduction @@ -149,6 +149,26 @@ Renaming an unsynced collection to a name that is included in your Sync/Streams/ Circular renames (e.g., renaming `todos` → `todos_old` → `todos`) are not directly supported. To reprocess the database after such changes, a Sync Streams/Sync Rules update must be deployed. +## Convex Specifics + +Convex support is currently in an [Open Alpha release](/resources/feature-status). + +For Convex, most schema changes are document-shape changes rather than DDL events. PowerSync reads the JSON documents returned by Convex snapshots and document deltas, not Convex schema metadata, so added fields and type changes are replicated through normal Convex writes. + +Convex tables use `_id` as the replication identity for PowerSync. There is no source-specific primary key or replica identity definition to track. + +If a Convex migration updates existing documents, those updates should appear in `document_deltas` and replicate like other writes. You do not need to re-snapshot merely because a field was added or changed type. + +If you remove a field from Convex documents, PowerSync stops replicating new values for that field. Previously synced values can remain on clients until the affected data is reprocessed. To completely remove a field from synced client data, update and redeploy your Sync Config. + +A re-snapshot is still required for cases that change the selected data set or invalidate the source cursor. This includes initial replication, a Sync Streams deployment that selects new existing data, restarting an incomplete initial snapshot, or recovering from a lost or expired Convex cursor. + +### Dropping Tables + +Dropping Convex tables has a known limitation. Deleting a table from the Convex dashboard does not emit per-document delete rows in `document_deltas`, so PowerSync does not automatically remove previously synced rows for that table. + +To decommission a table while preserving replication correctness, clear the table before deleting it. In the Convex dashboard, use **Clear Table** first, then delete the table after those document removals have replicated. Deleting documents through Convex mutations is also valid when that path emits document delete deltas. Otherwise, treat dashboard table deletion or schema-only table removal as a Sync Config deployment change and clear or re-replicate affected PowerSync state. + ## MySQL Specifics MySQL support is currently in a [Beta release](/resources/feature-status). @@ -319,4 +339,4 @@ if CDC is enabled on the table again, the data will be removed and the table re- * [JSON, Arrays and Custom Types](/client-sdks/advanced/custom-types-arrays-and-json) -* [Deploying Schema Changes](/maintenance-ops/deploying-schema-changes) \ No newline at end of file +* [Deploying Schema Changes](/maintenance-ops/deploying-schema-changes) diff --git a/maintenance-ops/replication-lag.mdx b/maintenance-ops/replication-lag.mdx index 94f34a80..03125470 100644 --- a/maintenance-ops/replication-lag.mdx +++ b/maintenance-ops/replication-lag.mdx @@ -3,7 +3,7 @@ title: "Replication Lag" description: "Understand, monitor, and reduce replication lag between your source database and the PowerSync Service." --- -Replication lag is the delay between a change being committed in your source database (Postgres, MongoDB, MySQL, SQL Server) and that change being available in the PowerSync Service for clients to sync. A small amount of lag is normal. Sustained or growing lag usually points to a specific cause that you can investigate and act on. +Replication lag is the delay between a change being committed in your source database (Postgres, MongoDB, MySQL, SQL Server or Convex) and that change being available in the PowerSync Service for clients to sync. A small amount of lag is normal. Sustained or growing lag usually points to a specific cause that you can investigate and act on. This page covers what replication lag is, how to monitor it, what commonly causes it, and how to reduce it. @@ -16,6 +16,7 @@ A change committed in the source database goes through roughly three stages befo - **MongoDB**: change streams backed by the oplog. - **MySQL**: the binary log (binlog), read using GTIDs. - **SQL Server**: Change Data Capture (CDC) change tables, populated by a capture job that scans the transaction log. + - **Convex**: document deltas from the Convex Streaming Export API, read on a polling interval. 2. The PowerSync Service reads the change from that stream and processes it into its internal bucket storage. 3. Connected clients receive the change on their next checkpoint. @@ -25,6 +26,10 @@ Replication lag refers specifically to stage 2: the time or volume of changes th SQL Server has an additional source of latency inside stage 1: the CDC capture job itself runs on an interval (default 5 seconds on SQL Server, fixed at 20 seconds on Azure SQL), so changes do not appear in the CDC change tables instantly. See [SQL Server](#sql-server) below. + + Convex replication also has polling latency. PowerSync polls the Convex `document_deltas` endpoint at a configurable interval, which defaults to once every 1000ms. See [Convex Source Database Setup](/configuration/source-db/setup#latency). + + ## How to Monitor Replication Lag ### PowerSync Dashboard diff --git a/resources/feature-status.mdx b/resources/feature-status.mdx index 8427356e..da79eac1 100644 --- a/resources/feature-status.mdx +++ b/resources/feature-status.mdx @@ -46,6 +46,7 @@ Below is a summary of the current main PowerSync features and their release stat | **Category / Item** | **Status** | | -------------------------------------------------- | -------------- | | **Database Connectors** | | +| Convex | Open Alpha | | SQL Server | Beta | | MySQL | Beta | | MongoDB | GA | diff --git a/resources/usage-and-billing/usage-and-billing-faq.mdx b/resources/usage-and-billing/usage-and-billing-faq.mdx index 8dfd0bc3..81c0ee93 100644 --- a/resources/usage-and-billing/usage-and-billing-faq.mdx +++ b/resources/usage-and-billing/usage-and-billing-faq.mdx @@ -74,7 +74,7 @@ description: "Troubleshoot usage and billing issues and find answers to common q Data processing was calculated as the total uncompressed size of data replicated from your source database(s) to PowerSync Service instances, plus data synced from PowerSync Service instances to user devices. These values are still available in your [Usage metrics](/maintenance-ops/monitoring-and-alerting#usage-metrics) as "Data replicated per day/hour" and "Data synced per day/hour". - Data replicated refers to activity from your backend source database (Postgres, MongoDB, MySQL, or SQL Server database) to the PowerSync Service — this is not billed. + Data replicated refers to activity from your backend source database (Postgres, MongoDB, MySQL, SQL Server, or Convex) to the PowerSync Service — this is not billed. Data synced refers to data streamed from the PowerSync Service to client devices — this is used for billing. @@ -254,4 +254,4 @@ Data hosted can temporarily spike during Sync Rule deployments and defragmentati # Accident Forgiveness -Accidentally ran up a high bill? No problem — we've got your back. Reach out to us at [support@powersync.com](mailto:support@powersync.com) and we'll work with you to resolve the issue and prevent it from happening again. \ No newline at end of file +Accidentally ran up a high bill? No problem — we've got your back. Reach out to us at [support@powersync.com](mailto:support@powersync.com) and we'll work with you to resolve the issue and prevent it from happening again. diff --git a/sync/streams/overview.mdx b/sync/streams/overview.mdx index 56dff2ab..d91e7955 100644 --- a/sync/streams/overview.mdx +++ b/sync/streams/overview.mdx @@ -268,7 +268,7 @@ const sub = await db.syncStream('todos', { list_id: 'abc' }) - **SQL Syntax**: Stream queries use a SQL-like syntax with `SELECT` statements. You can use subqueries, `INNER JOIN`, and [CTEs](/sync/streams/ctes) for filtering. `GROUP BY`, `ORDER BY`, and `LIMIT` are not supported. See [Writing Queries](/sync/streams/queries) for details on joins, multiple queries per stream, and other features. -- **Type Conversion**: Data types from your source database (Postgres, MongoDB, MySQL, SQL Server) are converted when synced to the client's SQLite database. SQLite has a limited type system, so most types become `text` and you may need to parse or cast values in your app code. See [Type Mapping](/sync/types) for details on how each type is handled. +- **Type Conversion**: Data types from your source database (Postgres, MongoDB, MySQL, SQL Server or Convex) are converted when synced to the client's SQLite database. SQLite has a limited type system, so most types become `text` and you may need to parse or cast values in your app code. See [Type Mapping](/sync/types) for details on how each type is handled. - **Primary Key**: PowerSync requires every synced table to have a primary key column named `id` of type `text`. If your backend uses a different column name or type, you'll need to map it. For MongoDB, collections use `_id` as the ID field; you must alias it in your stream queries (e.g. `SELECT _id as id, * FROM your_collection`). diff --git a/sync/types.mdx b/sync/types.mdx index 0f911a6f..fd80bb0c 100644 --- a/sync/types.mdx +++ b/sync/types.mdx @@ -1,7 +1,7 @@ --- title: "Types" sidebarTitle: "Type Mapping" -description: "How Postgres, MongoDB, MySQL and SQL Server types map to PowerSync's SQLite-based sync column definitions." +description: "How Postgres, MongoDB, MySQL, SQL Server and Convex types map to PowerSync's SQLite-based sync column definitions." --- import BinaryType from '/snippets/binary-type.mdx'; @@ -42,10 +42,34 @@ Postgres types are mapped to SQLite types as follows: | Custom types | `text` | Depending on [compatibility options](/sync/advanced/compatibility#custom-postgres-types), JSON object or raw wire representation (legacy). | | (Multi-)ranges | `text` | Depending on [compatibility options](/sync/advanced/compatibility#custom-postgres-types), JSON object (array for multi-ranges) or raw wire representation (legacy). | - +## Convex Type Mapping + +Convex support is currently in an [Open Alpha release](/resources/feature-status). + +Convex values are mapped to SQLite types as follows: + +| Convex Type | TS/JS Type | PowerSync / SQLite Column Type | Notes | +| ----------- | ---------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Id` | `string` | `text` | Convex document IDs are exposed as `_id` and can be synced as `text`. For synced client tables, use client-side ID mapping with a stable UUID column as `id` instead of relying on Convex-generated `_id` values. | +| `Null` | `null` | `null` | | +| `Int64` | `base-10 string` | `text` | Cast to `INTEGER` in Sync Streams when you want to sync the value as a SQLite integer. | +| `Float64` | `number` | `real` | | +| `Boolean` | `boolean` | `integer` | `1` for true, `0` for false. There is no dedicated boolean data type in SQLite. | +| `String` | `string` | `text` | | +| `Bytes` | `base64 string` | `text` | Decode from base64 in your app if you need binary data on the client. | +| `Array` | `Array` | `text` | Converted to a JSON string. | +| `Object` | `Object` | `text` | Converted to a JSON string. | +| `Record` | `Record` | `text` | Converted to a JSON string. | + +- Convex documents are converted to a flat list of columns, one column per top-level field. +- Convex system fields used internally by replication are not synced as normal columns. The document `_id` is included, while `_creationTime` is excluded. +- Nested objects and arrays are converted to JSON, and [JSON functions and operators](/sync/supported-sql#operators) can be used to query them in Sync Streams or on the client-side SQLite database. +- Cast Convex `Int64` fields to `INTEGER` in Sync Streams when you want SQLite integer values on the client, for example `CAST(an_int64_column AS INTEGER) AS an_int64_column`. + + ## MongoDB Type Mapping MongoDB types are mapped to SQLite types as follows: