diff --git a/configuration/source-db/connection.mdx b/configuration/source-db/connection.mdx index 32c88ccc..49928474 100644 --- a/configuration/source-db/connection.mdx +++ b/configuration/source-db/connection.mdx @@ -253,7 +253,7 @@ Also see: 2. "**Name**" can be any name for the connection. 3. "**Host**" is the endpoint for your SQL Server instance. 4. "**Port**" is typically 1433 for SQL Server (default port). - 5. "**Database name**" is the database where CDC is enabled. + 5. "**Database name**" is the database you want to replicate. CDC should be enabled on this database. 6. "**Username**" and "**Password**" maps to the database user created in [Source Database Setup](/configuration/source-db/setup#sql-server-alpha) (e.g., `powersync_user`). 4. Click **Test Connection** and fix any errors. 5. Click **Save Connection**. diff --git a/configuration/source-db/setup.mdx b/configuration/source-db/setup.mdx index 054f3e78..86765f4e 100644 --- a/configuration/source-db/setup.mdx +++ b/configuration/source-db/setup.mdx @@ -433,16 +433,15 @@ For more information about CDC, see: ### Supported Editions/Versions -| Database | Edition | Version | Min Service Tier | -| ---------------- | ------------------------------------------- | ------- | ------------------------------------------------------------------------------------- | -| SQL Server 2022+ | Standard, Enterprise, Developer, Evaluation | 16.0+ | N/A | +| Database | Edition | Version | Min Service Tier | +| ---------------- | ------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| SQL Server 2019+ | Standard, Enterprise, Developer, Evaluation | 15.0+ | N/A | | Azure SQL\* | Database, Managed instance | N/A | Any service tier on vCore purchasing model. S3 tier and up on DTU purchasing model. See: [Azure SQL Database compute requirements](https://learn.microsoft.com/en-us/azure/azure-sql/database/change-data-capture-overview?view=azuresql#azure-sql-database-compute-requirements) | \* Azure SQL Database is always running on the latest version of the SQL Server DB Engine ### Limitations / Known Issues -- Schema change handling is not supported yet. - Spatial data types are returned as JSON objects as supplied by the Tedious `node-mssql` client. See the notes [here](https://github.com/tediousjs/node-mssql?tab=readme-ov-file#geography-and-geometry). - There is an inherent latency in replicating data from SQL Server to PowerSync. See [Latency](#latency) for more details. @@ -615,13 +614,17 @@ Due to the fundamental differences in how CDC works compared to logical replicat 2. **Polling Interval**: The frequency at which PowerSync polls the CDC change tables for changes. The default value is once every 1000ms. This can be changed by setting the `pollingIntervalMs` parameter in the PowerSync configuration. -Configuration parameters for SQL Server like `pollingIntervalMs` and `pollingBatchSize` (see below) can currently only be [set when self-hosting](/intro/setup-guide#self-hosted-2) PowerSync. We are working on exposing these settings for SQL Server source database connections in the PowerSync Dashboard for PowerSync Cloud instances. - ### Memory Management During each polling cycle, PowerSync will read a limited number of transactions from the CDC change tables. The default value of 10 transactions can be changed by setting the `pollingBatchSize` parameter in the PowerSync configuration. Increasing this will increase throughput at the cost of increased memory usage. If the volume of transactions being replicated is high, and memory is available, it is recommended to increase this value. + + Connection configuration parameters for the PowerSync SQL Server Adapter like `pollingIntervalMs` and `pollingBatchSize` can currently only be set when [self-hosting](/intro/setup-guide#self-hosted-2) PowerSync. + See [SQL Server Additional Configuration](/configuration/source-db/sql-server-additional-configuration#additional-configuration-options) + for more details. + We are working on exposing these settings for SQL Server source database connections in the PowerSync Dashboard for PowerSync Cloud instances. + ## Next Step diff --git a/configuration/source-db/sql-server-additional-configuration.mdx b/configuration/source-db/sql-server-additional-configuration.mdx new file mode 100644 index 00000000..9b703c06 --- /dev/null +++ b/configuration/source-db/sql-server-additional-configuration.mdx @@ -0,0 +1,97 @@ +--- +title: "SQL Server Additional Configuration" +--- + +## Introduction + +Some additional configuration for the SQL Server Adapter is available to users who are self-hosting the PowerSync Service. + +## Connect to Azure SQL Database using a Service Principal + +As an alternative to using standard SQL Login via username and password, PowerSync can connect to your Azure SQL Database using a Service Principal. + +Update your `config.yaml` file to use the Service Principal for authentication: + +```yaml config.yaml +# Settings for source database replication +replication: + connections: + - type: mssql + database: + schema: # Optional, defaults to dbo + hostname: + port: 1433 # Default port for SQL Server + + authentication: + type: azure-active-directory-service-principal-secret + options: + clientId: + tenantId: + clientSecret: +``` + + + +### Create a New App Registration + +1. Go to the [Azure portal](https://portal.azure.com/) and navigate to **App registrations** >. +2. Click **New registration**. +3. Fill in the required fields: + - **Name**: Give your app user a name, e.g. "powersync_user". + - **Redirect URI**: Leave this blank. + - **Supported account types**: Select **Accounts in this organizational directory only**. +4. Click **Register**. +5. Copy the **Application (Client) ID** and **Directory (Tenant) ID**. + +### Create a Client Secret + +1. Navigate to the app registration you created in the previous step. +2. In the left navigation menu, under **Manage**, click **Certificates & secrets**. +3. Click **New client secret**. +4. Fill in the required fields: + - **Description**: Enter a description for the secret + - **Expires**: Select an expiration date +5. Click **Add**. +6. **Important**: Copy the **Value** - this is the client secret and will not be shown again. + +### Configure the Database User + +1. Connect to the SQL Server database using an account with administrative privileges. +2. Create a new user linked to the app registration: + +```sql +CREATE USER [powersync_user] FROM EXTERNAL PROVIDER; +``` + +3. Complete the rest of the user setup by following the instructions in [Source Database Setup](/configuration/source-db/setup#2-create-the-powersync-database-user). + + + +## Additional Configuration Options + +The below configuration options can be set in the `additionalConfig` section under the `connections` section of the `config.yaml` file. + + + All of these configuration options are optional and can be omitted if the + default values are acceptable. + + +**pollingIntervalMs**: +The polling interval is the frequency at which the PowerSync SQL Server Adapter polls the SQL Server Change tables for changes. +Lowering this value helps reduce latency in the replication process, but also increases the load on the SQL Server database and the PowerSync Service. + +**pollingBatchSize** +The polling batch size is the number of transactions that the PowerSync SQL Server Adapter will process in each polling cycle. +Increasing this value can increase throughput at the cost of increased memory usage. If the volume of transactions being replicated is high, and memory is available, it is recommended to increase this value. + +**trustServerCertificate**: +Whether to trust the server certificate. Set to true for local development and self-signed certificates. Set to false when connecting to Azure SQL Database. + +```yaml config.yaml +connections: + - type: mssql + additionalConfig: + pollingIntervalMs: 1000 # Default is 1000ms + pollingBatchSize: 10 # Default is 10 + trustServerCertificate: false # Default is false +``` diff --git a/docs.json b/docs.json index ce83a758..f1ee514b 100644 --- a/docs.json +++ b/docs.json @@ -120,7 +120,8 @@ "configuration/source-db/connection", "configuration/source-db/security-and-ip-filtering", "configuration/source-db/private-endpoints", - "configuration/source-db/postgres-maintenance" + "configuration/source-db/postgres-maintenance", + "configuration/source-db/sql-server-additional-configuration" ] }, { @@ -474,7 +475,7 @@ "tools/diagnostics-client", "tools/local-development", "tools/ai-tools" - ] + ] }, { "group": "Resources", diff --git a/maintenance-ops/implementing-schema-changes.mdx b/maintenance-ops/implementing-schema-changes.mdx index d2fb2164..9a8e4459 100644 --- a/maintenance-ops/implementing-schema-changes.mdx +++ b/maintenance-ops/implementing-schema-changes.mdx @@ -148,7 +148,6 @@ Renaming an unsynced collection to a name that is included in your [Sync Streams Circular renames (e.g., renaming `todos` → `todos_old` → `todos`) are not directly supported. To reprocess the database after such changes, a [Sync Streams](/sync/streams/overview) update (or [Sync Rules](/sync/rules/overview)) must be deployed. - ## MySQL (Beta) Specifics PowerSync keeps the [buckets](/architecture/powersync-service#bucket-system) up to date with any incremental data changes as recorded in the MySQL [binary log](https://dev.mysql.com/doc/refman/8.4/en/binary-log.html). @@ -220,6 +219,92 @@ Removing a column will not have the values automatically removed for existing ro Changing a column type, and/or changing the default value of a column using an `ALTER TABLE` statement, will not be automatically replicated to PowerSync. In some cases, the change will have no effect on PowerSync (for example, changing between `VARCHAR` and `TEXT` types). When the values are expected to change, make an update to every existing row to propagate the changes. +## SQL Server (Alpha) Specifics + +SQL Server CDC replication is designed to protect downstream consumers from schema changes. This means some schema changes, like changing the data type of a primary key column are blocked on the database level if CDC is enabled on a table. +Other schema changes are allowed but are not automatically propagated to the capture instance for a table. This means that the only sure-fire way to update a capture instance with the latest schema changes is to drop and recreate it, or to create a new capture instance. +PowerSync will automatically detect and use the new capture instance for the source table. Note that SQL Server CDC allows a maximum of 2 [capture instances](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server?view=sql-server-ver17#capture-instance) per table. + +#### Dropping and Recreating a Capture Instance + +```sql +-- Disable CDC for the source table +EXEC sys.sp_cdc_disable_table + @source_schema = N'dbo', + @source_name = N'', + @capture_instance = 'all'; -- This drops all capture instances for the source table. If you only want to drop a specific capture instance, use the capture instance name instead of 'all'. + +-- Re-enable CDC for the source table +EXEC sys.sp_cdc_enable_table + @source_schema = N'dbo', + @source_name = N'', + @role_name = N'cdc_reader', + @supports_net_changes = 0; +``` + +#### Creating a New Capture Instance + +```sql +-- Create a new capture instance for the source table by specifying a new capture instance name +EXEC sys.sp_cdc_enable_table + @source_schema = N'dbo', + @source_name = N'', + @role_name = N'cdc_reader', + @supports_net_changes = 0, + @capture_instance = N''; -- If a capture instance for the table already exists, you have to specify a different name for the new capture instance. +``` + +### Supported SQL Server schema changes: + +#### CREATE table + +Table creation is automatically detected when a new capture instance for a source table that matches the Sync Rules is created. The table is snapshotted before replication can resume. + +#### DROP table + +PowerSync can detect a table drop by checking for the table existence when the capture instance for a table is dropped. This only works if PowerSync is running at the time of the table drop. + + + PowerSync cannot detect that a table was dropped if it was dropped while the + PowerSync service was stopped. The Sync Rules will need to be redeployed to + ensure that corresponding table's buckets are removed. + + +#### RENAME table + +Renaming a table is automatically detected and results in the removal of the bucket data for the old table, followed by a snapshot of the newly renamed table. +Once the snapshot is completed, replication will resume. + +#### Column changes + +Some column changes are blocked on the database level if CDC is enabled on a table. These include: + +- column renames +- changing the primary key +- changing the data type of the primary key column + +To perform these changes, CDC needs to be disabled and then re-enabled for the table. + +Column changes that are not blocked: + +- adding a new column: Until the capture instance has been recreated, the new column will not be replicated. +- dropping a column: Until the capture instance has been recreated, replicated rows will contain a NULL value for the dropped column. +- changing the data type of a column to another compatible type: PowerSync will replicate updated rows with the new data type, but historic rows will not be updated. To propagate the changes, make an update to every existing row to propagate the changes. + +PowerSync will automatically detect and warn about the above changes, but to update the capture instance with the latest schema changes, CDC needs to be disabled and then re-enabled for the table. +See the [Dropping and Recreating a Capture Instance](#dropping-and-recreating-a-capture-instance) section for more details. + +#### New Capture Instance + +New capture instances are automatically detected for tables in the Sync Rules. If a capture instance is newer than the one currently in use, PowerSync will automatically use the new capture instance. +Switching to a new capture instance requires re-snapshotting the source table before replication can resume. + +#### Capture instance missing or deleted + +If a table in the sync rules is not enabled for CDC, the table cannot be replicated and PowerSync will log a warning. +If a table in the sync rules is being replicated and the capture instance is dropped, PowerSync logs a warning and stops replication for that table. Existing data will not be removed, but +if CDC is enabled on the table again, the data will be removed and the table re-snapshotted. + ## See Also * [JSON, Arrays and Custom Types](/client-sdks/advanced/custom-types-arrays-and-json)