Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion configuration/source-db/connection.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ Also see:
2. "**Name**" can be any name for the connection.
3. "**Host**" is the endpoint for your SQL Server instance.
4. "**Port**" is typically 1433 for SQL Server (default port).
5. "**Database name**" is the database where CDC is enabled.
5. "**Database name**" is the database you want to replicate. CDC should be enabled on this database.
6. "**Username**" and "**Password**" maps to the database user created in [Source Database Setup](/configuration/source-db/setup#sql-server-alpha) (e.g., `powersync_user`).
4. Click **Test Connection** and fix any errors.
5. Click **Save Connection**.
Expand Down
15 changes: 9 additions & 6 deletions configuration/source-db/setup.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -433,16 +433,15 @@

### Supported Editions/Versions

| Database | Edition | Version | Min Service Tier |
| ---------------- | ------------------------------------------- | ------- | ------------------------------------------------------------------------------------- |
| SQL Server 2022+ | Standard, Enterprise, Developer, Evaluation | 16.0+ | N/A |
| Database | Edition | Version | Min Service Tier |
| ---------------- | ------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| SQL Server 2019+ | Standard, Enterprise, Developer, Evaluation | 15.0+ | N/A |
| Azure SQL\* | Database, Managed instance | N/A | Any service tier on vCore purchasing model. S3 tier and up on DTU purchasing model. See: [Azure SQL Database compute requirements](https://learn.microsoft.com/en-us/azure/azure-sql/database/change-data-capture-overview?view=azuresql#azure-sql-database-compute-requirements) |

Check warning on line 439 in configuration/source-db/setup.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

configuration/source-db/setup.mdx#L439

Did you really mean 'vCore'?

\* Azure SQL Database is always running on the latest version of the SQL Server DB Engine

### Limitations / Known Issues

- Schema change handling is not supported yet.
- Spatial data types are returned as JSON objects as supplied by the Tedious `node-mssql` client. See the notes [here](https://github.com/tediousjs/node-mssql?tab=readme-ov-file#geography-and-geometry).
- There is an inherent latency in replicating data from SQL Server to PowerSync. See [Latency](#latency) for more details.

Expand All @@ -469,7 +468,7 @@

- Read/Write permissions on the `_powersync_checkpoints` table
- Read permissions on the replicated tables
- `cdc_reader` role (grants access to CDC changetables and functions)

Check warning on line 471 in configuration/source-db/setup.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

configuration/source-db/setup.mdx#L471

Did you really mean 'changetables'?
- `SELECT` permission on the CDC schema (grants access to CDC metadata tables)
- `VIEW DATABASE PERFORMANCE STATE` (SQL Server and Azure SQL)
- `VIEW SERVER PERFORMANCE STATE` (SQL Server only)
Expand Down Expand Up @@ -615,13 +614,17 @@

2. **Polling Interval**: The frequency at which PowerSync polls the CDC change tables for changes. The default value is once every 1000ms. This can be changed by setting the `pollingIntervalMs` parameter in the PowerSync configuration.

<Warning>Configuration parameters for SQL Server like `pollingIntervalMs` and `pollingBatchSize` (see below) can currently only be [set when self-hosting](/intro/setup-guide#self-hosted-2) PowerSync. We are working on exposing these settings for SQL Server source database connections in the PowerSync Dashboard for PowerSync Cloud instances.</Warning>

### Memory Management

During each polling cycle, PowerSync will read a limited number of transactions from the CDC change tables. The default value of 10 transactions can be changed by setting the `pollingBatchSize` parameter in the PowerSync configuration.
Increasing this will increase throughput at the cost of increased memory usage. If the volume of transactions being replicated is high, and memory is available, it is recommended to increase this value.

<Note>
Connection configuration parameters for the PowerSync SQL Server Adapter like `pollingIntervalMs` and `pollingBatchSize` can currently only be set when [self-hosting](/intro/setup-guide#self-hosted-2) PowerSync.
See [SQL Server Additional Configuration](/configuration/source-db/sql-server-additional-configuration#additional-configuration-options)
for more details.
We are working on exposing these settings for SQL Server source database connections in the PowerSync Dashboard for PowerSync Cloud instances.
</Note>

## Next Step

Expand Down
97 changes: 97 additions & 0 deletions configuration/source-db/sql-server-additional-configuration.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---
title: "SQL Server Additional Configuration"
---

## Introduction

Some additional configuration for the SQL Server Adapter is available to users who are self-hosting the PowerSync Service.

## Connect to Azure SQL Database using a Service Principal

As an alternative to using standard SQL Login via username and password, PowerSync can connect to your Azure SQL Database using a Service Principal.

Update your `config.yaml` file to use the Service Principal for authentication:

```yaml config.yaml
# Settings for source database replication
replication:
connections:
- type: mssql
database: <YOUR_DATABASE_NAME>
schema: <YOUR_SCHEMA_NAME> # Optional, defaults to dbo
hostname: <YOUR_HOSTNAME>
port: 1433 # Default port for SQL Server

authentication:
type: azure-active-directory-service-principal-secret
options:
clientId: <YOUR_CLIENT_ID>
tenantId: <YOUR_TENANT_ID>
clientSecret: <YOUR_CLIENT_SECRET>
```

<Accordion title="Set up a Service Principal for Azure SQL Database">

### Create a New App Registration

1. Go to the [Azure portal](https://portal.azure.com/) and navigate to **App registrations** >.
2. Click **New registration**.
3. Fill in the required fields:
- **Name**: Give your app user a name, e.g. "powersync_user".

Check warning on line 40 in configuration/source-db/sql-server-additional-configuration.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

configuration/source-db/sql-server-additional-configuration.mdx#L40

Did you really mean 'powersync_user'?
- **Redirect URI**: Leave this blank.
- **Supported account types**: Select **Accounts in this organizational directory only**.
4. Click **Register**.
5. Copy the **Application (Client) ID** and **Directory (Tenant) ID**.

### Create a Client Secret

1. Navigate to the app registration you created in the previous step.
2. In the left navigation menu, under **Manage**, click **Certificates & secrets**.
3. Click **New client secret**.
4. Fill in the required fields:
- **Description**: Enter a description for the secret
- **Expires**: Select an expiration date
5. Click **Add**.
6. **Important**: Copy the **Value** - this is the client secret and will not be shown again.

### Configure the Database User

1. Connect to the SQL Server database using an account with administrative privileges.
2. Create a new user linked to the app registration:

```sql
CREATE USER [powersync_user] FROM EXTERNAL PROVIDER;
```

3. Complete the rest of the user setup by following the instructions in [Source Database Setup](/configuration/source-db/setup#2-create-the-powersync-database-user).

</Accordion>

## Additional Configuration Options

The below configuration options can be set in the `additionalConfig` section under the `connections` section of the `config.yaml` file.

<Note>
All of these configuration options are optional and can be omitted if the
default values are acceptable.
</Note>

**pollingIntervalMs**:
The polling interval is the frequency at which the PowerSync SQL Server Adapter polls the SQL Server Change tables for changes.
Lowering this value helps reduce latency in the replication process, but also increases the load on the SQL Server database and the PowerSync Service.

**pollingBatchSize**
The polling batch size is the number of transactions that the PowerSync SQL Server Adapter will process in each polling cycle.
Increasing this value can increase throughput at the cost of increased memory usage. If the volume of transactions being replicated is high, and memory is available, it is recommended to increase this value.

**trustServerCertificate**:
Whether to trust the server certificate. Set to true for local development and self-signed certificates. Set to false when connecting to Azure SQL Database.

```yaml config.yaml
connections:
- type: mssql
additionalConfig:
pollingIntervalMs: 1000 # Default is 1000ms
pollingBatchSize: 10 # Default is 10
trustServerCertificate: false # Default is false
```
5 changes: 3 additions & 2 deletions docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,8 @@
"configuration/source-db/connection",
"configuration/source-db/security-and-ip-filtering",
"configuration/source-db/private-endpoints",
"configuration/source-db/postgres-maintenance"
"configuration/source-db/postgres-maintenance",
"configuration/source-db/sql-server-additional-configuration"
]
},
{
Expand Down Expand Up @@ -474,7 +475,7 @@
"tools/diagnostics-client",
"tools/local-development",
"tools/ai-tools"
]
]
},
{
"group": "Resources",
Expand Down
87 changes: 86 additions & 1 deletion maintenance-ops/implementing-schema-changes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,6 @@

Circular renames (e.g., renaming `todos` → `todos_old` → `todos`) are not directly supported. To reprocess the database after such changes, a [Sync Streams](/sync/streams/overview) update (or [Sync Rules](/sync/rules/overview)) must be deployed.


## <Icon icon="dolphin" iconType="solid" size={32} /> MySQL (Beta) Specifics

PowerSync keeps the [buckets](/architecture/powersync-service#bucket-system) up to date with any incremental data changes as recorded in the MySQL [binary log](https://dev.mysql.com/doc/refman/8.4/en/binary-log.html).
Expand Down Expand Up @@ -220,6 +219,92 @@
Changing a column type, and/or changing the default value of a column using an `ALTER TABLE` statement, will not be automatically replicated to PowerSync.
In some cases, the change will have no effect on PowerSync (for example, changing between `VARCHAR` and `TEXT` types). When the values are expected to change, make an update to every existing row to propagate the changes.

## <Icon icon="database" iconType="solid" size={32} /> SQL Server (Alpha) Specifics

SQL Server CDC replication is designed to protect downstream consumers from schema changes. This means some schema changes, like changing the data type of a primary key column are blocked on the database level if CDC is enabled on a table.
Other schema changes are allowed but are not automatically propagated to the capture instance for a table. This means that the only sure-fire way to update a capture instance with the latest schema changes is to drop and recreate it, or to create a new capture instance.
PowerSync will automatically detect and use the new capture instance for the source table. Note that SQL Server CDC allows a maximum of 2 [capture instances](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/about-change-data-capture-sql-server?view=sql-server-ver17#capture-instance) per table.

#### Dropping and Recreating a Capture Instance

```sql
-- Disable CDC for the source table
EXEC sys.sp_cdc_disable_table
@source_schema = N'dbo',
@source_name = N'<YOUR_TABLE_NAME>',
@capture_instance = 'all'; -- This drops all capture instances for the source table. If you only want to drop a specific capture instance, use the capture instance name instead of 'all'.

-- Re-enable CDC for the source table
EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name = N'<YOUR_TABLE_NAME>',
@role_name = N'cdc_reader',
@supports_net_changes = 0;
```

#### Creating a New Capture Instance

```sql
-- Create a new capture instance for the source table by specifying a new capture instance name
EXEC sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name = N'<YOUR_TABLE_NAME>',
@role_name = N'cdc_reader',
@supports_net_changes = 0,
@capture_instance = N'<NEW_CAPTURE_INSTANCE_NAME>'; -- If a capture instance for the table already exists, you have to specify a different name for the new capture instance.
```

### Supported SQL Server schema changes:

#### CREATE table

Table creation is automatically detected when a new capture instance for a source table that matches the Sync Rules is created. The table is snapshotted before replication can resume.

Check warning on line 261 in maintenance-ops/implementing-schema-changes.mdx

View check run for this annotation

Mintlify / Mintlify Validation (powersync) - vale-spellcheck

maintenance-ops/implementing-schema-changes.mdx#L261

Did you really mean 'snapshotted'?

#### DROP table

PowerSync can detect a table drop by checking for the table existence when the capture instance for a table is dropped. This only works if PowerSync is running at the time of the table drop.

<Warning>
PowerSync cannot detect that a table was dropped if it was dropped while the
PowerSync service was stopped. The Sync Rules will need to be redeployed to
ensure that corresponding table's buckets are removed.
</Warning>

#### RENAME table

Renaming a table is automatically detected and results in the removal of the bucket data for the old table, followed by a snapshot of the newly renamed table.
Once the snapshot is completed, replication will resume.

#### Column changes

Some column changes are blocked on the database level if CDC is enabled on a table. These include:

- column renames
- changing the primary key
- changing the data type of the primary key column

To perform these changes, CDC needs to be disabled and then re-enabled for the table.

Column changes that are not blocked:

- adding a new column: Until the capture instance has been recreated, the new column will not be replicated.
- dropping a column: Until the capture instance has been recreated, replicated rows will contain a NULL value for the dropped column.
- changing the data type of a column to another compatible type: PowerSync will replicate updated rows with the new data type, but historic rows will not be updated. To propagate the changes, make an update to every existing row to propagate the changes.

PowerSync will automatically detect and warn about the above changes, but to update the capture instance with the latest schema changes, CDC needs to be disabled and then re-enabled for the table.
See the [Dropping and Recreating a Capture Instance](#dropping-and-recreating-a-capture-instance) section for more details.

#### New Capture Instance

New capture instances are automatically detected for tables in the Sync Rules. If a capture instance is newer than the one currently in use, PowerSync will automatically use the new capture instance.
Switching to a new capture instance requires re-snapshotting the source table before replication can resume.

#### Capture instance missing or deleted

If a table in the sync rules is not enabled for CDC, the table cannot be replicated and PowerSync will log a warning.
If a table in the sync rules is being replicated and the capture instance is dropped, PowerSync logs a warning and stops replication for that table. Existing data will not be removed, but
if CDC is enabled on the table again, the data will be removed and the table re-snapshotted.

## See Also

* [JSON, Arrays and Custom Types](/client-sdks/advanced/custom-types-arrays-and-json)
Expand Down