feat: Initial PoC#1
Conversation
| await transaction.complete(result.checkpoint); | ||
| break; | ||
| case 'fatal_error': | ||
| console.error('Fatal error:', result.failedOperation?.error_code, result.message); |
There was a problem hiding this comment.
Fatal errors should also call transaction.complete().
There was a problem hiding this comment.
So that the client doesn't get stuck. That would mean we lose the data, unless we have a dead-letter queue on the backend?
There was a problem hiding this comment.
That's correct. The assumption is that if it's a "fatal" error as opposed to "retryable", there is nothing we can resolve by retrying, so the only option is discarding the data.
FWIW, a dead-letter queue on the client could be another option.
| const cp = await this.options.transport.putCheckpoint(this.options.userId, this.options.clientId); | ||
| result.checkpoint = cp.checkpoint; |
There was a problem hiding this comment.
Write checkpoints make up a big topic on its own - see https://github.com/orgs/powersync-ja/discussions/317
We probably want to finalize that before implementing a new pattern around custom write checkpoints here.
One specific thing to note is that creating a new write checkpoint after every transaction can add a lot of overhead when there are many small transactions - ideally it should only be done once at the end of the queue. But the current uploadData API is not really designed well for that appraoch.
| transport: { | ||
| async postTransaction(body) { | ||
| const { data } = await client.POST('/api/data', { body }); | ||
| if (!data) throw new Error('No response from /api/data'); |
There was a problem hiding this comment.
This should check the specific error response rather, like in putCheckpoint.
|
|
||
| this._clientId = await database.getClientId(); | ||
| const writeClient = await this.getWriteClient(database); | ||
| const result = await writeClient.processTransaction(transaction); |
There was a problem hiding this comment.
There are multiple error paths here:
- Error is returned, which can be
fatal_errororretryable_error. - Error is thrown, which should always be retryable.
Thrown errors can include network errors, gateway errors, db errors - anything not producing a valid response.
It may be worth having returned retryable_error use the same path as thrown errors, to reduce the cases the connector has to handle. For example, if the client throws those errors, we can remove the case 'retryable_error' below.
It's not important though, as long as the different cases are properly documented. Assuming every app is not going to provide a custom connector like this anymore, that logic can also be in the connector itself.
2 participants
This PoC explores ideas for defining a template write path that should simplify implementing writes for PowerSync users. It is based on the node backend and client demos.
The core shift in this PoC is moving the "translation problem" from the client to the server edge. Right now, every developer has to write their own bespoke
uploadDatahandler that fetches CRUD ops, maps types, talks to their backend, and classifies errors. This PoC says: what if the client just sends the raw CRUD data over the wire, and a standardised server-side stack handles everything from there? The backend includes generated endpoint handler code (via OpenAPI), a mechanism for mapping sqlite crud data to the format of a source database, and a persister for each support source database.This is essentially a foundation layer, it defines the protocol with the happy path in mind, and both the mutator example (PoC 2) and the Supabase edge function approach (PoC 3) are described as things that can ride on top of it. We also still need to improve the implementation detail at this level in terms of interface definitions and hooks for supporting custom logic on every layer, as well as advanced tools like dead letter queues.
Adding OpenAPI codegen means that some of the implementation can be spread to new languages/platforms without reimplementing the wire code each time
AI disclosure
This PR was created with the help of Claude Code. Help constitutes assistance in research, planning, and rough outline of implementation. Beyond having a hand in the implementation, I have also manually tested this work.