Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .config/dotnet-tools.json
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,13 @@
"csharpier"
],
"rollForward": false
},
"docfx": {
"version": "2.78.5",
"commands": [
"docfx"
],
"rollForward": false
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ on:
description: 'Package version from the release tag'
required: false
type: string
workflow_dispatch:
inputs:
version:
description: 'Package version to advertise on the site (e.g. 0.9.6-beta). Leave blank to use the default in versions.cjs.'
required: false
type: string

permissions:
contents: read
Expand All @@ -28,7 +34,7 @@ jobs:
- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: '9.0.x'
dotnet-version: '10.0.x'

- name: Setup Node.js
uses: actions/setup-node@v4
Expand All @@ -54,6 +60,11 @@ jobs:

- name: Build Eleventy site
working-directory: Website
env:
DATAPROVIDER_VERSION: ${{ inputs.version }}
DATAPROVIDERMIGRATE_VERSION: ${{ inputs.version }}
LQL_VERSION: ${{ inputs.version }}
NIMBLESITE_VERSION: ${{ inputs.version }}
run: npm run build

- name: Add GitHub Pages workflow to build output
Expand Down
149 changes: 142 additions & 7 deletions DataProvider/README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,155 @@
# DataProvider

A .NET source generator that creates compile-time safe database extension methods from SQL queries. Validates queries at compile time and generates strongly-typed C# code with `Result<T,E>` error handling.
A build-time CLI code generator for .NET that creates compile-time safe database extension methods from SQL and LQL query files. Every generated method returns `Result<T, SqlError>` — no exceptions, no reflection, no runtime overhead.

Supports SQLite and SQL Server. Works with both `.sql` and `.lql` files.
Supports **SQLite**, **PostgreSQL**, and **SQL Server**.

## Quick Start
## How it works

DataProvider is a **dotnet CLI tool**, not a Roslyn analyzer. It runs during the build, reads your SQL/LQL files plus a `DataProvider.json` manifest, and emits `.g.cs` files that your project compiles normally. The three tools form a pipeline:

```mermaid
flowchart LR
Yaml["example-schema.yaml"] -->|DataProviderMigrate| Db["invoices.db"]
Lql["GetCustomers.lql"] -->|Lql| Sql["GetCustomers.generated.sql"]
Config["DataProvider.json"] --> Gen["DataProvider"]
Sql --> Gen
Gen --> Cs["Generated/*.g.cs"]
```

## Install

```bash
dotnet new tool-manifest
dotnet tool install DataProvider --version ${DATAPROVIDER_VERSION}
dotnet add package Nimblesite.DataProvider.SQLite --version ${NIMBLESITE_VERSION}
```

Replace `SQLite` with `Postgres` or `SqlServer` as needed.

## Runtime packages

| Package | Purpose |
|---------|---------|
| `Nimblesite.DataProvider.Core` | Shared runtime types (`Result<T,E>`, `SqlError`) |
| `Nimblesite.DataProvider.SQLite` | SQLite runtime |
| `Nimblesite.DataProvider.Postgres` | PostgreSQL runtime |
| `Nimblesite.DataProvider.SqlServer` | SQL Server runtime |

## DataProvider.json

Describes what to generate from your SQL/LQL files and tables:

```json
{
"connectionString": "Data Source=app.db",
"queries": [
{
"name": "GetCustomers",
"sqlFile": "GetCustomers.generated.sql"
}
],
"tables": [
{
"schema": "main",
"name": "Customer",
"primaryKeyColumns": ["Id"],
"generateInsert": true,
"generateUpdate": true,
"generateDelete": true
}
]
}
```

## Running the generator

Manually:

```bash
dotnet DataProvider sqlite --project-dir . --config DataProvider.json --out ./Generated
dotnet DataProvider postgres --project-dir . --config DataProvider.json --out ./Generated --connection-string "Host=localhost;Database=mydb;..."
```

Or wire it into MSBuild so every build regenerates code:

```xml
<PackageReference Include="DataProvider.SQLite" Version="*" />
<Target Name="RunDataProvider" BeforeTargets="CoreCompile">
<Exec Command="dotnet DataProvider sqlite --project-dir . --config DataProvider.json --out ./Generated" />
<ItemGroup>
<Compile Include="Generated/**/*.g.cs" />
</ItemGroup>
</Target>
```

## Using generated methods (default template)

Out of the box, the generator emits methods that make errors explicit in the return type — no thrown exceptions on the query path:

```csharp
using Microsoft.Data.Sqlite;
using Nimblesite.DataProvider.Core;
using MyApp.Generated;

await using var connection = new SqliteConnection("Data Source=app.db");
await connection.OpenAsync();

var result = await connection.GetCustomersAsync(customerId: null);

switch (result)
{
case Result<IReadOnlyList<GetCustomersRow>, SqlError>.Ok ok:
foreach (var customer in ok.Value)
Console.WriteLine($"{customer.Id}: {customer.CustomerName}");
break;

case Result<IReadOnlyList<GetCustomersRow>, SqlError>.Error err:
Console.Error.WriteLine($"Query failed: {err.Value.Message}");
break;
}
```

Generated row types are immutable records. Generated insert/update/delete methods follow the same default shape.

## Customising generated code

The default output shape is **fully pluggable**. The code generator is driven by a `CodeGenerationConfig` record that holds a set of `Func<>` delegates — one per piece of the emitted code. Swap any of them and the generator emits whatever you want: raw `Task<T>`, `Option<T>`, thrown exceptions, custom result types, bespoke naming conventions, you name it.

Key extension points (see `Nimblesite.DataProvider.Core.CodeGeneration.CodeGenerationConfig`):

| Delegate | What it controls |
|---|---|
| `GenerateDataAccessMethod` | The signature and body of each query method (this is where you change the return type) |
| `GenerateModelType` | How row/parameter records are emitted |
| `GenerateGroupedModels` | How nested grouping models (parent + child collections) are emitted |
| `GenerateSourceFile` | The overall file layout (`using`s, namespace, class wrapper) |

Minimal custom template (programmatic):

```csharp
var result = await connection.GetCustomersAsync(isActive: true);
using Nimblesite.DataProvider.Core.CodeGeneration;

var config = new CodeGenerationConfig(getColumnMetadata, tableOpGenerator)
{
// Emit methods that throw instead of returning Result<T, SqlError>
GenerateDataAccessMethod = (className, methodName, sql, parameters, columns, connType) =>
$$"""
public static async Task<IReadOnlyList<{{className}}Row>> {{methodName}}Async(
this {{connType}} connection /* … */)
{
// your bespoke body
}
"""
};

// Pass the config to the platform-specific generator
SqliteCodeGenerator.GenerateCodeWithMetadata(config: config, /* … */);
```

## Documentation
> **Today this hook is programmatic-only** — custom templates are wired up in code that drives the generator library directly. The `DataProvider` CLI does not yet accept a `--template` flag or a `template` field in `DataProvider.json`; support for CLI-level template selection is tracked as future work. If you need a custom template right now, reference `Nimblesite.DataProvider.Core` from a small generator project and call `GenerateCodeWithMetadata` yourself.

## Related

- Full usage and configuration details are in the [DataProvider website docs](../Website/src/docs/dataprovider.md)
- [LQL](../Lql/README.md) — cross-database query language that transpiles to SQL
- [Migrations](../Migration/README.md) — YAML schema definitions consumed by `DataProviderMigrate`
- Migration CLI spec: [docs/specs/migration-cli-spec.md](../docs/specs/migration-cli-spec.md)
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 Nimblesite Pty Ltd

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
88 changes: 80 additions & 8 deletions Lql/README.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,99 @@
# Lambda Query Language (LQL)

A functional pipeline-style DSL that transpiles to SQL. Write database logic once, run it on PostgreSQL, SQLite, or SQL Server.
A functional, pipeline-style DSL that transpiles to SQL. Write database logic once and run it against **PostgreSQL**, **SQLite**, or **SQL Server**.

```lql
```
Users
|> filter(fn(row) => row.Age > 18 and row.Status = 'active')
|> join(Orders, on = Users.Id = Orders.UserId)
|> select(Users.Name, sum(Orders.Total) as TotalSpent)
|> group_by(Users.Id)
|> order_by(TotalSpent desc)
```

LQL is **database platform independent**. The same `.lql` source must produce semantically identical results on every target dialect.

## Install

### Build-time transpilation (recommended)

```bash
dotnet new tool-manifest
dotnet tool install Lql --version ${LQL_VERSION}
```

Then transpile during the build:

```bash
dotnet Lql sqlite --input GetCustomers.lql --output GetCustomers.generated.sql
dotnet Lql postgres --input GetCustomers.lql --output GetCustomers.generated.sql
```

### Runtime transpilation

Reference one of the library packages to transpile LQL in your application code:

```bash
dotnet add package Nimblesite.Lql.SQLite --version ${NIMBLESITE_VERSION}
dotnet add package Nimblesite.Lql.Postgres --version ${NIMBLESITE_VERSION}
dotnet add package Nimblesite.Lql.SqlServer --version ${NIMBLESITE_VERSION}
```

## Runtime API

```csharp
using Nimblesite.Lql.Core;
using Nimblesite.Lql.Postgres;
using Nimblesite.Sql.Model;

var lql = """
Customer
|> filter(fn(row) => Customer.Active = true)
|> select(Customer.Id, Customer.Name)
""";

var statement = new LqlStatement(lql);
Result<string, SqlError> result = statement.ToPostgreSql();

var sql = result switch
{
Result<string, SqlError>.Ok ok => ok.Value,
Result<string, SqlError>.Error err =>
throw new InvalidOperationException(err.Value.Message)
};
```

`statement.ToSqlite()` and `statement.ToSqlServer()` are also available from their respective packages.

## Projects

| Project | Description |
|---------|-------------|
| `Nimblesite.Lql.Core` | Core transpiler library |
| `Nimblesite.Lql.Core` | Core transpiler library and AST |
| `Lql` | Unified CLI transpiler tool (subcommands: `postgres`, `sqlite`) |
| `Nimblesite.Lql.Postgres` | `ToPostgreSql()` extension |
| `Nimblesite.Lql.SQLite` | `ToSqlite()` extension |
| `Nimblesite.Lql.SqlServer` | `ToSqlServer()` extension |
| `LqlExtension` | VS Code extension (TypeScript) |
| `lql-lsp-rust` | Language server (Rust) |
| `lql-lsp-rust` | Language server (Rust, ANTLR-generated parser) |
| `Nimblesite.Lql.TypeProvider.FSharp` | F# type provider for compile-time validation |

## Documentation
## Pipeline operators

| Operator | Purpose |
|----------|---------|
| `filter(fn(row) => ...)` | WHERE |
| `select(col1, col2, ...)` | SELECT projection |
| `join(Table, on = ...)` | INNER JOIN (plus `left_join`, `right_join`, `full_join`) |
| `group_by(col)` | GROUP BY |
| `order_by(col [asc|desc])` | ORDER BY |
| `limit(n)` | LIMIT / TOP |
| `distinct()` | DISTINCT |

Aggregates include `count`, `sum`, `avg`, `min`, `max`. Parameters are declared with `@name`.

## Related documentation

- LQL spec: [docs/specs/lql-spec.md](../docs/specs/lql-spec.md)
- LQL language spec: [docs/specs/lql-spec.md](../docs/specs/lql-spec.md)
- LQL design system: [docs/specs/lql-design-system.md](../docs/specs/lql-design-system.md)
- LQL reference (compiled into LSP): [lql-lsp-rust/crates/lql-reference.md](lql-lsp-rust/crates/lql-reference.md)
- Website docs: [Website/src/docs/lql.md](../Website/src/docs/lql.md)
- LSP reference (used by IDE): [lql-lsp-rust/crates/lql-reference.md](lql-lsp-rust/crates/lql-reference.md)
Loading
Loading