From feb1b83abfdc58661c969d8b0b5692a87aaf8173 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 10 Apr 2026 14:14:22 +0000 Subject: [PATCH 1/9] docs: fill documentation gaps for DBLab Engine 4.1 - Add protection lease docs: config reference (3 new params), clone protection howto rewritten with lease examples, CLI/API usage - Add Prometheus monitoring page: all metrics, PromQL examples, alerting rules, OpenTelemetry integration - Add Teleport integration howto: architecture, prerequisites, config, sidecar setup, troubleshooting - Add RDS/Aurora refresh howto: config, IAM policy, scheduling, orphan protection - Update CLI reference: add teleport command, protection duration flag semantics, snapshot delete --force - Update API reference: add v4.1 link, full endpoint summary table, v4.1 changes section - Update config reference: add databaseRename to logicalSnapshot and physicalSnapshot, add clone_delete webhook trigger - Update supported databases: drop PostgreSQL 9.6 (EOL Nov 2021) - Update version references from 4.0.3 to 4.1.0 across all howtos - Update OpenAPI spec references and sidebar navigation https://claude.ai/code/session_011sPDgBjzL2N2X6jiYyoTjQ --- docs/database-lab/index.md | 3 +- docs/database-lab/prometheus-monitoring.md | 245 ++++++++++++++++++ docs/database-lab/supported-databases.md | 1 - .../ci-observer-postgres-log-masking.md | 2 +- .../administration/data/custom.md | 4 +- docs/dblab-howtos/administration/data/dump.md | 4 +- .../administration/data/pg_basebackup.md | 4 +- .../administration/data/pgbackrest.md | 4 +- .../administration/data/rds-refresh.md | 234 +++++++++++++++++ docs/dblab-howtos/administration/data/rds.md | 8 +- .../dblab-howtos/administration/data/rsync.md | 4 +- .../dblab-howtos/administration/data/wal-g.md | 4 +- .../administration/engine-manage.md | 6 +- .../administration/install-dle-manually.md | 18 +- .../administration/teleport-integration.md | 203 +++++++++++++++ docs/dblab-howtos/cloning/clone-protection.md | 93 ++++++- .../database-lab-engine-api-reference.md | 80 +++++- ...base-lab-engine-configuration-reference.md | 18 +- .../dblab-client-cli-reference.md | 75 +++++- docs/tutorials/database-lab-tutorial.md | 2 +- sidebars.js | 3 + 21 files changed, 967 insertions(+), 48 deletions(-) create mode 100644 docs/database-lab/prometheus-monitoring.md create mode 100644 docs/dblab-howtos/administration/data/rds-refresh.md create mode 100644 docs/dblab-howtos/administration/teleport-integration.md diff --git a/docs/database-lab/index.md b/docs/database-lab/index.md index 1d8a2bbb..44bf8bf1 100644 --- a/docs/database-lab/index.md +++ b/docs/database-lab/index.md @@ -14,6 +14,7 @@ import useBaseUrl from '@docusaurus/useBaseUrl'; - [DBLab tutorial for Amazon RDS Postgres](/docs/tutorials/database-lab-tutorial-amazon-rds) - [Supported databases](/docs/database-lab/supported-databases) - [DBLab UI](/docs/database-lab/user-interface) +- [Prometheus monitoring](/docs/database-lab/prometheus-monitoring) - [Data masking](/docs/database-lab/masking) - [DB Migration Checker](/docs/database-lab/db-migration-checker) - [Telemetry](/docs/database-lab/telemetry) @@ -69,7 +70,7 @@ Some problems that can be solved by using DBLab: - Works well both on-premise and in clouds. - Thin provisioning in seconds thanks to copy-on-write (CoW) provided by [ZFS](https://en.wikipedia.org/wiki/ZFS) and a special methodology for preparing PostgreSQL database snapshots. There is also an option to use [LVM](https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)) instead of ZFS. - Unlimited size of databases (Postgres database size [is unlimited](https://www.postgresql.org/docs/current/limits.html), ZFS volume can be up to 21^28 bytes, or [256 trillion yobibytes](https://en.wikipedia.org/wiki/ZFS)). -- Supports PostgreSQL from version 9.6 up to the most recently released version. +- Supports PostgreSQL from version 10 up to the most recently released version. - Thin cloning takes only a few seconds, regardless of the database size. - REST API. - Client CLI included. diff --git a/docs/database-lab/prometheus-monitoring.md b/docs/database-lab/prometheus-monitoring.md new file mode 100644 index 00000000..5561f036 --- /dev/null +++ b/docs/database-lab/prometheus-monitoring.md @@ -0,0 +1,245 @@ +--- +title: Prometheus monitoring +sidebar_label: Prometheus monitoring +--- + +DBLab Engine exposes Prometheus metrics via the `/metrics` endpoint. These metrics can be used to monitor the health and performance of the DBLab instance. + +:::note +Prometheus metrics support was added in DBLab Engine 4.1. +::: + +## Endpoint + +``` +GET /metrics +``` + +The endpoint is publicly accessible (no authentication required) and returns metrics in Prometheus text format. + +## Available metrics + +### Instance metrics + +| Metric Name | Type | Labels | Description | +|-------------|------|--------|-------------| +| `dblab_instance_info` | Gauge | `instance_id`, `version`, `edition` | Information about the DBLab instance (always 1) | +| `dblab_instance_uptime_seconds` | Gauge | - | Time in seconds since the DBLab instance started | +| `dblab_instance_status_code` | Gauge | - | Status code of the DBLab instance (0=OK, 1=Warning, 2=Bad) | +| `dblab_retrieval_status` | Gauge | `mode`, `status` | Status of data retrieval (1=active for status) | + +### Disk/pool metrics + +| Metric Name | Type | Labels | Description | +|-------------|------|--------|-------------| +| `dblab_disk_total_bytes` | Gauge | `pool` | Total disk space in bytes | +| `dblab_disk_free_bytes` | Gauge | `pool` | Free disk space in bytes | +| `dblab_disk_used_bytes` | Gauge | `pool` | Used disk space in bytes | +| `dblab_disk_used_by_snapshots_bytes` | Gauge | `pool` | Disk space used by snapshots in bytes | +| `dblab_disk_used_by_clones_bytes` | Gauge | `pool` | Disk space used by clones in bytes | +| `dblab_disk_data_size_bytes` | Gauge | `pool` | Size of the data directory in bytes | +| `dblab_disk_compress_ratio` | Gauge | `pool` | Compression ratio of the filesystem (ZFS) | +| `dblab_pool_status` | Gauge | `pool`, `mode`, `status` | Status of the pool (1=active for status) | + +### Clone metrics (aggregate) + +| Metric Name | Type | Labels | Description | +|-------------|------|--------|-------------| +| `dblab_clones_total` | Gauge | - | Total number of clones | +| `dblab_clones_by_status` | Gauge | `status` | Number of clones by status | +| `dblab_clone_max_age_seconds` | Gauge | - | Maximum age of any clone in seconds | +| `dblab_clone_total_diff_size_bytes` | Gauge | - | Total extra disk space used by all clones (sum of diffs from snapshots) | +| `dblab_clone_total_logical_size_bytes` | Gauge | - | Total logical size of all clone data | +| `dblab_clone_total_cpu_usage_percent` | Gauge | - | Total CPU usage percentage across all clone containers | +| `dblab_clone_avg_cpu_usage_percent` | Gauge | - | Average CPU usage percentage across all clone containers with valid data | +| `dblab_clone_total_memory_usage_bytes` | Gauge | - | Total memory usage in bytes across all clone containers | +| `dblab_clone_total_memory_limit_bytes` | Gauge | - | Total memory limit in bytes across all clone containers | +| `dblab_clone_protected_count` | Gauge | - | Number of protected clones | + +### Snapshot metrics (aggregate) + +| Metric Name | Type | Labels | Description | +|-------------|------|--------|-------------| +| `dblab_snapshots_total` | Gauge | - | Total number of snapshots | +| `dblab_snapshots_by_pool` | Gauge | `pool` | Number of snapshots by pool | +| `dblab_snapshot_max_age_seconds` | Gauge | - | Maximum age of any snapshot in seconds | +| `dblab_snapshot_total_physical_size_bytes` | Gauge | - | Total physical disk space used by all snapshots | +| `dblab_snapshot_total_logical_size_bytes` | Gauge | - | Total logical size of all snapshot data | +| `dblab_snapshot_max_data_lag_seconds` | Gauge | - | Maximum data lag of any snapshot in seconds | +| `dblab_snapshot_total_num_clones` | Gauge | - | Total number of clones across all snapshots | + +### Branch metrics + +| Metric Name | Type | Labels | Description | +|-------------|------|--------|-------------| +| `dblab_branches_total` | Gauge | - | Total number of branches | + +### Dataset metrics + +| Metric Name | Type | Labels | Description | +|-------------|------|--------|-------------| +| `dblab_datasets_total` | Gauge | `pool` | Total number of datasets (slots) in the pool | +| `dblab_datasets_available` | Gauge | `pool` | Number of available (non-busy) dataset slots for reuse | + +### Sync instance metrics (physical mode) + +These metrics are only available when DBLab is running in physical mode with a sync instance enabled. They track the WAL replay status of the sync instance. + +| Metric Name | Type | Labels | Description | +|-------------|------|--------|-------------| +| `dblab_sync_status` | Gauge | `status` | Status of the sync instance (1=active for status code) | +| `dblab_sync_wal_lag_seconds` | Gauge | - | WAL replay lag in seconds for the sync instance | +| `dblab_sync_uptime_seconds` | Gauge | - | Uptime of the sync instance in seconds | +| `dblab_sync_last_replayed_timestamp` | Gauge | - | Unix timestamp of the last replayed transaction | + +### Observability metrics + +These metrics help monitor the health of the metrics collection system itself. + +| Metric Name | Type | Labels | Description | +|-------------|------|--------|-------------| +| `dblab_scrape_success_timestamp` | Gauge | - | Unix timestamp of last successful metrics collection | +| `dblab_scrape_duration_seconds` | Gauge | - | Duration of last metrics collection in seconds | +| `dblab_scrape_errors_total` | Counter | - | Total number of errors during metrics collection | + +## Prometheus configuration + +Add the following to your `prometheus.yml`: + +```yaml +scrape_configs: + - job_name: 'dblab' + static_configs: + - targets: [':'] + metrics_path: /metrics +``` + +Replace `` and `` with your DBLab instance's host and API port (default: `2345`). + +## Example queries + +### Free disk space percentage + +```promql +100 * dblab_disk_free_bytes / dblab_disk_total_bytes +``` + +### Number of active clones + +```promql +dblab_clones_total +``` + +### Maximum clone age in hours + +```promql +dblab_clone_max_age_seconds / 3600 +``` + +### Data freshness (lag from current time) + +```promql +dblab_snapshot_max_data_lag_seconds / 60 +``` + +### WAL replay lag (physical mode) + +```promql +dblab_sync_wal_lag_seconds +``` + +## Alerting examples + +### Low disk space alert + +```yaml +- alert: DBLabLowDiskSpace + expr: (dblab_disk_free_bytes / dblab_disk_total_bytes) * 100 < 20 + for: 5m + labels: + severity: warning + annotations: + summary: "DBLab low disk space" + description: "DBLab pool {{ $labels.pool }} has less than 20% free disk space" +``` + +### Stale snapshot alert + +```yaml +- alert: DBLabStaleSnapshot + expr: dblab_snapshot_max_data_lag_seconds > 86400 + for: 10m + labels: + severity: warning + annotations: + summary: "DBLab snapshot data is stale" + description: "DBLab snapshot data is more than 24 hours old" +``` + +### High clone count alert + +```yaml +- alert: DBLabHighCloneCount + expr: dblab_clones_total > 50 + for: 5m + labels: + severity: warning + annotations: + summary: "DBLab has many clones" + description: "DBLab has {{ $value }} clones running" +``` + +### High WAL replay lag alert (physical mode) + +```yaml +- alert: DBLabHighWALLag + expr: dblab_sync_wal_lag_seconds > 3600 + for: 10m + labels: + severity: warning + annotations: + summary: "DBLab sync instance has high WAL lag" + description: "DBLab sync instance WAL replay is {{ $value | humanizeDuration }} behind" +``` + +## OpenTelemetry integration + +DBLab metrics can be exported to OpenTelemetry-compatible backends using the OpenTelemetry Collector. This allows you to send metrics to Grafana Cloud, Datadog, New Relic, and other observability platforms. + +### Quick start + +1. Install the OpenTelemetry Collector: + ```bash + docker pull otel/opentelemetry-collector-contrib:latest + ``` + +2. Copy the example configuration from the DBLab Engine repository: + ```bash + cp engine/configs/otel-collector.example.yml otel-collector.yml + ``` + +3. Edit `otel-collector.yml` to configure your backend: + ```yaml + exporters: + otlp: + endpoint: "your-otlp-endpoint:4317" + headers: + Authorization: "Bearer " + ``` + +4. Run the collector: + ```bash + docker run -v $(pwd)/otel-collector.yml:/etc/otelcol/config.yaml \ + -p 4317:4317 -p 8889:8889 \ + otel/opentelemetry-collector-contrib:latest + ``` + +### Supported backends + +The OTel Collector can export to: +- **Grafana Cloud** — use OTLP exporter with Grafana Cloud endpoint +- **Datadog** — use the datadog exporter +- **New Relic** — use OTLP exporter with New Relic endpoint +- **Prometheus Remote Write** — use prometheusremotewrite exporter +- **AWS CloudWatch** — use awsemf exporter +- **Any OTLP-compatible backend** diff --git a/docs/database-lab/supported-databases.md b/docs/database-lab/supported-databases.md index 936b119b..4cb63d78 100644 --- a/docs/database-lab/supported-databases.md +++ b/docs/database-lab/supported-databases.md @@ -4,7 +4,6 @@ title: PostgreSQL versions and extensions supported in DBLab Engine ## PostgreSQL versions Currently, DBLab Engine fully supports the following [PostgreSQL major versions](https://www.postgresql.org/support/versioning/): -- 9.6 (released: 2016-09-29; EOL: 2021-11-11) - 10 (released: 2017-10-05; EOL: 2022-11-10) - 11 (released: 2018-10-18; EOL: 2023-11-09) - 12 (released: 2019-10-03; EOL: 2024-11-14) diff --git a/docs/dblab-howtos/administration/ci-observer-postgres-log-masking.md b/docs/dblab-howtos/administration/ci-observer-postgres-log-masking.md index 92ea3e0e..a29d7d72 100644 --- a/docs/dblab-howtos/administration/ci-observer-postgres-log-masking.md +++ b/docs/dblab-howtos/administration/ci-observer-postgres-log-masking.md @@ -14,7 +14,7 @@ keywords: ## Configure masking for PostgreSQL log When Database Lab's CI Observer is used for automated testing of database migrations, it stores PostgreSQL log in DBLab Platform's centralized storage. You can optionally configure masking rules for sensitive data in the PostgreSQL log. Such rules will be continuously applied before sending any PostgreSQL log entries to the Platform's storage. -You can define masking rules in the form of regular expressions. To do it, open the DBLab Engine configuration file (usually, `~/.dblab/engine/configs/server.yml`; see config file examples [here](https://gitlab.com/postgres-ai/database-lab/-/tree/v4.0.3/engine/configs)) and define subsection `replacementRules` in the section `replacementRules`. A basic example: +You can define masking rules in the form of regular expressions. To do it, open the DBLab Engine configuration file (usually, `~/.dblab/engine/configs/server.yml`; see config file examples [here](https://gitlab.com/postgres-ai/database-lab/-/tree/v4.1.0/engine/configs)) and define subsection `replacementRules` in the section `replacementRules`. A basic example: ```yaml observer: replacementRules: diff --git a/docs/dblab-howtos/administration/data/custom.md b/docs/dblab-howtos/administration/data/custom.md index 6ccd7857..1b098073 100644 --- a/docs/dblab-howtos/administration/data/custom.md +++ b/docs/dblab-howtos/administration/data/custom.md @@ -16,7 +16,7 @@ To set up it you need to use following jobs: - [physicalSnapshot](/docs/reference-guides/database-lab-engine-configuration-reference#job-physicalsnapshot) ### Options -Copy the example configuration file [`config.example.physical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.0.3/engine/configs/config.example.physical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml`. For demo purposes we've used `pg_basebackup` tool, but you can use any tool suitable for the task. Check and update the following options: +Copy the example configuration file [`config.example.physical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.1.0/engine/configs/config.example.physical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml`. For demo purposes we've used `pg_basebackup` tool, but you can use any tool suitable for the task. Check and update the following options: - Set secure `server:verificationToken`, it will be used to authorize API requests to the Engine - Set connection options in `physicalRestore:options:envs`, based on your tool - Set PostgreSQL commands in `physicalRestore:options:customTool`: @@ -43,7 +43,7 @@ sudo docker run \ --env DOCKER_API_VERSION=1.39 \ --detach \ --restart on-failure \ - postgresai/dblab-server:4.0.3 + postgresai/dblab-server:4.1.0 ``` :::info diff --git a/docs/dblab-howtos/administration/data/dump.md b/docs/dblab-howtos/administration/data/dump.md index eba65682..46bb29c6 100644 --- a/docs/dblab-howtos/administration/data/dump.md +++ b/docs/dblab-howtos/administration/data/dump.md @@ -15,7 +15,7 @@ In order to set up DBLab Engine to automatically get the data from database usin - [logicalSnapshot](/docs/reference-guides/database-lab-engine-configuration-reference#job-logicalsnapshot) ### Options -Copy the contents of configuration example [`config.example.logical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.0.3/engine/configs/config.example.logical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml` and update the following options: +Copy the contents of configuration example [`config.example.logical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.1.0/engine/configs/config.example.logical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml` and update the following options: - Set secure `server:verificationToken`, it will be used to authorize API requests to the Engine - Set connection options in `retrieval:spec:logicalDump:options:source:connection`: - `dbname`: database name to connect to @@ -44,7 +44,7 @@ sudo docker run \ --env DOCKER_API_VERSION=1.39 \ --detach \ --restart on-failure \ - postgresai/dblab-server:4.0.3 + postgresai/dblab-server:4.1.0 ``` You can use PGPASSWORD env to set the password. diff --git a/docs/dblab-howtos/administration/data/pg_basebackup.md b/docs/dblab-howtos/administration/data/pg_basebackup.md index 0d69ab03..a3d0d23e 100644 --- a/docs/dblab-howtos/administration/data/pg_basebackup.md +++ b/docs/dblab-howtos/administration/data/pg_basebackup.md @@ -14,7 +14,7 @@ In order to set up DBLab Engine to automatically get the data from database usin - [physicalSnapshot](/docs/reference-guides/database-lab-engine-configuration-reference#job-physicalsnapshot) ### Options -Copy the contents of configuration example [`config.example.physical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.0.3/engine/configs/config.example.physical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml` and update the following options: +Copy the contents of configuration example [`config.example.physical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.1.0/engine/configs/config.example.physical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml` and update the following options: - Set secure `server:verificationToken`, it will be used to authorize API requests to the Engine - Set connection options in `physicalRestore:options:envs`: - `PGUSER`: database user name @@ -44,7 +44,7 @@ sudo docker run \ --env DOCKER_API_VERSION=1.39 \ --detach \ --restart on-failure \ - postgresai/dblab-server:4.0.3 + postgresai/dblab-server:4.1.0 ``` :::info diff --git a/docs/dblab-howtos/administration/data/pgbackrest.md b/docs/dblab-howtos/administration/data/pgbackrest.md index 55261108..f88a5d21 100644 --- a/docs/dblab-howtos/administration/data/pgbackrest.md +++ b/docs/dblab-howtos/administration/data/pgbackrest.md @@ -16,7 +16,7 @@ In order to configure DLE to automatically restore the database using the [pgBac - [physicalSnapshot](/docs/reference-guides/database-lab-engine-configuration-reference#job-physicalsnapshot) ### Options -Copy the example configuration file [`config.example.physical_pgbackrest.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.0.3/engine/configs/config.example.physical_pgbackrest.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml` and update the following options: +Copy the example configuration file [`config.example.physical_pgbackrest.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.1.0/engine/configs/config.example.physical_pgbackrest.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml` and update the following options: - Set secure `server:verificationToken`, it will be used to authorize API requests to the Engine - Set repository options in `physicalRestore:options:envs`: - pgBackRest allows using environment variables instead of command-line options (see [pgBackRest docs](https://pgbackrest.org/command.html#introduction)): @@ -70,7 +70,7 @@ sudo docker run \ --env DOCKER_API_VERSION=1.39 \ --detach \ --restart on-failure \ - postgresai/dblab-server:4.0.3 + postgresai/dblab-server:4.1.0 ``` :::info diff --git a/docs/dblab-howtos/administration/data/rds-refresh.md b/docs/dblab-howtos/administration/data/rds-refresh.md new file mode 100644 index 00000000..e882228a --- /dev/null +++ b/docs/dblab-howtos/administration/data/rds-refresh.md @@ -0,0 +1,234 @@ +--- +title: "Data source: RDS/Aurora refresh" +sidebar_label: "RDS/Aurora refresh" +--- + +:::note +This component was added in DBLab Engine 4.1. +::: + +The RDS/Aurora refresh tool provides an alternative approach to refreshing DBLab data from Amazon RDS and Aurora databases. Instead of running `pg_dump` directly against production, it dumps from a **temporary RDS clone**, leaving production untouched. + +## Why use this approach? + +Running `pg_dump` directly against a production database can be problematic: +- **Holds xmin horizon for hours** leading to bloat accumulation +- **Creates load on production** for the duration of the dump +- **Requires direct network access** to the production database + +The RDS/Aurora refresh tool avoids all of these issues: + +``` +Production --> RDS Snapshot --> RDS Clone --> pg_dump --> DBLab + (automated) (temporary) +``` + +## Quick start + +### 1. Configure + +Create a configuration file: + +```yaml +source: + type: rds # or "aurora-cluster" + identifier: my-prod-db + dbName: postgres + username: postgres + password: ${DB_PASSWORD} + +clone: + instanceClass: db.t3.medium + securityGroups: [sg-xxx] # must allow DBLab inbound + +dblab: + apiEndpoint: https://dblab:2345 + token: ${DBLAB_TOKEN} + +aws: + region: us-east-1 +``` + +### 2. Test + +```bash +docker run --rm \ + -v $PWD/config.yaml:/config.yaml \ + -e DB_PASSWORD -e DBLAB_TOKEN -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \ + postgresai/rds-refresh -config /config.yaml -dry-run +``` + +### 3. Run + +```bash +docker run --rm \ + -v $PWD/config.yaml:/config.yaml \ + -e DB_PASSWORD -e DBLAB_TOKEN -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \ + postgresai/rds-refresh -config /config.yaml +``` + +## Configuration reference + +| Field | Required | Description | +|-------|----------|-------------| +| `source.type` | Yes | `rds` or `aurora-cluster` | +| `source.identifier` | Yes | RDS instance or Aurora cluster identifier | +| `source.dbName` | Yes | Database name | +| `source.username` | Yes | Database user | +| `source.password` | Yes | Password (supports `${ENV_VAR}` syntax) | +| `clone.instanceClass` | Yes | RDS clone instance type (e.g., `db.t3.medium`) | +| `clone.securityGroups` | No | Security groups allowing DBLab access | +| `clone.subnetGroup` | No | DB subnet group | +| `clone.maxAge` | No | Max age before clone is considered stale (default: `48h`) | +| `dblab.apiEndpoint` | Yes | DBLab API URL | +| `dblab.token` | Yes | DBLab verification token | +| `dblab.timeout` | No | Max refresh wait (default: `4h`) | +| `aws.region` | Yes | AWS region | + +## IAM policy + +The AWS user or role running the tool needs these permissions: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "rds:DescribeDBSnapshots", + "rds:DescribeDBClusterSnapshots", + "rds:DescribeDBInstances", + "rds:DescribeDBClusters" + ], + "Resource": "*" + }, + { + "Effect": "Allow", + "Action": [ + "rds:RestoreDBInstanceFromDBSnapshot", + "rds:RestoreDBClusterFromSnapshot", + "rds:CreateDBInstance", + "rds:DeleteDBInstance", + "rds:DeleteDBCluster", + "rds:AddTagsToResource", + "rds:ModifyDBInstance", + "rds:ModifyDBCluster" + ], + "Resource": [ + "arn:aws:rds:*:ACCOUNT:db:dblab-refresh-*", + "arn:aws:rds:*:ACCOUNT:cluster:dblab-refresh-*", + "arn:aws:rds:*:ACCOUNT:snapshot:*", + "arn:aws:rds:*:ACCOUNT:cluster-snapshot:*", + "arn:aws:rds:*:ACCOUNT:subgrp:*", + "arn:aws:rds:*:ACCOUNT:pg:*" + ] + } + ] +} +``` + +Replace `ACCOUNT` with your AWS account ID. + +## DBLab setup + +DBLab must run in **logical mode**. The tool updates config via API (no SSH required). + +```yaml +retrieval: + refresh: + timetable: "" # disable built-in scheduler — rds-refresh handles timing + jobs: [logicalDump, logicalRestore, logicalSnapshot] + spec: + logicalDump: + options: + source: + connection: + host: placeholder # updated by rds-refresh + port: 5432 +``` + +## Scheduling + +### Cron (weekly, Sunday 2 AM) + +```bash +0 2 * * 0 docker run --rm -v /etc/dblab/config.yaml:/config.yaml \ + --env-file /etc/dblab/env postgresai/rds-refresh -config /config.yaml +``` + +### Kubernetes CronJob + +```yaml +apiVersion: batch/v1 +kind: CronJob +metadata: + name: dblab-refresh +spec: + schedule: "0 2 * * 0" + concurrencyPolicy: Forbid + jobTemplate: + spec: + template: + spec: + serviceAccountName: dblab-refresh # IRSA + containers: + - name: refresh + image: postgresai/rds-refresh + args: ["-config", "/config/config.yaml"] + envFrom: + - secretRef: + name: dblab-refresh-secrets + volumeMounts: + - name: config + mountPath: /config + volumes: + - name: config + configMap: + name: dblab-refresh-config + restartPolicy: Never +``` + +## How it works + +1. **Startup cleanup**: check for orphaned clones from previous runs +2. Check DBLab health +3. Find latest RDS snapshot +4. Create RDS clone from RDS snapshot (`dblab-refresh-YYYYMMDD-HHMMSS`) +5. Wait for RDS clone to become available (~15 min) +6. Update DBLab config via API to point to the temporary clone +7. Trigger refresh, wait for completion +8. Delete RDS clone (always, even on error) + +## Orphan protection + +The tool has multiple layers of protection against orphaned RDS clones: + +1. **Defer cleanup**: clone is deleted when process exits normally +2. **Signal handlers**: catches SIGINT, SIGTERM, SIGHUP (SSH disconnect) +3. **State file**: tracks active clone in `./meta/rds-refresh.state` +4. **Tag scan**: finds clones by `ManagedBy=dblab-rds-refresh` tag + +### Manual cleanup + +```bash +# Dry run — see what would be deleted +rds-refresh cleanup -config config.yaml -dry-run + +# Delete stale clones older than 24 hours +rds-refresh cleanup -config config.yaml -max-age 24h +``` + +## Networking + +The RDS clone must be reachable from DBLab on port 5432. Use the same VPC or VPC peering. + +## Cost + +RDS clone cost is only incurred while running (~2-5 hours): +- `db.t3.medium`: ~$0.35 +- `db.r5.large`: ~$1.20 + +## Related +- [Data source: AWS RDS (direct)](/docs/dblab-howtos/administration/data/rds) +- [Logical full refresh](/docs/dblab-howtos/administration/logical-full-refresh) diff --git a/docs/dblab-howtos/administration/data/rds.md b/docs/dblab-howtos/administration/data/rds.md index 4629e598..250b13f1 100644 --- a/docs/dblab-howtos/administration/data/rds.md +++ b/docs/dblab-howtos/administration/data/rds.md @@ -25,7 +25,7 @@ If you want to use **IAM database authentication**, read how to enable it [here] You need to know the **master password**. If you lost the password it can be reset. Read how to reset it [here](https://aws.amazon.com/premiumsupport/knowledge-center/reset-master-user-password-rds/). ::: -Copy the contents of configuration example [`config.example.logical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.0.3/engine/configs/config.example.logical_generic.yml) from the DBLab Engine repository to `~/.dblab/engine/configs/server.yml` and update the following options: +Copy the contents of configuration example [`config.example.logical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.1.0/engine/configs/config.example.logical_generic.yml) from the DBLab Engine repository to `~/.dblab/engine/configs/server.yml` and update the following options: - Set secure `server:verificationToken`, it will be used to authorize API requests to the Engine - Set connection options in `retrieval:spec:logicalDump:options:source:connection`: - `dbname`: database name to connect to @@ -54,7 +54,7 @@ sudo docker run \ --env DOCKER_API_VERSION=1.39 \ --detach \ --restart on-failure \ - postgresai/dblab-server:4.0.3 + postgresai/dblab-server:4.1.0 ``` :::info @@ -80,7 +80,7 @@ Alternatively, you can add `AmazonRDSFullAccess`, `IAMFullAccess` policies to an ::: ### Set up and run DBLab Engine -Copy the contents of configuration example [`config.example.logical_rds_iam.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.0.3/engine/configs/config.example.logical_rds_iam.yml) from the DBLab Engine repository to `~/.dblab/engine/configs/server.yml` and update the following options: +Copy the contents of configuration example [`config.example.logical_rds_iam.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.1.0/engine/configs/config.example.logical_rds_iam.yml) from the DBLab Engine repository to `~/.dblab/engine/configs/server.yml` and update the following options: - Set secure `server:verificationToken`, it will be used to authorize API requests to the Engine - Set connection options `retrieval:spec:logicalDump:options:source:connection`: - `dbname`: database name to connect to @@ -120,7 +120,7 @@ sudo docker run \ --env DOCKER_API_VERSION=1.39 \ --detach \ --restart on-failure \ - postgresai/dblab-server:4.0.3 + postgresai/dblab-server:4.1.0 ``` :::info diff --git a/docs/dblab-howtos/administration/data/rsync.md b/docs/dblab-howtos/administration/data/rsync.md index 2b847f1f..19755155 100644 --- a/docs/dblab-howtos/administration/data/rsync.md +++ b/docs/dblab-howtos/administration/data/rsync.md @@ -14,7 +14,7 @@ In order to set up DBLab Engine to automatically get the data from database usin - [physicalSnapshot](/docs/reference-guides/database-lab-engine-configuration-reference#job-physicalsnapshot) ### Options -Copy the example configuration file [`config.example.physical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.0.3/engine/configs/config.example.physical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml` and update the following options: +Copy the example configuration file [`config.example.physical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.1.0/engine/configs/config.example.physical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml` and update the following options: - Set secure `server:verificationToken`, it will be used to authorize API requests to the Engine - Set connection options in `physicalRestore:options:envs`: - `PGUSER`: database user name @@ -54,7 +54,7 @@ sudo docker run \ --env DOCKER_API_VERSION=1.39 \ --detach \ --restart on-failure \ - postgresai/dblab-server:4.0.3 + postgresai/dblab-server:4.1.0 ``` :::info diff --git a/docs/dblab-howtos/administration/data/wal-g.md b/docs/dblab-howtos/administration/data/wal-g.md index 1ef16984..e94f3a60 100644 --- a/docs/dblab-howtos/administration/data/wal-g.md +++ b/docs/dblab-howtos/administration/data/wal-g.md @@ -14,7 +14,7 @@ In order to set up DBLab Engine to automatically get the data from database usin - [physicalSnapshot](/docs/reference-guides/database-lab-engine-configuration-reference#job-physicalsnapshot) ### Options -Copy the example configuration file [`config.example.physical_walg.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.0.3/engine/configs/config.example.physical_walg.yml) from the DBLab Engine repository to `~/.dblab/engine/configs/server.yml` and update the following options: +Copy the example configuration file [`config.example.physical_walg.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.1.0/engine/configs/config.example.physical_walg.yml) from the DBLab Engine repository to `~/.dblab/engine/configs/server.yml` and update the following options: - Set secure `server:verificationToken`, it will be used to authorize API requests to the Engine - Set connection options in `physicalRestore:options:envs`: - Use WAL-G environment variables to configure the job, see the [WAL-G configuration reference](https://github.com/wal-g/wal-g#configuration) @@ -50,7 +50,7 @@ sudo docker run \ --env DOCKER_API_VERSION=1.39 \ --detach \ --restart on-failure \ - postgresai/dblab-server:4.0.3 + postgresai/dblab-server:4.1.0 ``` :::info diff --git a/docs/dblab-howtos/administration/engine-manage.md b/docs/dblab-howtos/administration/engine-manage.md index 3c8df329..13e442dc 100644 --- a/docs/dblab-howtos/administration/engine-manage.md +++ b/docs/dblab-howtos/administration/engine-manage.md @@ -16,7 +16,7 @@ All YAML features can be used, including anchors and aliases, to help you conven For instance, you can define a binding with `&` and then refer to it using an alias denoted by `*`. -See config examples [here](https://gitlab.com/postgres-ai/database-lab/-/tree/v4.0.3/engine/configs) +See config examples [here](https://gitlab.com/postgres-ai/database-lab/-/tree/v4.1.0/engine/configs) ::: After configuring DBLab Engine, run the following command: @@ -37,7 +37,7 @@ sudo docker run \ --volume /lib/modules:/lib/modules:ro \ --volume /proc:/host_proc:ro \ --env DOCKER_API_VERSION=1.41 \ - postgresai/dblab-server:4.0.3 + postgresai/dblab-server:4.1.0 ``` :::info @@ -105,7 +105,7 @@ If you need to save the logs in a file: sudo docker logs dblab_server 2>&1 | gzip > dblab_server.log.gz ``` -If you want to see more details, enable debug mode setting option `debug` to `true` (see [example](https://gitlab.com/postgres-ai/database-lab/-/tree/v4.0.3/engine/configs)). Next, follow [the reconfiguration guidelines](#reconfigure-database-lab) to apply the change. +If you want to see more details, enable debug mode setting option `debug` to `true` (see [example](https://gitlab.com/postgres-ai/database-lab/-/tree/v4.1.0/engine/configs)). Next, follow [the reconfiguration guidelines](#reconfigure-database-lab) to apply the change. :::caution When debug mode is turned on, logs may contain sensitive data such as API secret keys for the backup system. diff --git a/docs/dblab-howtos/administration/install-dle-manually.md b/docs/dblab-howtos/administration/install-dle-manually.md index f4daa699..78639180 100644 --- a/docs/dblab-howtos/administration/install-dle-manually.md +++ b/docs/dblab-howtos/administration/install-dle-manually.md @@ -242,11 +242,11 @@ sudo docker stop dblab_pg_initdb sudo docker rm dblab_pg_initdb ``` -Now, we need to take care of DBLab Engine configuration. Copy the contents of configuration example [`config.example.logical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.0.3/engine/configs/config.example.logical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml`: +Now, we need to take care of DBLab Engine configuration. Copy the contents of configuration example [`config.example.logical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.1.0/engine/configs/config.example.logical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml`: ```bash mkdir -p ~/.dblab/engine/configs -curl -fsSL https://gitlab.com/postgres-ai/database-lab/-/raw/v4.0.3/engine/configs/config.example.logical_generic.yml \ +curl -fsSL https://gitlab.com/postgres-ai/database-lab/-/raw/v4.1.0/engine/configs/config.example.logical_generic.yml \ --output ~/.dblab/engine/configs/server.yml ``` @@ -263,11 +263,11 @@ Open `~/.dblab/engine/configs/server.yml` and edit the following options: If you want to try Database Lab for an existing database, you need to copy the data to PostgreSQL data directory on the Database Lab server, to the directory `/var/lib/dblab/dblab_pool/data`. This step is called "thick cloning". It only needs to be completed once. There are several options to physically copy the data directory. Here we will use the standard PostgreSQL tool, `pg_basebackup`. However, we are not going to use it directly (although, it is possible) – we will specify its options in the DBLab Engine configuration file. -First, copy the example configuration file [`config.example.physical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.0.3/engine/configs/config.example.physical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml`: +First, copy the example configuration file [`config.example.physical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.1.0/engine/configs/config.example.physical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml`: ```bash mkdir -p ~/.dblab/engine/configs -curl -fsSL https://gitlab.com/postgres-ai/database-lab/-/raw/v4.0.3/engine/configs/config.example.physical_generic.yml \ +curl -fsSL https://gitlab.com/postgres-ai/database-lab/-/raw/v4.1.0/engine/configs/config.example.physical_generic.yml \ --output ~/.dblab/engine/configs/server.yml ``` @@ -288,11 +288,11 @@ If you want to try Database Lab for an existing database, you need to copy the d Here we will configure DBLab Engine to use a "logical" method of thick cloning, dump/restore. -First, copy the configuration example configuration file [`config.example.logical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.0.3/engine/configs/config.example.logical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml`: +First, copy the configuration example configuration file [`config.example.logical_generic.yml`](https://gitlab.com/postgres-ai/database-lab/-/blob/v4.1.0/engine/configs/config.example.logical_generic.yml) from the Database Lab repository to `~/.dblab/engine/configs/server.yml`: ```bash mkdir -p ~/.dblab/engine/configs -curl -fsSL https://gitlab.com/postgres-ai/database-lab/-/raw/v4.0.3/engine/configs/config.example.logical_generic.yml \ +curl -fsSL https://gitlab.com/postgres-ai/database-lab/-/raw/v4.1.0/engine/configs/config.example.logical_generic.yml \ --output ~/.dblab/engine/configs/server.yml ``` @@ -340,7 +340,7 @@ sudo docker run \ --env DOCKER_API_VERSION=1.39 \ --detach \ --restart on-failure \ - postgresai/dblab-server:4.0.3 + postgresai/dblab-server:4.1.0 ``` @@ -363,7 +363,7 @@ sudo docker run \ --env DOCKER_API_VERSION=1.39 \ --detach \ --restart on-failure \ - postgresai/dblab-server:4.0.3 + postgresai/dblab-server:4.1.0 ``` @@ -386,7 +386,7 @@ sudo docker run \ --env DOCKER_API_VERSION=1.39 \ --detach \ --restart on-failure \ - postgresai/dblab-server:4.0.3 + postgresai/dblab-server:4.1.0 ``` diff --git a/docs/dblab-howtos/administration/teleport-integration.md b/docs/dblab-howtos/administration/teleport-integration.md new file mode 100644 index 00000000..715680e8 --- /dev/null +++ b/docs/dblab-howtos/administration/teleport-integration.md @@ -0,0 +1,203 @@ +--- +title: Teleport integration +sidebar_label: Teleport integration +--- + +DBLab Engine 4.1 includes built-in integration with [Teleport](https://goteleport.com/), enabling secure, audited access to database clones through Teleport's access control. The integration works as a sidecar process that automatically registers and deregisters DBLab clones as Teleport database resources. + +## Architecture + +``` ++--------------+ webhooks +------------------+ tctl +------------------+ +| DBLab Engine |-------------->| dblab teleport |---------->| Teleport Auth | +| (Docker) | | serve (sidecar) | | Server | ++--------------+ +------------------+ +------------------+ + | | + | clone containers | + v v ++--------------+ +------------------+ +------------------+ +| Clone PG |<--------------| Teleport DB Agent|<----------| tsh proxy db | +| (port 6000) | proxied | (db_service) | tunnel | (end user) | ++--------------+ +------------------+ +------------------+ +``` + +The sidecar: +- Receives `clone_create` / `clone_delete` webhooks from DBLab Engine +- Calls `tctl create` / `tctl rm` to register/deregister Teleport DB resources +- Runs startup reconciliation to catch missed events + +The sidecar does **not** proxy database connections. A separate Teleport agent with `db_service` enabled handles the actual proxying. + +## Prerequisites + +### 1. Teleport bot role + +Create a bot role with permissions to manage database resources: + +```yaml +kind: role +version: v7 +metadata: + name: dblab-bot +spec: + allow: + db_labels: + '*': '*' + db_names: ['*'] + db_users: ['*'] + rules: + - resources: [db, db_server] + verbs: [list, create, read, update, delete] + - resources: [app, app_server] + verbs: [list, create, read, update, delete] +``` + +Apply with `tctl create -f dblab-bot-role.yaml`. + +### 2. Teleport bot identity + +Create a bot and generate the identity file. The role from step 1 must already exist before this step. + +**Self-hosted Teleport:** +```bash +tctl bots add dblab-sidecar --roles=dblab-bot +tctl auth sign --format=tls --user=bot-dblab-sidecar -o /etc/teleport/dblab-identity +``` + +**Teleport Cloud:** +```bash +tctl bots add dblab-sidecar --roles=dblab-bot +# Use the token from the output above +tbot start --oneshot \ + --token= \ + --proxy-server=yourcluster.teleport.sh:443 \ + --join-method=token \ + --data-dir=/etc/teleport/bot-data \ + --destination-dir=/etc/teleport/bot-dest +# The identity file is at /etc/teleport/bot-dest/identity +``` + +### 3. Teleport database agent + +A Teleport agent must run on the DBLab host with `db_service` enabled: + +```yaml +# /etc/teleport.yaml (on the DBLab host) +db_service: + enabled: true + resources: + - labels: + dblab: "true" +``` + +### 4. SSL/TLS for Postgres clones + +Teleport always initiates TLS to backend databases. DBLab clones must have SSL enabled. + +**Generate self-signed certs:** +```bash +openssl req -new -x509 -days 3650 -nodes \ + -out /etc/dblab/certs/server.crt \ + -keyout /etc/dblab/certs/server.key \ + -subj "/CN=dblab-clone" + +chown 999:999 /etc/dblab/certs/server.crt /etc/dblab/certs/server.key +chmod 600 /etc/dblab/certs/server.key +``` + +**Export the Teleport DB CA certificate:** +```bash +tctl auth export --type=db-client > /etc/dblab/certs/teleport-ca.crt +chown 999:999 /etc/dblab/certs/teleport-ca.crt +``` + +### 5. pg_hba.conf — certificate authentication + +Starting with DBLab Engine 4.1, the default `pg_hba.conf` includes a `hostssl ... cert` rule that enables Teleport certificate authentication out of the box: + +``` +local all all trust +hostssl all all 0.0.0.0/0 cert +host all all 0.0.0.0/0 md5 +``` + +No custom `pg_hba.conf` or volume mount is required for Teleport. + +## Configuration + +### DBLab Engine server.yml + +Add SSL configuration and webhook settings: + +```yaml +databaseContainer: &db_container + dockerImage: "postgresai/extended-postgres:16" + containerConfig: + "shm-size": 1gb + volume: "/etc/dblab/certs:/var/lib/postgresql/cert:ro" + +databaseConfigs: &db_configs + configs: + ssl: "on" + ssl_cert_file: "/var/lib/postgresql/cert/server.crt" + ssl_key_file: "/var/lib/postgresql/cert/server.key" + ssl_ca_file: "/var/lib/postgresql/cert/teleport-ca.crt" + +webhooks: + hooks: + - url: "http://host.docker.internal:9876/teleport-sync" + secret: "your-webhook-secret" + trigger: + - clone_create + - clone_delete +``` + +:::tip +After adding or changing `databaseConfigs`, a data refresh is required. These settings are applied during snapshot creation. Existing snapshots are not affected. +::: + +## Running the sidecar + +```bash +dblab teleport serve \ + --environment-id production \ + --teleport-proxy teleport.example.com:3025 \ + --teleport-identity /etc/teleport/dblab-identity \ + --listen-addr 0.0.0.0:9876 \ + --dblab-url http://localhost:2345 \ + --dblab-token "$DBLAB_TOKEN" \ + --webhook-secret "$WEBHOOK_SECRET" +``` + +See the [CLI reference](/docs/reference-guides/dblab-client-cli-reference#command-teleport) for all available options. + +## Connecting to a clone + +Once everything is running, users connect through Teleport: + +```bash +# Login to Teleport +tsh login --proxy=teleport.example.com + +# List available databases (clones appear automatically) +tsh db ls + +# Connect to a clone +tsh db connect dblab-clone-production--6000 \ + --db-user postgres --db-name postgres + +# Or use a local tunnel (works with any psql client) +tsh proxy db --tunnel dblab-clone-production--6000 +``` + +## Troubleshooting + +| Symptom | Cause | Fix | +|---------|-------|-----| +| Clone registered but can't connect | No Teleport DB agent running | Start `teleport` with `db_service.enabled: true` | +| TLS handshake failure | Clone doesn't have SSL enabled | Add `ssl: "on"` + cert paths to `databaseConfigs.configs` | +| "no pg_hba.conf entry" | Missing `hostssl ... cert` entry | Upgrade to DBLab Engine 4.1+ which includes this rule by default | +| "root certificate store not available" | Missing `ssl_ca_file` | Export Teleport DB CA with `tctl auth export --type=db-client` | +| SSL settings not applied to new clones | Snapshot created before SSL config | Trigger a data refresh to create a new snapshot | +| Webhook not received | Docker networking issue | Use `host.docker.internal` or bridge IP for webhook URL | +| Permission denied on cert files | Wrong file ownership | `chown 999:999` on cert files | diff --git a/docs/dblab-howtos/cloning/clone-protection.md b/docs/dblab-howtos/cloning/clone-protection.md index 92acdd0a..28fae330 100644 --- a/docs/dblab-howtos/cloning/clone-protection.md +++ b/docs/dblab-howtos/cloning/clone-protection.md @@ -13,6 +13,35 @@ DBLab Engine automatically deletes idle unprotected clones after the idle interv Please be careful: abandoned protected clones may cause out-of-disk-space events. Check disk space on a daily basis and delete protected clones once the work is done. ::: +## Protection leases (DBLab Engine 4.1+) + +Starting with DBLab Engine 4.1, clone protection supports **time-limited leases**. Instead of protecting a clone indefinitely, you can specify a duration after which the protection automatically expires. This prevents abandoned protected clones from consuming disk space indefinitely. + +### How it works + +- When a clone is protected with a lease, the `protectedTill` field indicates when protection expires +- After expiration, the clone becomes unprotected and subject to normal idle deletion rules +- The default lease duration and maximum allowed duration are configurable + +### Configuration + +Configure protection lease defaults in the `cloning` section of `server.yml`: + +```yaml +cloning: + accessHost: "localhost" + maxIdleMinutes: 120 + protectionLeaseDurationMinutes: 1440 # Default: 1 day + protectionMaxDurationMinutes: 10080 # Maximum: 7 days + protectionExpiryWarningMinutes: 1440 # Warning webhook 24 hours before expiry +``` + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `protectionLeaseDurationMinutes` | 1440 (1 day) | Default lease duration when `--protected true` is used. `0` means infinite protection. | +| `protectionMaxDurationMinutes` | 10080 (7 days) | Maximum duration users can request. `0` means no limit. | +| `protectionExpiryWarningMinutes` | 1440 (1 day) | Send webhook warning this many minutes before expiry. | + ## GUI From the **Database Lab clone** page enable or disable the **Enable deletion protection** checkbox. ![DBLab Engine page / Create clone](/assets/guides/clone-protection-1.png) @@ -21,18 +50,23 @@ From the **Database Lab clone** page enable or disable the **Enable deletion pro Before you run any commands, install Database Lab CLI and initialize configuration. For more information, see [Install and initialize Database Lab CLI](/docs/dblab-howtos/cli/cli-install-init). ### Reference +- Command [`dblab clone create`](/docs/reference-guides/dblab-client-cli-reference#subcommand-create) - Command [`dblab clone update`](/docs/reference-guides/dblab-client-cli-reference#subcommand-update) -### Protect a clone +### Protect a clone with default lease duration ```bash -dblab clone update --protected CLONE_ID +dblab clone update --protected true CLONE_ID ``` +### Protect a clone for a specific duration (in minutes) +```bash +# Protect for 8 hours (480 minutes) +dblab clone update --protected 480 CLONE_ID ``` -{ - "id": "CLONE_ID", - "protected": true, -} + +### Protect a clone indefinitely (no expiry) +```bash +dblab clone update --protected 0 CLONE_ID ``` ### Unprotect a clone @@ -40,14 +74,53 @@ dblab clone update --protected CLONE_ID dblab clone update CLONE_ID ``` +### Protect at creation time +```bash +# With default lease duration +dblab clone create --username user --password pass --protected true --id my-clone + +# With custom duration (2 hours) +dblab clone create --username user --password pass --protected 120 --id my-clone +``` + +## API + +### Protect with lease duration +```bash +curl -X PATCH \ + -H "Verification-Token: YOUR_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"protected": true, "protectionDurationMinutes": 480}' \ + http://localhost:2345/clone/CLONE_ID +``` + +### Create a protected clone +```bash +curl -X POST \ + -H "Verification-Token: YOUR_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "id": "my-clone", + "protected": true, + "protectionDurationMinutes": 1440, + "db": {"username": "user", "password": "pass"} + }' \ + http://localhost:2345/clone ``` + +The response includes `protectedTill` showing when protection expires: +```json { - "id": "CLONE_ID", - "protected": false, + "id": "my-clone", + "protected": true, + "protectedTill": "2026-04-11T14:00:00Z", + "metadata": { + "protectionLeaseDurationMinutes": 1440, + "protectionMaxDurationMinutes": 10080 + } } ``` -Also, clones can marked as protected at creation time. See [Create a clone](/docs/dblab-howtos/cloning/create-clone). - ## Related - Guide: [Destroy a clone](/docs/dblab-howtos/cloning/destroy-clone) +- Reference: [Cloning configuration](/docs/reference-guides/database-lab-engine-configuration-reference#section-cloning-thin-cloning-policies) diff --git a/docs/reference-guides/database-lab-engine-api-reference.md b/docs/reference-guides/database-lab-engine-api-reference.md index b2ade16a..b621fdb1 100644 --- a/docs/reference-guides/database-lab-engine-api-reference.md +++ b/docs/reference-guides/database-lab-engine-api-reference.md @@ -1,7 +1,7 @@ --- title: DBLab API reference sidebar_label: DBLab API -description: "API reference for DBLab Engine – Swagger, OpenAPI" +description: "API reference for DBLab Engine -- Swagger, OpenAPI" keywords: - "database lab API" - "dblab engine API" @@ -16,7 +16,83 @@ DBLab API (DLE API) is a REST API. It can be used in multiple ways: - indirectly, in browser: [DBLab UI](https://postgres.ai/docs/database-lab/user-interface), being a React application, speaks to the DLE API as well DBLab API reference documentation is available at the following locations: -- [DLE 3.5.x API Reference](https://dblab.readme.io/v3.5.0/) +- [DBLab 4.1.x API Reference](https://dblab.readme.io/v4.1.0/) - [DBLab 4.0.x API Reference](https://dblab.readme.io/v4.0.0/) +- [DLE 3.5.x API Reference](https://dblab.readme.io/v3.5.0/) The references are published using the comprehensive ReadMe service, equipped with a developer dashboard and provides code snippets in numerous languages. + +## Authentication + +All API endpoints (except `/healthz` and `/metrics`) require the `Verification-Token` header: + +```bash +curl -H "Verification-Token: YOUR_TOKEN" http://localhost:2345/status +``` + +## Endpoint summary + +### Instance + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/status` | Instance status, info, and list of clones | +| GET | `/healthz` | Health check (no auth required) | +| GET | `/metrics` | Prometheus metrics (no auth required, DLE 4.1+) | +| GET | `/instance/retrieval` | Data refresh status | +| POST | `/full-refresh` | Trigger full data refresh (DLE 4.0+) | + +### Clones + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/clones` | List all clones (DLE 4.0+) | +| POST | `/clone` | Create a clone | +| GET | `/clone/{id}` | Retrieve a clone | +| PATCH | `/clone/{id}` | Update a clone (protection status) | +| DELETE | `/clone/{id}` | Delete a clone | +| POST | `/clone/{id}/reset` | Reset a clone to a snapshot | + +### Snapshots + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/snapshots` | List all snapshots | +| POST | `/snapshot` | Create a snapshot (DLE 4.0+) | +| DELETE | `/snapshot/{id}` | Delete a snapshot (DLE 4.0+) | +| GET | `/branch/snapshot/{id}` | Retrieve a snapshot (DLE 4.0+) | +| POST | `/branch/snapshot` | Create a snapshot from clone (DLE 4.0+) | + +### Branches (DLE 4.0+) + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/branches` | List all branches | +| POST | `/branch` | Create a branch | +| DELETE | `/branch/{branchName}` | Delete a branch | +| GET | `/branch/{branchName}/log` | Retrieve branch log (snapshot history) | + +### Observation (experimental) + +| Method | Path | Description | +|--------|------|-------------| +| POST | `/observation/start` | Start observation session | +| POST | `/observation/stop` | Stop observation session | +| GET | `/observation/summary/{clone_id}/{session_id}` | Get observation summary | +| GET | `/observation/download` | Download observation artifact | + +### Admin + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/admin/config` | Get config (JSON) | +| POST | `/admin/config` | Set config | +| GET | `/admin/config.yaml` | Get full config (YAML) | +| POST | `/admin/test-db-source` | Test source database connection | +| POST | `/admin/ws-auth` | WebSocket authentication | + +## New in DBLab Engine 4.1 + +- **`/metrics` endpoint**: Prometheus metrics for monitoring (no authentication required). See [Prometheus monitoring](/docs/database-lab/prometheus-monitoring). +- **Protection leases**: The `CreateClone` and `UpdateClone` requests now accept a `protectionDurationMinutes` field for time-limited clone protection. The `Clone` response includes `protectedTill` showing when protection expires. See [Clone protection](/docs/dblab-howtos/cloning/clone-protection). +- **`clone_delete` webhook**: A new webhook trigger type for clone deletion events. See [Webhook configuration](/docs/reference-guides/database-lab-engine-configuration-reference#section-webhooks-webhook-configuration). diff --git a/docs/reference-guides/database-lab-engine-configuration-reference.md b/docs/reference-guides/database-lab-engine-configuration-reference.md index 17b5615d..406a02ba 100644 --- a/docs/reference-guides/database-lab-engine-configuration-reference.md +++ b/docs/reference-guides/database-lab-engine-configuration-reference.md @@ -10,7 +10,7 @@ DBLab Engine behavior can be controlled using the main configuration file that h DBLab Engine supports [YAML 1.2](https://yaml.org/spec/1.2/spec.html) including anchors, aliases, tags, map merging. ::: -Example config files can be found here: https://gitlab.com/postgres-ai/database-lab/-/tree/v4.0.3/engine/configs. +Example config files can be found here: https://gitlab.com/postgres-ai/database-lab/-/tree/v4.1.0/engine/configs. You may store configuration files in any suitable location. The recommended location of configuration files for DBLab Engine is `~/.dblab/engine/configs`. @@ -279,6 +279,12 @@ Options: Prepares a snapshot for logical restored PostgreSQL database. Options: +- `databaseRename` (key-value, optional) - rename databases before finalizing the snapshot. Runs after `preprocessingScript`. Each entry maps the original database name to the new name. This is useful when you want clones to use different database names than production (e.g., renaming `mydb_prod` to `mydb_dev`). Supported since DBLab Engine 4.1. + ```yaml + databaseRename: + mydb_prod: mydb_dev + analytics_production: analytics_dblab + ``` - `dataPatching` (key-value, optional) - defines SQL queries for data patching. This allows you to run custom SQL queries against the restored database before creating the snapshot, useful for data masking, test data setup, or schema modifications - `dockerImage` (string, optional) - specifies the Docker image to run a data patching container. Can be inherited using YAML anchors (see `databaseContainer` pattern above) - `containerConfig` (key-value, optional) - options to pass custom parameters to data patching container. Supports standard Docker options like memory/CPU limits @@ -336,6 +342,12 @@ Options: - `configs` (key-value, optional) - applies PostgreSQL configuration parameters to the promotion instance - `sysctls` (key-value, optional) - allows configuring namespaced kernel parameters (sysctls) of Docker container for a promotion stage of taking a snapshot. See supported parameters: https://docs.docker.com/reference/cli/docker/container/run/#sysctl - `preprocessingScript` (string, optional) - path on the host machine to a pre-processing script +- `databaseRename` (key-value, optional) - rename databases before finalizing the snapshot. Runs after `preprocessingScript`. Each entry maps the original database name to the new name. Supported since DBLab Engine 4.1. + ```yaml + databaseRename: + example_production: example_dblab + analytics_prod: analytics_dblab + ``` - `configs` (key-value, optional) - applies PostgreSQL configuration parameters to snapshot. These parameters are inherited by all clones. See also: [How to configure PostgreSQL used by DBLab Engine](/docs/dblab-howtos/administration/postgresql-configuration) - `envs` (key-value, optional) - passes custom environment variables to the promotion Docker container - `scheduler` (key-value, required) - contains tasks which run on a schedule: @@ -348,6 +360,9 @@ Options: ## Section `cloning`: thin cloning policies - `accessHost` (string, required) - the host that will be specified in the database connection string to inform users about how to connect to database clones. This should match one of the addresses specified in `provision.cloneAccessAddresses` or be a hostname that resolves to one of those addresses. Use public IP address if database connections are allowed from outside, or "localhost"/private IP for local-only access. - `maxIdleMinutes` (integer, optional, default: 120) - automatically delete clones after the specified minutes of inactivity, 0 is being used to disable this feature. Inactivity means no active sessions (queries being processed) and no recently logged queries in the query log. +- `protectionLeaseDurationMinutes` (integer, optional, default: 1440) - default protection lease duration in minutes when a clone is marked as protected. When a clone is protected with a lease, it will automatically become unprotected after this duration elapses. Use `0` for infinite protection (no automatic expiration). Supported since DBLab Engine 4.1. +- `protectionMaxDurationMinutes` (integer, optional, default: 10080) - maximum allowed protection duration in minutes. Users cannot request a protection duration longer than this value. Use `0` to remove the limit. Supported since DBLab Engine 4.1. +- `protectionExpiryWarningMinutes` (integer, optional, default: 1440) - send a warning webhook notification the specified number of minutes before a protection lease expires. Supported since DBLab Engine 4.1. ## Section `platform`: PostgresAI Platform integration - `url` (string, optional, default: "https://postgres.ai/api/general") - Platform API URL @@ -429,6 +444,7 @@ Webhooks provide a way to notify external systems about clone lifecycle events. - `trigger` (list of strings, required) - specifies which clone events should trigger this webhook. Available trigger types: - `clone_create` - triggered when a new clone is created - `clone_reset` - triggered when an existing clone is reset to a different snapshot + - `clone_delete` - triggered when a clone is deleted. Supported since DBLab Engine 4.1. ### Webhook payload format Webhook requests are sent as HTTP POST with JSON payload containing: diff --git a/docs/reference-guides/dblab-client-cli-reference.md b/docs/reference-guides/dblab-client-cli-reference.md index f31d5ca8..5e9167f6 100644 --- a/docs/reference-guides/dblab-client-cli-reference.md +++ b/docs/reference-guides/dblab-client-cli-reference.md @@ -93,6 +93,7 @@ COMMANDS: clone create, update, delete, reset, or retrieve clone instance display instance info snapshot create, retrieve, or delete snapshot + teleport Teleport integration commands (DLE 4.1+) config configure CLI environments help, h shows a list of commands or help for one command ``` @@ -303,7 +304,7 @@ dblab clone create [command options] - `--id` (string, optional) - clone ID - `--snapshot-id` (string, optional; DLE 4.0+) - snapshot ID - `--branch` (string, optional; DLE 4.0+) - branch name -- `--protected` , `-p` (boolean, default: false) - mark instance as protected from deletion +- `--protected` , `-p` (string, optional) - enable deletion protection. Accepts: `true` for default lease duration, a number of minutes for custom duration, or `0` for infinite protection (no expiry). When omitted, clone is not protected. DLE 4.1+ supports time-limited protection leases — see [Protection leases](/docs/dblab-howtos/cloning/clone-protection). - `--async` , `-a` (boolean, default: false) - run the command asynchronously - `--extra-config` (string, optional) set an extra database configuration for the clone. An example: statement_timeout='1s' - `--help` , `-h` (boolean, default: false) - show help @@ -313,6 +314,11 @@ dblab clone create [command options] dblab clone create --username someuser --password SomePassword --branch main --id test ``` +Create a clone with protection for 8 hours (480 minutes): +```bash +dblab clone create --username someuser --password SomePassword --branch main --id test --protected 480 +``` + --- ### Subcommand `update` Update the specified clone. @@ -325,14 +331,21 @@ dblab clone update [command options] CLONE_ID - `CLONE_ID` (string, required) - an ID of the Database Lab clone to update parameters **Options** -- `--protected` , `-p` (boolean, optional) - mark instance as protected from deletion +- `--protected` , `-p` (string, optional) - enable deletion protection. Accepts: `true` for default lease duration, a number of minutes for custom duration, or `0` for infinite protection (no expiry). DLE 4.1+ supports time-limited protection leases — see [Protection leases](/docs/dblab-howtos/cloning/clone-protection). - `--help` , `-h` (boolean, default: false) - show help **Example** + +Protect a clone with default lease duration: ```bash dblab clone update --protected TestCloneID ``` +Protect a clone for 24 hours (1440 minutes): +```bash +dblab clone update --protected 1440 TestCloneID +``` + --- ### Subcommand `reset` Reset the clone's state. @@ -631,15 +644,23 @@ Delete a snapshot. **Usage** ```bash -dblab snapshot delete SNAPSHOT_ID +dblab snapshot delete [command options] SNAPSHOT_ID ``` +**Options** +- `--force` (boolean, default: false) - force deletion even if dependent clones or datasets exist + **Example** ```bash dblab snapshot delete "dblab_pool/dataset_1@snapshot_20241028174127" ``` +Force delete a snapshot with dependent clones: +```bash +dblab snapshot delete --force "dblab_pool/dataset_1@snapshot_20241028174127" +``` + --- ### Subcommand `help` , `h` Show help for the command. @@ -650,6 +671,54 @@ dblab snapshot help ``` +## Command: `teleport` +:::note +Requires DBLab 4.1 or higher +::: +Teleport integration commands. The `teleport serve` subcommand runs a sidecar process that automatically registers and deregisters DBLab clones as Teleport database resources, enabling secure access to clones through Teleport's access control. + +For a full setup guide, see the [Teleport integration howto](/docs/dblab-howtos/administration/teleport-integration). + +**Usage** +```bash +dblab teleport command [command options] [arguments...] +``` + +**Subcommands** +- `serve` - start the Teleport sidecar + +--- +### Subcommand `serve` +Start the Teleport integration sidecar. This process listens for clone lifecycle webhooks from DBLab Engine and automatically registers/deregisters clones as Teleport database resources. + +**Usage** +```bash +dblab teleport serve [command options] +``` + +**Options** +- `--environment-id` (string, required) - environment identifier used in Teleport resource names +- `--teleport-proxy` (string, required) - Teleport Auth Server or Proxy address (e.g., `teleport.example.com:3025`) +- `--teleport-identity` (string, required) - path to the Teleport bot identity file for authentication +- `--listen-addr` (string, optional, default: "0.0.0.0:9876") - address and port to listen for incoming webhooks +- `--dblab-url` (string, required) - DBLab API URL (e.g., `http://localhost:2345`) +- `--dblab-token` (string, required) - DBLab verification token +- `--webhook-secret` (string, optional) - webhook secret for verifying incoming webhook payloads + +**Example** +```bash +dblab teleport serve \ + --environment-id production \ + --teleport-proxy teleport.example.com:3025 \ + --teleport-identity /etc/teleport/dblab-identity \ + --listen-addr 0.0.0.0:9876 \ + --dblab-url http://localhost:2345 \ + --dblab-token "$DBLAB_TOKEN" \ + --webhook-secret "$WEBHOOK_SECRET" +``` + +--- + ## Command: `config` Configure CLI environments. diff --git a/docs/tutorials/database-lab-tutorial.md b/docs/tutorials/database-lab-tutorial.md index c142ac0d..bb676ad9 100644 --- a/docs/tutorials/database-lab-tutorial.md +++ b/docs/tutorials/database-lab-tutorial.md @@ -144,7 +144,7 @@ ok: [root@5.161.212.233] => { "", "5) DBLab CLI:", " - CLI ('dblab') setup:", - " export DBLAB_CLI_VERSION=4.0.3", + " export DBLAB_CLI_VERSION=4.1.0", " curl -sSL dblab.sh | bash", " dblab init --environment-id=dblab-demo --token=edlhYHOgBPkr4ix1qP3YvQMytfK2JSxH --url=http://127.0.0.1:2346/api", " - CLI docs: https://cli-docs.dblab.dev/", diff --git a/sidebars.js b/sidebars.js index a9c25e2d..b085c16c 100644 --- a/sidebars.js +++ b/sidebars.js @@ -90,6 +90,7 @@ module.exports = { "database-lab/index", "database-lab/supported-databases", "database-lab/user-interface", + "database-lab/prometheus-monitoring", "database-lab/masking", "database-lab/db-migration-checker", "database-lab/telemetry", @@ -126,6 +127,7 @@ module.exports = { "dblab-howtos/administration/engine-manage", "dblab-howtos/administration/joe-manage", "dblab-howtos/administration/engine-secure", + "dblab-howtos/administration/teleport-integration", "dblab-howtos/administration/logical-full-refresh", "dblab-howtos/administration/ci-observer-postgres-log-masking", "dblab-howtos/administration/add-disk-space-to-zfs-pool", @@ -133,6 +135,7 @@ module.exports = { "Data sources": [ "dblab-howtos/administration/data/index", "dblab-howtos/administration/data/rds", + "dblab-howtos/administration/data/rds-refresh", "dblab-howtos/administration/data/dump", "dblab-howtos/administration/data/wal-g", "dblab-howtos/administration/data/pgbackrest", From 28e99efdd50bf404f1699ee7d413646718f1b779 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 10 Apr 2026 14:16:37 +0000 Subject: [PATCH 2/9] docs: add protection lease webhook trigger events Add clone_protection_expiring and clone_protection_expired webhook trigger types to the configuration reference. https://claude.ai/code/session_011sPDgBjzL2N2X6jiYyoTjQ --- .../database-lab-engine-configuration-reference.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference-guides/database-lab-engine-configuration-reference.md b/docs/reference-guides/database-lab-engine-configuration-reference.md index 406a02ba..38e45d0f 100644 --- a/docs/reference-guides/database-lab-engine-configuration-reference.md +++ b/docs/reference-guides/database-lab-engine-configuration-reference.md @@ -445,6 +445,8 @@ Webhooks provide a way to notify external systems about clone lifecycle events. - `clone_create` - triggered when a new clone is created - `clone_reset` - triggered when an existing clone is reset to a different snapshot - `clone_delete` - triggered when a clone is deleted. Supported since DBLab Engine 4.1. + - `clone_protection_expiring` - triggered when a clone's protection lease is about to expire (based on `protectionExpiryWarningMinutes`). Supported since DBLab Engine 4.1. + - `clone_protection_expired` - triggered when a clone's protection lease has expired and protection has been automatically removed. Supported since DBLab Engine 4.1. ### Webhook payload format Webhook requests are sent as HTTP POST with JSON payload containing: From 24afe24a0911a7559dda8a70a15c60027de7cb86 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 10 Apr 2026 18:30:53 +0000 Subject: [PATCH 3/9] docs: second-round fixes from cross-check analysis - Add 4 missing webhook trigger events: snapshot_create, snapshot_delete, branch_create, branch_delete - Fix stale PG 9.6 references in FAQ and Joe Bot overview - Add missing admin endpoints to API reference table (billing-status, activate, instance/logs WebSocket) - Fix ws-auth method from POST to GET in API reference - Add cross-link from RDS howto to RDS/Aurora refresh tool https://claude.ai/code/session_011sPDgBjzL2N2X6jiYyoTjQ --- docs/dblab-howtos/administration/data/rds.md | 4 ++++ docs/joe-bot/index.md | 2 +- docs/questions-and-answers.md | 2 +- docs/reference-guides/database-lab-engine-api-reference.md | 5 ++++- .../database-lab-engine-configuration-reference.md | 4 ++++ 5 files changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/dblab-howtos/administration/data/rds.md b/docs/dblab-howtos/administration/data/rds.md index 250b13f1..62555ca5 100644 --- a/docs/dblab-howtos/administration/data/rds.md +++ b/docs/dblab-howtos/administration/data/rds.md @@ -143,3 +143,7 @@ sudo rm -rf /var/lib/dblab/dblab_pool/data/* sudo umount /var/lib/dblab/dblab_pool/dump sudo rm -rf /var/lib/dblab/dblab_pool/dump ``` + +## Alternative: RDS/Aurora refresh tool + +For large production databases, running `pg_dump` directly against production can hold xmin horizon for hours and create significant load. The [RDS/Aurora refresh tool](/docs/dblab-howtos/administration/data/rds-refresh) provides an alternative approach that dumps from a temporary RDS clone instead, leaving production untouched. diff --git a/docs/joe-bot/index.md b/docs/joe-bot/index.md index 14ba4011..82d9cb5b 100644 --- a/docs/joe-bot/index.md +++ b/docs/joe-bot/index.md @@ -23,7 +23,7 @@ This process is fully automated and takes only a few seconds, even for multi-ter ## Features - "Serverless EXPLAIN": engineers do not need to worry about the provisioning of independent database clones. The process is fully automated, so all the work looks like requests to analyze some query execution plan or modify database schema – and Joe takes care of it, ensuring that delivered results are identical to production. -- PostgreSQL versions 9.6, 10, 11, and 12 are currently supported. +- PostgreSQL versions 10 through 18 are supported. - Currently, Joe is provided in the form of Slack chatbot. - The provisioning of a new clone takes only a few seconds, regardless of the database size. - Each database clone is fully independent, so developers do not interfere with each other and do not need to wait. diff --git a/docs/questions-and-answers.md b/docs/questions-and-answers.md index 5adeaf81..d44fc94e 100644 --- a/docs/questions-and-answers.md +++ b/docs/questions-and-answers.md @@ -98,7 +98,7 @@ Yes. Monitoring can run in PostgresAI Cloud or in your own infrastructure. We se ## What Postgres versions are supported? -postgres_ai monitoring supports Postgres 14+. DBLab Engine supports Postgres 9.6+. +postgres_ai monitoring supports Postgres 14+. DBLab Engine supports Postgres 10+. ## Does it work with managed Postgres? diff --git a/docs/reference-guides/database-lab-engine-api-reference.md b/docs/reference-guides/database-lab-engine-api-reference.md index b621fdb1..28a973aa 100644 --- a/docs/reference-guides/database-lab-engine-api-reference.md +++ b/docs/reference-guides/database-lab-engine-api-reference.md @@ -89,7 +89,10 @@ curl -H "Verification-Token: YOUR_TOKEN" http://localhost:2345/status | POST | `/admin/config` | Set config | | GET | `/admin/config.yaml` | Get full config (YAML) | | POST | `/admin/test-db-source` | Test source database connection | -| POST | `/admin/ws-auth` | WebSocket authentication | +| GET | `/admin/ws-auth` | WebSocket authentication token | +| GET | `/admin/billing-status` | Billing status | +| POST | `/admin/activate` | Activate billing | +| GET | `/instance/logs` | Instance logs (WebSocket) | ## New in DBLab Engine 4.1 diff --git a/docs/reference-guides/database-lab-engine-configuration-reference.md b/docs/reference-guides/database-lab-engine-configuration-reference.md index 38e45d0f..010328bf 100644 --- a/docs/reference-guides/database-lab-engine-configuration-reference.md +++ b/docs/reference-guides/database-lab-engine-configuration-reference.md @@ -447,6 +447,10 @@ Webhooks provide a way to notify external systems about clone lifecycle events. - `clone_delete` - triggered when a clone is deleted. Supported since DBLab Engine 4.1. - `clone_protection_expiring` - triggered when a clone's protection lease is about to expire (based on `protectionExpiryWarningMinutes`). Supported since DBLab Engine 4.1. - `clone_protection_expired` - triggered when a clone's protection lease has expired and protection has been automatically removed. Supported since DBLab Engine 4.1. + - `snapshot_create` - triggered when a new snapshot is created. Supported since DBLab Engine 4.1. + - `snapshot_delete` - triggered when a snapshot is deleted. Supported since DBLab Engine 4.1. + - `branch_create` - triggered when a new branch is created. Supported since DBLab Engine 4.1. + - `branch_delete` - triggered when a branch is deleted. Supported since DBLab Engine 4.1. ### Webhook payload format Webhook requests are sent as HTTP POST with JSON payload containing: From 9d440c1d0e2f79c1d3d1e199db1fd1b864b4a402 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 10 Apr 2026 18:37:02 +0000 Subject: [PATCH 4/9] docs: round-3 howto improvements - create-clone: add --branch, protection lease duration, --extra-config examples - destroy-clone: add --async example, update protection note for leases - delete-snapshot: add --force CLI example - logical-full-refresh: add CLI/API trigger method (DLE 4.0+) - engine-secure: add cross-link to Teleport integration - teleport-integration: add missing sections (user role, volume mounting, Docker networking), fix prerequisite numbering - rds-refresh: add 12 missing config fields (parameterGroup, optionGroup, storageType, IAM auth, etc.) https://claude.ai/code/session_011sPDgBjzL2N2X6jiYyoTjQ --- .../administration/data/rds-refresh.md | 12 +++++ .../administration/engine-secure.md | 4 ++ .../administration/logical-full-refresh.md | 24 +++++++++- .../administration/teleport-integration.md | 44 ++++++++++++++++++- docs/dblab-howtos/cloning/create-clone.md | 28 +++++++++++- docs/dblab-howtos/cloning/destroy-clone.md | 8 +++- .../dblab-howtos/snapshots/delete-snapshot.md | 7 +++ 7 files changed, 121 insertions(+), 6 deletions(-) diff --git a/docs/dblab-howtos/administration/data/rds-refresh.md b/docs/dblab-howtos/administration/data/rds-refresh.md index e882228a..8c81deaf 100644 --- a/docs/dblab-howtos/administration/data/rds-refresh.md +++ b/docs/dblab-howtos/administration/data/rds-refresh.md @@ -76,12 +76,24 @@ docker run --rm \ | `source.dbName` | Yes | Database name | | `source.username` | Yes | Database user | | `source.password` | Yes | Password (supports `${ENV_VAR}` syntax) | +| `source.snapshotIdentifier` | No | Specific snapshot ID to use; if empty, uses latest automated snapshot | | `clone.instanceClass` | Yes | RDS clone instance type (e.g., `db.t3.medium`) | | `clone.securityGroups` | No | Security groups allowing DBLab access | | `clone.subnetGroup` | No | DB subnet group | +| `clone.parameterGroup` | No | RDS parameter group name | +| `clone.optionGroup` | No | RDS option group name (RDS instances only) | +| `clone.clusterParameterGroup` | No | Cluster parameter group (Aurora only) | +| `clone.publiclyAccessible` | No | Make clone publicly accessible (default: `false`) | +| `clone.enableIAMAuth` | No | Enable IAM database authentication (default: `false`) | +| `clone.storageType` | No | Storage type: `gp2`, `gp3`, `io1`, `io2` | +| `clone.deletionProtection` | No | Enable deletion protection on clone (default: `false`) | +| `clone.port` | No | Custom port for the clone (default: RDS default) | +| `clone.tags` | No | Additional tags (key-value map) for the RDS clone | | `clone.maxAge` | No | Max age before clone is considered stale (default: `48h`) | | `dblab.apiEndpoint` | Yes | DBLab API URL | | `dblab.token` | Yes | DBLab verification token | +| `dblab.insecure` | No | Skip TLS certificate verification (default: `false`) | +| `dblab.pollInterval` | No | Status polling interval (default: `30s`) | | `dblab.timeout` | No | Max refresh wait (default: `4h`) | | `aws.region` | Yes | AWS region | diff --git a/docs/dblab-howtos/administration/engine-secure.md b/docs/dblab-howtos/administration/engine-secure.md index 42e3bd15..a0d66804 100644 --- a/docs/dblab-howtos/administration/engine-secure.md +++ b/docs/dblab-howtos/administration/engine-secure.md @@ -49,3 +49,7 @@ PGPASSWORD=secret_password psql \ ``` Adjust the port numbers accordingly for other clones, following the pattern `original_port+3000` (e.g., `6001->9001`, `6002->9002`, etc.). + +## Alternative: Teleport integration + +For zero-trust access control with audit logging, certificate-based authentication, and role-based access, consider using [Teleport integration](/docs/dblab-howtos/administration/teleport-integration) (DBLab Engine 4.1+). diff --git a/docs/dblab-howtos/administration/logical-full-refresh.md b/docs/dblab-howtos/administration/logical-full-refresh.md index a2d874a1..e5e00b63 100644 --- a/docs/dblab-howtos/administration/logical-full-refresh.md +++ b/docs/dblab-howtos/administration/logical-full-refresh.md @@ -11,7 +11,29 @@ Note, that the process described here requires a maintenance window (brief perio If you are using the "physical" provisioning mode, read [how to configure the "sync" instance](/docs/dblab-howtos/administration/postgresql-configuration#the-sync-instance) instead. -## Refresh data from source +## Triggering a full refresh via CLI or API (DBLab Engine 4.0+) + +If you have multiple pools (disks) configured, you can trigger a full refresh without downtime using the CLI or API. DBLab Engine will refresh data on an inactive pool while clones continue to run on the active pool. + +**CLI:** +```bash +dblab instance full-refresh +``` + +**API:** +```bash +curl -X POST -H "Verification-Token: YOUR_TOKEN" http://localhost:2345/full-refresh +``` + +:::tip +A scheduled full refresh can also be configured using the `retrieval.refresh.timetable` option in `server.yml` (crontab format). +::: + +## Manual refresh (single disk) + +The process described below requires a maintenance window and deletes existing clones. + +### Refresh data from source ### 1. Cleanup Stop and remove the existing containers, then clean up the data directory and destroy the pool: ```bash diff --git a/docs/dblab-howtos/administration/teleport-integration.md b/docs/dblab-howtos/administration/teleport-integration.md index 715680e8..126f8125 100644 --- a/docs/dblab-howtos/administration/teleport-integration.md +++ b/docs/dblab-howtos/administration/teleport-integration.md @@ -90,7 +90,24 @@ db_service: dblab: "true" ``` -### 4. SSL/TLS for Postgres clones +### 4. User role for database access + +Teleport users who need to connect to DBLab clones need a role granting database access: + +```yaml +kind: role +version: v7 +metadata: + name: dblab-user +spec: + allow: + db_labels: + dblab: "true" + db_names: ['*'] + db_users: ['*'] +``` + +### 5. SSL/TLS for Postgres clones Teleport always initiates TLS to backend databases. DBLab clones must have SSL enabled. @@ -111,7 +128,7 @@ tctl auth export --type=db-client > /etc/dblab/certs/teleport-ca.crt chown 999:999 /etc/dblab/certs/teleport-ca.crt ``` -### 5. pg_hba.conf — certificate authentication +### 6. pg_hba.conf — certificate authentication Starting with DBLab Engine 4.1, the default `pg_hba.conf` includes a `hostssl ... cert` rule that enables Teleport certificate authentication out of the box: @@ -123,6 +140,29 @@ host all all 0.0.0.0/0 md5 No custom `pg_hba.conf` or volume mount is required for Teleport. +### 7. Volume mounting for certs + +Clone containers only inherit DBLab Engine container volumes whose source is under `poolManager.mountDir`. For SSL certs stored outside the pool, use `containerConfig`: + +```yaml +databaseContainer: &db_container + dockerImage: "postgresai/extended-postgres:16" + containerConfig: + "shm-size": 1gb + volume: "/etc/dblab/certs:/var/lib/postgresql/cert:ro" +``` + +Cert files on the host must have uid 999 ownership before DBLab Engine starts, because the postgres user inside the container runs as uid 999. + +### 8. Webhook URL — Docker networking + +DBLab Engine runs inside Docker, so `localhost:9876` from within the Engine container resolves to the container itself, not the host. + +Options: +- Use `host.docker.internal:9876` (Docker Desktop / Docker 20.10+) +- Use the Docker bridge IP (typically `172.17.0.1:9876`) +- Run the sidecar in the same Docker network as DBLab Engine + ## Configuration ### DBLab Engine server.yml diff --git a/docs/dblab-howtos/cloning/create-clone.md b/docs/dblab-howtos/cloning/create-clone.md index de6c9dc1..53522ee1 100644 --- a/docs/dblab-howtos/cloning/create-clone.md +++ b/docs/dblab-howtos/cloning/create-clone.md @@ -94,16 +94,34 @@ $ dblab clone create --username USERNAME --password PASSWORD --id CLONE_ID --sna } ``` +### Create a clone from a branch +:::note +Requires DBLab 4.0 or higher +::: + +Create a clone from a specific branch: +```bash +$ dblab clone create --username USERNAME --password PASSWORD --id CLONE_ID --branch main +``` + ### Protected status -You can make clone protected during the creation or later (if needed). Please be careful: abandoned protected clones may cause out-of-disk-space events. Read the details [here](/docs/dblab-howtos/cloning/clone-protection). +You can make a clone protected during creation or later. Please be careful: abandoned protected clones may cause out-of-disk-space events. Read the details [here](/docs/dblab-howtos/cloning/clone-protection). + +Protect with default lease duration: ```bash -$ dblab clone create --username USERNAME --password PASSWORD --id CLONE_ID --protected +$ dblab clone create --username USERNAME --password PASSWORD --id CLONE_ID --protected true +``` + +Protect for a specific duration (e.g., 8 hours = 480 minutes): +```bash +$ dblab clone create --username USERNAME --password PASSWORD --id CLONE_ID --protected 480 ``` ```json { "id": "democlone", "protected": true, + "protectedTill": "2026-04-11T06:00:00Z", "status": { "code": "OK", "message": "Clone is ready to accept Postgres connections." @@ -112,6 +130,12 @@ $ dblab clone create --username USERNAME --password PASSWORD --id CLONE_ID --pro } ``` +### Extra PostgreSQL configuration +You can set additional PostgreSQL configuration parameters for a clone: +```bash +$ dblab clone create --username USERNAME --password PASSWORD --id CLONE_ID --extra-config statement_timeout='30s' +``` + ## Related - Guide: [Connect to a clone](/docs/dblab-howtos/cloning/connect-clone) - Guide: [Destroy a clone](/docs/dblab-howtos/cloning/destroy-clone) diff --git a/docs/dblab-howtos/cloning/destroy-clone.md b/docs/dblab-howtos/cloning/destroy-clone.md index 72daaa8f..90842a94 100644 --- a/docs/dblab-howtos/cloning/destroy-clone.md +++ b/docs/dblab-howtos/cloning/destroy-clone.md @@ -8,7 +8,7 @@ DBLab Engine automatically deletes idle unprotected clones after the idle interv ::: :::info -The protected clone could not be deleted automatically or manually. In order to delete the clone, you would need to [unprotect it](/docs/dblab-howtos/cloning/clone-protection). +A protected clone cannot be deleted automatically or manually. To delete it, first [remove protection](/docs/dblab-howtos/cloning/clone-protection). With protection leases (DBLab Engine 4.1+), protection expires automatically after the configured duration. ::: ## GUI @@ -31,6 +31,12 @@ dblab clone destroy CLONE_ID The clone has been successfully destroyed: CLONE_ID ``` +### Destroy a clone asynchronously +For long-running operations, use the `--async` flag: +```bash +dblab clone destroy --async CLONE_ID +``` + ## Related - Guide: [Clone protection from manual and automatic deletion](/docs/dblab-howtos/cloning/clone-protection) - Guide: [Resetting a clone state](/docs/dblab-howtos/cloning/reset-clone) diff --git a/docs/dblab-howtos/snapshots/delete-snapshot.md b/docs/dblab-howtos/snapshots/delete-snapshot.md index 09f55da8..563bea72 100644 --- a/docs/dblab-howtos/snapshots/delete-snapshot.md +++ b/docs/dblab-howtos/snapshots/delete-snapshot.md @@ -33,5 +33,12 @@ Delete a snapshot with `dblab snapshot` command, using subcommand `delete`. $ dblab snapshot delete SNAPSHOT_ID ``` +### Force-delete a snapshot with dependent clones +If the snapshot has dependent clones or datasets, use `--force`: + +```bash +$ dblab snapshot delete --force SNAPSHOT_ID +``` + ## Related - Guide: [Create a snapshot](/docs/dblab-howtos/snapshots/create-snapshot) From 7409b7375c3c202fa38acaa133e5f42c3a727c29 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 10 Apr 2026 18:38:19 +0000 Subject: [PATCH 5/9] docs: fix CLI reference accuracy issues from cross-check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Fix teleport serve: --listen-addr default to "localhost:9876" (not "0.0.0.0"), --dblab-url to optional with default, --webhook-secret to required (matches code) - Remove incorrect --force flag from snapshot delete CLI docs — force deletion is only available via API/UI, not CLI - Fix init --token from required to optional (code doesn't enforce it) https://claude.ai/code/session_011sPDgBjzL2N2X6jiYyoTjQ --- .../dblab-howtos/snapshots/delete-snapshot.md | 9 +++----- .../dblab-client-cli-reference.md | 22 ++++++++----------- 2 files changed, 12 insertions(+), 19 deletions(-) diff --git a/docs/dblab-howtos/snapshots/delete-snapshot.md b/docs/dblab-howtos/snapshots/delete-snapshot.md index 563bea72..2b674c5b 100644 --- a/docs/dblab-howtos/snapshots/delete-snapshot.md +++ b/docs/dblab-howtos/snapshots/delete-snapshot.md @@ -33,12 +33,9 @@ Delete a snapshot with `dblab snapshot` command, using subcommand `delete`. $ dblab snapshot delete SNAPSHOT_ID ``` -### Force-delete a snapshot with dependent clones -If the snapshot has dependent clones or datasets, use `--force`: - -```bash -$ dblab snapshot delete --force SNAPSHOT_ID -``` +:::tip +If the snapshot has dependent clones, you can force-delete it via the GUI (see screenshots above) or via the API (`DELETE /snapshot/{id}?force=true`). +::: ## Related - Guide: [Create a snapshot](/docs/dblab-howtos/snapshots/create-snapshot) diff --git a/docs/reference-guides/dblab-client-cli-reference.md b/docs/reference-guides/dblab-client-cli-reference.md index 5e9167f6..749a1836 100644 --- a/docs/reference-guides/dblab-client-cli-reference.md +++ b/docs/reference-guides/dblab-client-cli-reference.md @@ -111,12 +111,12 @@ dblab init [command options] [arguments...] **Options** - `--environment-id` (string, required) - an arbitrary environment ID of Database Lab instance's API - `--url` (string, required) - URL of Database Lab instance's API - - `--token` (string, required) - verification token of Database Lab instance + - `--token` (string, optional) - verification token of Database Lab instance - `--insecure` (boolean, optional, default: false) - allow insecure server connections when using SSL - `--request-timeout` (string, optional, default: "") - change requests timeout - `--forwarding-server-url` (string, optional) - forwarding server URL of Database Lab instance. For example: `ssh://user@remote.host:22` - `--forwarding-local-port` (string, optional) - local port for forwarding to the Database Lab instance - - `--identity-file` (string, optional) - select a file from which the identity (private key) for public key authentication is read" + - `--identity-file` (string, optional) - select a file from which the identity (private key) for public key authentication is read **Example** ```bash @@ -644,22 +644,18 @@ Delete a snapshot. **Usage** ```bash -dblab snapshot delete [command options] SNAPSHOT_ID +dblab snapshot delete SNAPSHOT_ID ``` -**Options** -- `--force` (boolean, default: false) - force deletion even if dependent clones or datasets exist - **Example** ```bash dblab snapshot delete "dblab_pool/dataset_1@snapshot_20241028174127" ``` -Force delete a snapshot with dependent clones: -```bash -dblab snapshot delete --force "dblab_pool/dataset_1@snapshot_20241028174127" -``` +:::tip +Force deletion of snapshots with dependent clones is available through the API (`DELETE /snapshot/{id}?force=true`) or the UI, but is not currently supported via the CLI. +::: --- ### Subcommand `help` , `h` @@ -700,10 +696,10 @@ dblab teleport serve [command options] - `--environment-id` (string, required) - environment identifier used in Teleport resource names - `--teleport-proxy` (string, required) - Teleport Auth Server or Proxy address (e.g., `teleport.example.com:3025`) - `--teleport-identity` (string, required) - path to the Teleport bot identity file for authentication -- `--listen-addr` (string, optional, default: "0.0.0.0:9876") - address and port to listen for incoming webhooks -- `--dblab-url` (string, required) - DBLab API URL (e.g., `http://localhost:2345`) +- `--listen-addr` (string, optional, default: "localhost:9876") - address and port to listen for incoming webhooks. Use `0.0.0.0:9876` if the sidecar needs to be reachable from Docker containers. +- `--dblab-url` (string, optional, default: "http://localhost:2345") - DBLab API URL - `--dblab-token` (string, required) - DBLab verification token -- `--webhook-secret` (string, optional) - webhook secret for verifying incoming webhook payloads +- `--webhook-secret` (string, required) - shared secret that DBLab Engine sends in the `DBLab-Webhook-Token` header for webhook payload verification **Example** ```bash From bb6ffff078145c35d0f3b259802919572009c394 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 10 Apr 2026 18:39:28 +0000 Subject: [PATCH 6/9] docs: fix broken links and add cross-references - Fix broken link /docs/dblab-roadmap -> /docs/roadmap in all-features - Fix broken link /docs/database-lab/cli-reference -> correct path in platform/security - Add rds-refresh cross-link to data sources index page https://claude.ai/code/session_011sPDgBjzL2N2X6jiYyoTjQ --- docs/all-features.md | 2 +- docs/dblab-howtos/administration/data/index.md | 1 + docs/platform/security.md | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/all-features.md b/docs/all-features.md index ab6086d4..cf674422 100644 --- a/docs/all-features.md +++ b/docs/all-features.md @@ -72,4 +72,4 @@ keywords: |Standard support|❌|✅|✅| |Premium support (24/7, 1 hour), trainings|❌|✅|✅| -See also: [Development roadmap](/docs/dblab-roadmap). +See also: [Development roadmap](/docs/roadmap). diff --git a/docs/dblab-howtos/administration/data/index.md b/docs/dblab-howtos/administration/data/index.md index 1cdb8f79..3a757296 100644 --- a/docs/dblab-howtos/administration/data/index.md +++ b/docs/dblab-howtos/administration/data/index.md @@ -8,6 +8,7 @@ slug: /dblab-howtos/administration/data ### Logical - [Dump](/docs/dblab-howtos/administration/data/dump) - [RDS](/docs/dblab-howtos/administration/data/rds) +- [RDS/Aurora refresh](/docs/dblab-howtos/administration/data/rds-refresh) — refreshes from temporary RDS clone instead of production - [Full refresh](/docs/dblab-howtos/administration/logical-full-refresh) ### Physical diff --git a/docs/platform/security.md b/docs/platform/security.md index cb7d0b10..5dd2092a 100644 --- a/docs/platform/security.md +++ b/docs/platform/security.md @@ -60,7 +60,7 @@ Please contact Postgres.ai support to obtain your registration key and detailed Further, we discuss all Postgres.ai components that are to be installed in your infrastructure and what kind of information can be transferred to Postgres.ai. #### Database Lab -When integrated, the Database Lab component may receive only control signals such as "create clone", "destroy clone", "refresh clone", "list snapshots". The full list of capabilities you may find in [Database Lab CLI Reference](/docs/database-lab/cli-reference). By no means is data from your databases available to the Platform. +When integrated, the Database Lab component may receive only control signals such as "create clone", "destroy clone", "refresh clone", "list snapshots". The full list of capabilities you may find in [Database Lab CLI Reference](/docs/reference-guides/dblab-client-cli-reference). By no means is data from your databases available to the Platform. To be able to connect to a clone, users need to work inside your infrastructure, where connections to Database Lab clones (by default, ports 6000..6100) are possible, using the username and the password defined at clone creation time. Postgres.ai never stores passwords for clones, and it is the users' responsibility to remember them. From ef92ee03eff78b49b9a293260dec5e57ec9c34e4 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 10 Apr 2026 18:40:46 +0000 Subject: [PATCH 7/9] docs: fix default Postgres version from 17 to 18 Example configs in v4.1.0 use postgresai/extended-postgres:18-0.6.2 as the default Docker image. https://claude.ai/code/session_011sPDgBjzL2N2X6jiYyoTjQ --- docs/database-lab/supported-databases.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/database-lab/supported-databases.md b/docs/database-lab/supported-databases.md index 4cb63d78..fd73331a 100644 --- a/docs/database-lab/supported-databases.md +++ b/docs/database-lab/supported-databases.md @@ -14,7 +14,7 @@ Currently, DBLab Engine fully supports the following [PostgreSQL major versions] - 17 (released: 2024-09-26; EOL: 2029-11-08) - 18 (released: 2025-09-25; EOL: 2030-11-13) -By default, version 17 is used: `postgresai/extended-postgres:17`. +By default, version 18 is used in the example configurations: `postgresai/extended-postgres:18`. The images are published in [Docker Hub](https://hub.docker.com/r/postgresai/extended-postgres). From 833336a4d160e62b6e4dd4ec7144868bd83aa006 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 10 Apr 2026 18:41:14 +0000 Subject: [PATCH 8/9] docs: fix clone howto issues from deep review - destroy-clone: remove incorrect create-clone screenshot reference - reset-clone: add --async flag example https://claude.ai/code/session_011sPDgBjzL2N2X6jiYyoTjQ --- docs/dblab-howtos/cloning/destroy-clone.md | 1 - docs/dblab-howtos/cloning/reset-clone.md | 5 +++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/dblab-howtos/cloning/destroy-clone.md b/docs/dblab-howtos/cloning/destroy-clone.md index 90842a94..eca151d2 100644 --- a/docs/dblab-howtos/cloning/destroy-clone.md +++ b/docs/dblab-howtos/cloning/destroy-clone.md @@ -13,7 +13,6 @@ A protected clone cannot be deleted automatically or manually. To delete it, fir ## GUI 1. On the **Database Lab clone** page click the **Destroy** button. - ![DBLab Engine page / Create clone](/assets/guides/create-clone-1.png) 1. Accept confirmation dialog and wait for it. You will be redirected to the **Database Lab instance** page. ## CLI diff --git a/docs/dblab-howtos/cloning/reset-clone.md b/docs/dblab-howtos/cloning/reset-clone.md index c8e0ce60..b62759b3 100644 --- a/docs/dblab-howtos/cloning/reset-clone.md +++ b/docs/dblab-howtos/cloning/reset-clone.md @@ -44,5 +44,10 @@ dblab clone reset --snapshot-id SNAPSHOT_ID CLONE_ID The parameters `--latest` and `--snapshot-id` must not be specified at the same time. ::: +For long-running operations, use the `--async` flag: +```bash +dblab clone reset --async --latest CLONE_ID +``` + ## Related - Guide: [Destroy a clone](/docs/dblab-howtos/cloning/destroy-clone) From 808deda6715e3ed1cc193e51edac1c6c3694b443 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 10 Apr 2026 18:42:25 +0000 Subject: [PATCH 9/9] docs: add missing alerts, snapshot methods, and cross-links - prometheus-monitoring: add 2 missing alert definitions (DBLabMetricsStale, DBLabSyncDown) - create-snapshot: add pool-based snapshot creation method alongside existing clone-based method - branching/index: add link to Coolify preview environments guide https://claude.ai/code/session_011sPDgBjzL2N2X6jiYyoTjQ --- docs/database-lab/prometheus-monitoring.md | 26 +++++++++++++++++++ docs/dblab-howtos/branching/index.md | 3 ++- .../dblab-howtos/snapshots/create-snapshot.md | 15 +++++++++-- 3 files changed, 41 insertions(+), 3 deletions(-) diff --git a/docs/database-lab/prometheus-monitoring.md b/docs/database-lab/prometheus-monitoring.md index 5561f036..2c3388a6 100644 --- a/docs/database-lab/prometheus-monitoring.md +++ b/docs/database-lab/prometheus-monitoring.md @@ -202,6 +202,32 @@ dblab_sync_wal_lag_seconds description: "DBLab sync instance WAL replay is {{ $value | humanizeDuration }} behind" ``` +### Metrics collection stale alert + +```yaml +- alert: DBLabMetricsStale + expr: time() - dblab_scrape_success_timestamp > 300 + for: 5m + labels: + severity: warning + annotations: + summary: "DBLab metrics collection is stale" + description: "DBLab metrics have not been updated for more than 5 minutes" +``` + +### Sync instance down alert (physical mode) + +```yaml +- alert: DBLabSyncDown + expr: dblab_sync_status{status="down"} == 1 or dblab_sync_status{status="error"} == 1 + for: 5m + labels: + severity: critical + annotations: + summary: "DBLab sync instance is down" + description: "DBLab sync instance is not healthy" +``` + ## OpenTelemetry integration DBLab metrics can be exported to OpenTelemetry-compatible backends using the OpenTelemetry Collector. This allows you to send metrics to Grafana Cloud, Datadog, New Relic, and other observability platforms. diff --git a/docs/dblab-howtos/branching/index.md b/docs/dblab-howtos/branching/index.md index a65be5a3..d0bdb9a8 100644 --- a/docs/dblab-howtos/branching/index.md +++ b/docs/dblab-howtos/branching/index.md @@ -7,4 +7,5 @@ slug: /dblab-howtos/branching ## Guides - [How to create a database branch](/docs/dblab-howtos/branching/create-branch) -- [How to delete a database branch](/docs/dblab-howtos/branching/delete-branch) \ No newline at end of file +- [How to delete a database branch](/docs/dblab-howtos/branching/delete-branch) +- [Preview environments with DBLab and Coolify](/docs/dblab-howtos/branching/preview-environments-with-dblab-and-coolify) \ No newline at end of file diff --git a/docs/dblab-howtos/snapshots/create-snapshot.md b/docs/dblab-howtos/snapshots/create-snapshot.md index eab2a86f..ea42e63b 100644 --- a/docs/dblab-howtos/snapshots/create-snapshot.md +++ b/docs/dblab-howtos/snapshots/create-snapshot.md @@ -27,12 +27,23 @@ Before you run any commands, install Database Lab CLI and initialize configurati - Command [`dblab snapshot`](/docs/reference-guides/dblab-client-cli-reference#command-snapshot) - Command [`dblab commit`](/docs/reference-guides/dblab-client-cli-reference#command-commit) -### Create a snapshot -Create a snapshot using the `dblab commit` command and specify the clone ID and message (optional): +### Create a snapshot from a clone +Create a snapshot from an existing clone using the `dblab commit` command: ```bash $ dblab commit --clone-id CLONE_ID --message "Snapshot message" ``` +### Create a snapshot from the current pool state +You can also create a snapshot directly from the current state of a pool (without an existing clone): +```bash +$ dblab snapshot create +``` + +To specify a particular pool: +```bash +$ dblab snapshot create --pool POOL_NAME +``` + Command `dblab snapshot list` shows all snapshots, including the one just created: ```bash $ dblab snapshot list