diff --git a/app/components/IntegrationsList.vue b/app/components/IntegrationsList.vue index 92286701..c39951ce 100644 --- a/app/components/IntegrationsList.vue +++ b/app/components/IntegrationsList.vue @@ -36,6 +36,12 @@ const integrations = [ to: '/guides/integrations/framer', logo: '/docs/img/framer.png', }, + { + title: 'Airbyte', + description: 'Sync data from Directus collections into a warehouse or database using the Directus Airbyte source connector.', + to: '/guides/integrations/airbyte', + logo: '/docs/img/airbyte.svg', + }, ]; diff --git a/content/guides/12.integrations/6.airbyte/0.index.md b/content/guides/12.integrations/6.airbyte/0.index.md new file mode 100644 index 00000000..c1e1f4d3 --- /dev/null +++ b/content/guides/12.integrations/6.airbyte/0.index.md @@ -0,0 +1,141 @@ +--- +id: airbyte-integration +title: Integration +description: Sync data from Directus collections into a warehouse or database using the Directus Airbyte source connector. +technologies: + - airbyte +--- + +Copy data from your Directus instance into a warehouse or database (BigQuery, Snowflake, Postgres, and more) using the Directus Airbyte source connector. + +::callout{icon="heroicons-outline:rocket-launch"} +**Quick Start** + +1. **Import the connector**: Add `Airbyte.yaml` as a new source in Airbyte +2. **Configure the source**: Enter your API key, base URL, and collection name +3. **Test the connection**: Airbyte hits the `/files` endpoint to verify credentials +4. **Create a connection**: Pick your destination and enable only the streams you need +:: + +## Before You Start + +You'll need: + +- A Directus project with the collections you want to sync +- An API token with read access to those collections (go to **Settings** > **Access Tokens**) +- An Airbyte account and access to add a new source connector + +## Available Streams + +The connector exposes five streams. Enable only the ones your pipeline needs. + +| Stream | Endpoint | Typical use | +|--------|----------|-------------| +| `collection_items` | `/items/{collection}` | Posts, products, pages, custom collections | +| `files` | `/files` | File library assets | +| `users` | `/users` | User directory (contains PII - only enable if needed) | +| `activity` | `/activity` | Audit log | +| `collections_meta` | `/collections` | Collection schema metadata | + +Connection tests use the `files` stream. + +## Setup in Airbyte + +1. Import the connector manifest (`Airbyte.yaml`) as a new source in Airbyte. +2. Enter the required settings (see below). +3. Click **Test connection** to verify your credentials. +4. Create a **connection**, choose a destination, and enable only the streams you need. +5. Set the sync mode per stream. **Full refresh** is typical for CMS snapshots. Airbyte incremental sync is separate from Directus filters. + +## Required Settings + +| Field | Example | Notes | +|-------|---------|-------| +| **API Key** | `your-token` | **Settings** > **Access Tokens**. Needs read access on every stream you enable. | +| **Base URL** | `https://your-project.directus.app` | No trailing slash. | +| **Collection Name** | `posts` | Required even if you only sync `files`. Use any collection slug your token can read. | + +**Collection Name** feeds the `collection_items` stream. If you don't want to sync collection items, disable that stream on the Airbyte connection after setup. + +## Optional Query Controls + +These map directly to [Directus query parameters](/guides/connect/query-parameters) and apply to every stream. Use them to filter, sort, and limit what gets synced. + +| Field | Query param | Example | +|-------|-------------|---------| +| **Fields** | `fields` | `id,title,status` (or `*` for all fields, the default) | +| **Filter** | `filter` | `{"status":{"_eq":"published"}}` | +| **Sort** | `sort` | `-date_created` or `title,-date_updated` | +| **Search** | `search` | Free-text search string | + +### Filter Examples + +Published items only: + +```json +{"status":{"_eq":"published"}} +``` + +Activity creates only: + +```json +{"action":{"_eq":"create"}} +``` + +Items updated after a specific date: + +```json +{"date_updated":{"_gte":"2024-01-01T00:00:00.000Z"}} +``` + +Enter filter values as a JSON string in the source config. + +### Fields Example + +``` +id,title,status,date_updated +``` + +Relational fields use dot notation: `id,title,author.name` + +## Start Date + +When set, the connector adds a `filter[field][_gte]` parameter to limit results to data after the given date. + +| Stream | Field filtered | +|--------|----------------| +| `collection_items` | `date_updated` | +| `files` | `uploaded_on` | +| `activity` | `timestamp` | + +Use `YYYY-MM-DD` format (e.g. `2026-06-01`) or a full ISO datetime. The connector normalizes date-only values to UTC midnight. + +**Start Date** does not apply to `users` or `collections_meta`. Use **Filter** for those. + +::callout{icon="material-symbols:info-outline"} +The Directus admin UI may display a different date field for files than what the API filters on. The API uses `uploaded_on`. To filter on a different field, use the **Filter** option instead: `{"modified_on":{"_gte":"2026-06-01T00:00:00.000Z"}}`. +:: + +## Permissions + +Each stream requires read access on the corresponding Directus resource. + +| Stream | Required permission | +|--------|---------------------| +| `collection_items` | Read on your collection | +| `files` | Read files | +| `users` | Read users | +| `activity` | Read activity (often requires admin) | +| `collections_meta` | Read collections | + +If a stream returns a 403 error, fix the token's role permissions or disable that stream on the connection. + +## Syncing Multiple Collections + +The connector syncs one collection per source. To sync multiple collections, create one Airbyte source per collection with the same base URL and API key, but a different **Collection Name**. + +## Learn More + +- [Directus API reference](/api) +- [Directus filter rules](/guides/connect/filter-rules) +- [Airbyte docs](https://docs.airbyte.com/) diff --git a/public/img/airbyte.svg b/public/img/airbyte.svg new file mode 100644 index 00000000..0e6cc083 --- /dev/null +++ b/public/img/airbyte.svg @@ -0,0 +1,44 @@ + + + + + + + + + + + + + + + + + \ No newline at end of file