feat(connectors): add SurrealDB sink connector

[countradooku]

Summary

Adds a SurrealDB sink connector for writing Iggy messages into SurrealDB using the latest SurrealDB Rust SDK version available during implementation, 3.1.4.

The connector supports deterministic record IDs, bulk INSERT IGNORE writes for idempotent replay, configurable batch sizing, root/namespace/database/no-auth modes, optional table and offset-index definition, payload modes (auto, json, text, base64), metadata/header/checksum/origin timestamp fields, retry/backoff handling, and runtime metrics logging.

This also wires the connector into workspace membership, connector docs, example runtime config, binary artifact builds, edge-release output, version bump scripts, and Docker-backed integration test scaffolding.

Tests

• cargo fmt --all
• cargo sort --no-format --workspace
• cargo clippy --all-features --all-targets -- -D warnings
• cargo check --all --all-features
• cargo test -p iggy_connector_surrealdb_sink
• cargo test -p integration --no-run connectors::surrealdb
• cargo test --locked --doc
• cargo doc --no-deps --all-features --quiet
• ./scripts/ci/taplo.sh
• ./scripts/ci/license-headers.sh
• ./scripts/ci/shellcheck.sh
• ./scripts/ci/binary-artifacts.sh --check
• ./scripts/extract-version.sh --check
• git diff --check
• prek install

Known Review Items

• ./scripts/ci/third-party-licenses.sh --validate --manifest core/connectors/sinks/surrealdb_sink/Cargo.toml reports BUSL-1.1 license failures from SurrealDB SDK crates. This PR intentionally keeps the SDK dependency because the connector targets the latest SurrealDB SDK.

[codecov]

Codecov Report

❌ Patch coverage is 85.49346% with 122 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.02%. Comparing base (52b060d) to head (f5059a6).

Files with missing lines	Patch %	Lines
core/connectors/sinks/surrealdb_sink/src/lib.rs	85.38%	100 Missing and 22 partials ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #3453       +/-   ##
=============================================
- Coverage     74.26%   50.02%   -24.24%     
  Complexity      937      937               
=============================================
  Files          1259     1258        -1     
  Lines        125969   110755    -15214     
  Branches     101643    86474    -15169     
=============================================
- Hits          93551    55410    -38141     
- Misses        29403    52546    +23143     
+ Partials       3015     2799      -216     

Components	Coverage Δ
Rust Core	43.93% <85.49%> (-31.22%)	⬇️
Java SDK	58.57% <ø> (ø)
C# SDK	71.40% <ø> (-0.73%)	⬇️
Python SDK	88.88% <ø> (ø)
PHP SDK	84.29% <ø> (ø)
Node SDK	91.22% <ø> (ø)
Go SDK	40.36% <ø> (ø)

Files with missing lines	Coverage Δ
core/server/src/http/jwt/jwt_manager.rs	65.15% <100.00%> (+0.12%)	⬆️
core/server/src/http/jwt/mod.rs	100.00% <100.00%> (ø)
core/connectors/sinks/surrealdb_sink/src/lib.rs	85.38% <85.38%> (ø)

... and 331 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ryerraguntla]

/author

[ryerraguntla]

Please check the pre-checks failure

[countradooku]

Sure

[countradooku]

/ready

[ryerraguntla]

/author - could you please check the pre-checks failures

[countradooku]

/ready

[ryerraguntla]

surrealdb = { workspace = true } pulls surrealdb-core as a transitive dependency. surrealdb-core is licensed under Business Source License 1.1 (BUSL-1.1), an OSI-non-approved license incompatible with Apache 2.0 redistribution. The PR author explicitly flags this: ./scripts/ci/third-party-licenses.sh --validate --manifest core/connectors/sinks/surrealdb_sink/Cargo.toml reports BUSL-1.1 failures. The workspace-level CI passes only because that step runs against core/server/Cargo.toml and core/cli/Cargo.toml, which do not include this crate. The crates.io page for surrealdb v3.1.4 lists the license as "non-standard" and the README confirms surrealdb-core is BUSL-1.1. This cannot be distributed in an Apache Software Foundation project.

[ryerraguntla]

process_messages — every insertion failure is logged and swallowed, function always returns Ok(()):

if let Some(error) = outcome.error {
    self.insertion_errors.fetch_add(batch.len() as u64, Ordering::Relaxed);
    error!("Failed to insert SurrealDB batch ...: {error}");
}
// no early return, no Err propagation
Ok(())

consume() at L258 propagates this directly. The runtime receives Ok(()) and advances the consumer offset, permanently losing all messages from failed batches with no possibility of redelivery. Other connectors in the repo (Meilisearch, S3, Elasticsearch) return Err from consume() to trigger runtime retry.

[ryerraguntla]

build_record_id — the deterministic ID uses message_id (u128) not the message offset:

id.push_str("_m");
let _ = write!(&mut id, "{message_id:032x}");

INSERT IGNORE INTO {table} $records silently deduplicates on matching IDs. If a producer sends messages with id = 0 (the default when IggyMessage::builder().build() is used without setting .id()), every such message in the same stream/topic/partition maps to the same record ID. Only the first insert succeeds; all subsequent messages with id = 0 are silently dropped by SurrealDB with no error returned to the connector. The offset is written as a field but is not part of the record ID. Replace or augment with message.offset, which is unique per partition.

[ryerraguntla]

with_config_defaults — max_retries clamped to .max(1):

self.max_retries = self.config.max_retries.unwrap_or(DEFAULT_MAX_RETRIES).max(1);
Setting max_retries = 0 is silently raised to 1. Document the minimum-1 behavior or reject 0 with an error. Severity:

[ryerraguntla]

wait_until_ready — polls by attempting a full create_client() (WebSocket connect + sign in + use_ns/use_db + health check) on every attempt:

for _ in 0..SURREALDB_BOOT_ATTEMPTS {
    if let Ok(client) = self.create_client().await
        && client.health().await.is_ok()
    {
        return Ok(());
    }
    sleep(Duration::from_millis(SURREALDB_BOOT_INTERVAL_MS)).await;
}

With SURREALDB_BOOT_ATTEMPTS = 120 and SURREALDB_BOOT_INTERVAL_MS = 250, the max wait is 30s. Connection errors during the boot window are swallowed via if let Ok. This is acceptable in test code, but swallowing all errors including non-transient auth failures means a misconfigured test fixture fails with "SurrealDB did not become ready" rather than the actual error.

[ryerraguntla]

@countradooku - Please check the Cargo.toml comment on licensing and distribution in the apache redistribution. Please check with any other packages which are distributed with apache license. Unresolved this could be a show stopper. otherwise looks good with more of nits.

[ryerraguntla]

/author