Skip to content

feat(connectors): add Meilisearch sink connector#3497

Open
countradooku wants to merge 7 commits into
apache:masterfrom
countradooku:feat/meilisearch-sink-connector
Open

feat(connectors): add Meilisearch sink connector#3497
countradooku wants to merge 7 commits into
apache:masterfrom
countradooku:feat/meilisearch-sink-connector

Conversation

@countradooku

Copy link
Copy Markdown
Contributor

Summary

  • add the Meilisearch sink connector using the official meilisearch-sdk
  • document the sink configuration and payload mapping behavior
  • add Docker-backed integration coverage for indexing JSON messages into Meilisearch

Motivation

This splits the sink portion out of #3404 so the Meilisearch sink can be reviewed and merged independently from the source connector.

Closes #3495.

Validation

  • cargo sort --no-format --workspace
  • cargo test -p iggy_connector_meilisearch_sink
  • cargo clippy -p iggy_connector_meilisearch_sink --all-targets -- -D warnings
  • env CARGO_BIN_EXE_iggy-server=... CARGO_BIN_EXE_iggy-connectors=... cargo test -p integration --test mod -- connectors::meilisearch::meilisearch_sink
  • git diff --check

@countradooku countradooku marked this pull request as ready for review June 15, 2026 18:58
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jun 15, 2026
@codecov

codecov Bot commented Jun 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 78.12500% with 154 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.41%. Comparing base (4a48008) to head (3a80598).

Files with missing lines Patch % Lines
core/connectors/sinks/meilisearch_sink/src/lib.rs 78.12% 137 Missing and 17 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3497       +/-   ##
=============================================
- Coverage     74.27%   45.41%   -28.86%     
  Complexity      937      937               
=============================================
  Files          1259     1257        -2     
  Lines        125969   110618    -15351     
  Branches     101644    86335    -15309     
=============================================
- Hits          93558    50239    -43319     
- Misses        29396    57664    +28268     
+ Partials       3015     2715      -300     
Components Coverage Δ
Rust Core 37.94% <78.12%> (-37.22%) ⬇️
Java SDK 58.57% <ø> (ø)
C# SDK 71.40% <ø> (-0.71%) ⬇️
Python SDK 88.88% <ø> (ø)
PHP SDK 84.29% <ø> (ø)
Node SDK 91.13% <ø> (-0.10%) ⬇️
Go SDK 40.36% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/sinks/meilisearch_sink/src/lib.rs 78.12% <78.12%> (ø)

... and 375 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread Cargo.toml
Comment thread Cargo.toml
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs Outdated
@ryerraguntla

ryerraguntla commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

overall correct approach - JSON-object document model with generated PK is sound, wait_for_tasks + partial-error accounting is thoughtful. test coverage is present but there is
coverage gap — only happy-path (2 JSON messages indexed, iggy_id present). No test for: document_action = update semantics, include_metadata = false, raw-payload base64 fallback, or wait_for_tasks = false fire-and-forget path. Not a merge-blocker given this is the initial connector, but tracking it. Along with coverage, please take care of the above mentioned issues.

@ryerraguntla

Copy link
Copy Markdown
Contributor

/author

@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 16, 2026
@countradooku

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 16, 2026
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
@countradooku

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 17, 2026
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/Cargo.toml
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/config.toml
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
Comment thread core/integration/tests/connectors/meilisearch/meilisearch_sink.rs Outdated
Comment thread core/connectors/sinks/meilisearch_sink/src/lib.rs
@ryerraguntla

Copy link
Copy Markdown
Contributor

@countradooku - few blockers, many nits. One thing which is encouraged is to have more test coverage. - Integration coverage: no tests for Payload::Raw, Payload::Text, wait_for_tasks=false, custom primary_key, create_index_if_not_exists=false error path. (intro)

@ryerraguntla

Copy link
Copy Markdown
Contributor

/author

@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 19, 2026
@countradooku

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Meilisearch sink connector

3 participants