Skip to content

feat(connectors): add Meilisearch source connector#3498

Open
countradooku wants to merge 7 commits into
apache:masterfrom
countradooku:feat/meilisearch-source-connector
Open

feat(connectors): add Meilisearch source connector#3498
countradooku wants to merge 7 commits into
apache:masterfrom
countradooku:feat/meilisearch-source-connector

Conversation

@countradooku

Copy link
Copy Markdown
Contributor

Summary

  • add the Meilisearch source connector using the official meilisearch-sdk
  • document source configuration, retry behavior, and primary-key cursor requirements
  • add Docker-backed integration coverage for producing indexed Meilisearch documents into Iggy

Motivation

This splits the source portion out of #3404 so the Meilisearch source can be reviewed and merged independently from the sink connector.

Closes #3496.

Validation

  • cargo sort --no-format --workspace
  • cargo test -p iggy_connector_meilisearch_source
  • cargo clippy -p iggy_connector_meilisearch_source --all-targets -- -D warnings
  • env CARGO_BIN_EXE_iggy-server=... CARGO_BIN_EXE_iggy-connectors=... cargo test -p integration --test mod -- connectors::meilisearch::meilisearch_source
  • git diff --check

@countradooku countradooku marked this pull request as ready for review June 15, 2026 18:59
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jun 15, 2026
@codecov

codecov Bot commented Jun 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 78.58407% with 121 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.39%. Comparing base (4a48008) to head (d7d4405).

Files with missing lines Patch % Lines
...e/connectors/sources/meilisearch_source/src/lib.rs 78.58% 96 Missing and 25 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3498       +/-   ##
=============================================
- Coverage     74.27%   45.39%   -28.88%     
  Complexity      937      937               
=============================================
  Files          1259     1257        -2     
  Lines        125969   110479    -15490     
  Branches     101644    86198    -15446     
=============================================
- Hits          93558    50156    -43402     
- Misses        29396    57600    +28204     
+ Partials       3015     2723      -292     
Components Coverage Δ
Rust Core 37.88% <78.58%> (-37.28%) ⬇️
Java SDK 58.57% <ø> (ø)
C# SDK 71.43% <ø> (-0.68%) ⬇️
Python SDK 88.88% <ø> (ø)
PHP SDK 84.29% <ø> (ø)
Node SDK 91.35% <ø> (+0.12%) ⬆️
Go SDK 40.36% <ø> (ø)
Files with missing lines Coverage Δ
...e/connectors/sources/meilisearch_source/src/lib.rs 78.58% <78.58%> (ø)

... and 374 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread Cargo.toml
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs Outdated
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs Outdated
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs
Comment thread core/integration/tests/connectors/fixtures/meilisearch/container.rs
@ryerraguntla

Copy link
Copy Markdown
Contributor

overall correct direction - cursor-by-primary-key is the right shape for Meilisearch, test coverage is present but coverage gap — only happy-path covered. No test for: (a) state restored across restart → cursor starts from last_primary_key, not from zero; (b) second poll produces zero duplicates. refer The elasticsearch_source.rs has state_persists_across_connector_restart as a pattern. For a cursor-based source, duplicate-freedom is the core invariant; it should be integration-tested.

@ryerraguntla

Copy link
Copy Markdown
Contributor

/author

@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 16, 2026
@countradooku

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 16, 2026
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs
Comment thread core/integration/tests/connectors/fixtures/meilisearch/container.rs
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs
Comment thread core/connectors/sources/meilisearch_source/README.md
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs Outdated
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs Outdated
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs Outdated
Comment thread core/integration/tests/connectors/fixtures/meilisearch/container.rs
@ryerraguntla

Copy link
Copy Markdown
Contributor

@countradooku - Looks very good. There are no blockers/criticals , only warnings and nits. After this fix, it should be good. @hubcio - After these are taken care, could one of the committers review and merge

@ryerraguntla

Copy link
Copy Markdown
Contributor

/author

@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 16, 2026
@countradooku

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 17, 2026
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs
Comment thread core/connectors/sources/meilisearch_source/src/lib.rs
Comment thread core/connectors/sources/meilisearch_source/config.toml
Comment thread core/integration/tests/connectors/meilisearch/meilisearch_source.rs Outdated
Comment thread core/integration/tests/connectors/meilisearch/meilisearch_source.rs Outdated
@ryerraguntla

Copy link
Copy Markdown
Contributor

/author

@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-author PR is waiting on author response

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Meilisearch source connector

3 participants