Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions common/storage/milvus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@

This module implements vector database storage using Milvus for the vCon server.

> **Optional dependency.** `pymilvus` (which pulls in pandas + grpcio, ~85MB the
> rest of vcon-server does not use) is **not** part of the base `storage` group.
> Install it explicitly to use this backend:
> `uv sync --group storage --group storage-milvus`. Without it, importing
> `storage.milvus` will fail with `ImportError`/`ModuleNotFoundError` due to the missing dependency.

## Overview

Milvus storage provides high-performance vector similarity search capabilities, ideal for storing and retrieving vector embeddings of vCon data. It's particularly useful for semantic search and similarity matching applications.
Expand Down
5 changes: 5 additions & 0 deletions common/storage/milvus/test_milvus.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
import pytest
from unittest.mock import patch, MagicMock, mock_open

# pymilvus/openai are optional dependencies (group: storage-milvus). storage.milvus
# imports them at module load, so skip this whole module when they aren't installed.
pytest.importorskip("pymilvus")
pytest.importorskip("openai")

from lib.vcon_redis import VconRedis
from vcon import Vcon
from storage.milvus import (
Expand Down
5 changes: 5 additions & 0 deletions common/storage/milvus/test_milvus_branches.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@

import pytest

# pymilvus/openai are optional dependencies (group: storage-milvus). storage.milvus
# imports them at module load, so skip this whole module when they aren't installed.
pytest.importorskip("pymilvus")
pytest.importorskip("openai")

from storage import milvus as milvus_module
from storage.milvus import (
check_vcon_exists,
Expand Down
4 changes: 3 additions & 1 deletion docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,11 @@ COPY pyproject.toml uv.lock /app/

# Install all groups (conserver + api + dev) so the image works for both
# running services and running pytest.
# storage-milvus is included here (but NOT in the production Dockerfile.conserver /
# Dockerfile.api images) so the Milvus storage tests run in CI rather than skip.
# Venv at /opt/venv so docker-compose volume mounts don't wipe it.
RUN uv venv --seed /opt/venv && \
UV_PROJECT_ENVIRONMENT=/opt/venv uv sync --frozen --group conserver --group api --group dev
UV_PROJECT_ENVIRONMENT=/opt/venv uv sync --frozen --group conserver --group api --group dev --group storage-milvus
ENV PATH="/opt/venv/bin:$PATH"

# Auto-install OTel instrumentation packages for the installed libraries.
Expand Down
12 changes: 11 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,23 @@ dependencies = [
storage = [
"pymongo>=4.7.2",
"elasticsearch>=8.13.1,<9.0.0",
"pymilvus>=2.3.0",
"msal>=1.32.3",
"boto3>=1.34.52",
"psycopg2-binary>=2.9.9",
"peewee>=3.17.1",
]

# Optional: Milvus vector-database storage backend (storage.milvus).
# pymilvus drags in pandas (~48MB) and grpcio (~37MB), neither of which the
# rest of vcon-server uses, so it is kept out of the base `storage` group.
# Includes openai for the embedding calls the backend makes. Only installs
# pymilvus when explicitly requested; storage.milvus raises a clear ImportError
# otherwise. Install with: uv sync --group storage --group storage-milvus
Comment thread
howethomas marked this conversation as resolved.
Comment on lines +46 to +48
storage-milvus = [
"pymilvus>=2.3.0",
"openai>=1.60.0",
]

# API service dependencies — HTTP layer and API key management.
# Install with: uv sync --group api
api = [
Expand Down
14 changes: 8 additions & 6 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.