IMPR: split unit/integration test

[drewyangdev]

Improvement Request

Problem

Currently, datajoint-python pytest runs unit and integration test by default, which requires a DB

pytest --cov-report term-missing --cov=datajoint tests
# trying to connect to a DB, otherwise integration test would fail

so it actually needs runs in docker that comes with a DB container

docker compose --profile test up --quiet-pull --build --exit-code-from djtest djtest

Requirements

N/A

Justification

We should add unit test check in pre-commit locally, and let the CI to run integration test with docker

Alternative Considerations

N/A

Related Errors

N/A

Please include steps to reproduce provided errors as follows:
N/A

Screenshots

N/A

Additional Research and Context

N/A

[dimitri-yatsenko]

DataJoint 2.0 Implementation

This improvement is implemented in DataJoint 2.0. Tests are now split into separate directories with independent CI jobs.

Test Structure

tests/
├── unit/                    # No database required
│   ├── test_codecs.py
│   ├── test_condition.py
│   ├── test_hash.py
│   ├── test_pk_rules.py
│   └── test_settings.py
├── integration/             # Requires MySQL/MinIO (via testcontainers)
│   ├── test_blob.py
│   ├── test_fetch.py
│   └── ... (40+ test files)
└── conftest.py

CI Workflow

The GitHub Actions workflow (.github/workflows/test.yaml) now has two separate jobs:

jobs:
  # Full integration tests with testcontainers
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: prefix-dev/setup-pixi@v0.9.3
      - run: pixi run -e test test-cov

  # Unit tests only - faster feedback, no containers
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: prefix-dev/setup-pixi@v0.9.3
      - run: pixi run -e test pytest tests/unit -v

Local Development

Run unit tests locally without any database:

# With pixi (recommended)
pixi run -e test pytest tests/unit/

# With pip
pytest tests/unit/

Run full test suite (testcontainers auto-manages MySQL/MinIO):

pixi run test

Benefits

Aspect	Before	After
Local unit tests	Required DB connection	No dependencies
CI feedback	Single slow job	Fast unit-tests job + full integration
Pre-commit potential	Not feasible	Can add unit tests to pre-commit

The unit tests complete in ~30 seconds in CI, providing fast feedback before the full integration suite (~2-3 minutes).

[dimitri-yatsenko]

Closing as resolved in DataJoint 2.0. Tests are now split into tests/unit/ and tests/integration/ with separate CI jobs.

[drewyangdev]

@dimitri-yatsenko Might want to consider also adding it to pre-commit, so that the unit test can fail before the CI failed to save some time.