PSV2: endpoint to register pipelines

[carlosgjs]

Summary

Adds a pipeline registration API endpoint for V2 (async/NATS) processing services to register their available pipelines with a project. Also makes ProcessingService.endpoint_url nullable to support pull-mode services that don't expose an HTTP endpoint.

Endpoint: GET/POST /api/v2/projects/{project_pk}/pipelines/

• GET lists enabled pipelines for the project
• POST registers pipelines from an async processing service

Design

Uses a nested DRF ViewSet (ProjectPipelineViewSet) following the same pattern as /projects/{id}/members/:

• PipelineRegistrationSerializer for input validation (DRF serializer wrapping pydantic PipelineConfigResponse via SchemaField)
• transaction.atomic() wrapping get_or_create + projects.add + create_pipelines
• Idempotent re-registration (no error if service already associated)
• Role-based permissions (ProjectPipelineConfigPermission)

Key files

File	Change
ami/ml/views.py	New ProjectPipelineViewSet (list + create)
ami/ml/serializers.py	New PipelineRegistrationSerializer
config/api_router.py	Nested route registration
ami/ml/models/processing_service.py	Nullable endpoint_url, status handling for pull-mode
ami/ml/tasks.py	Skip health checks for services without endpoint
ami/main/tests.py	Tests for registration, idempotency, auth, listing

Closes #1086

[netlify]

✅ Deploy Preview for antenna-preview ready!

Name	Link
🔨 Latest commit	1724851
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6993fd710873ec0008a2dd32
😎 Deploy Preview	https://deploy-preview-1076--antenna-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	1 paths audited Performance: 74 (🟢 up 8 from production) Accessibility: 89 (🟢 up 9 from production) Best Practices: 92 (🔴 down 8 from production) SEO: 100 (🟢 up 8 from production) PWA: 80 (no change from production) View the detailed breakdown and full score reports

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR implements async/pull-mode pipeline registration by making ProcessingService endpoint URLs optional, introducing three new pipeline configuration permissions, adding a nested project pipelines API endpoint, establishing permission checks with a new ProjectPipelineConfigPermission class, and updating role-based access control.

Changes

Cohort / File(s)	Summary
Permission & Role Updates ami/main/models.py, ami/users/roles.py	Added three new pipeline configuration permissions (CREATE, UPDATE, DELETE) to Project model and assigned them to MLDataManager and ProjectManager roles. Enhanced permission checks for "pipelines" action requiring ProjectManager role.
Database Migration ami/ml/migrations/0026_make_processing_service_endpoint_url_nullable.py	Altered ProcessingService.endpoint_url to accept NULL values and blank strings, enabling pull-mode services without configured endpoints.
ProcessingService Model & Logic ami/ml/models/processing_service.py, ami/ml/tasks.py	Made endpoint_url nullable and added last_checked_latency field. Updated get_status and get_pipeline_configs to short-circuit when endpoint_url is None. Modified check_processing_services_online to target only v1 synchronous services with valid endpoints.
API Schemas & Serializers ami/ml/schemas.py, ami/ml/serializers.py	Added AsyncPipelineRegistrationRequest model, made ProcessingServiceStatusResponse.endpoint_url optional, and introduced PipelineRegistrationSerializer for pipeline registration payloads.
API Views & Routing ami/ml/views.py, config/api_router.py, ami/base/permissions.py	Introduced ProjectPipelineViewSet for nested /projects/{project_id}/pipelines endpoint with list and create operations. Added ProjectPipelineConfigPermission class for pipeline-scoped access control. Registered nested route in API router.
Test Coverage ami/main/tests.py, ami/ml/tests.py	Added TestProjectPipelinesAPI test suite covering pipeline registration, listing, and authorization scenarios. Added three test cases validating null endpoint_url behavior in ProcessingService operations.
Planning Documentation .agents/planning/pipeline-registration-drf.md	Added implementation guide for nested DRF routing of project pipelines with validation, transaction handling, and example payloads.

Sequence Diagram

sequenceDiagram
    actor Client
    participant ProjectPipelineViewSet as ProjectPipelineViewSet
    participant ProcessingService as ProcessingService
    participant Database as Database
    participant Project as Project

    Client->>ProjectPipelineViewSet: POST /projects/{id}/pipelines<br/>(service_name, configs)
    ProjectPipelineViewSet->>ProjectPipelineViewSet: Validate request<br/>Check permissions
    ProjectPipelineViewSet->>Database: get_or_create ProcessingService<br/>by name
    Database-->>ProcessingService: Service instance
    ProjectPipelineViewSet->>Project: Link service to project
    ProjectPipelineViewSet->>ProcessingService: create_pipelines(configs, project)
    ProcessingService->>Database: Create pipeline configs<br/>for project
    Database-->>ProjectPipelineViewSet: Pipeline data
    ProjectPipelineViewSet-->>Client: 201 Created<br/>Registered pipelines

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~28 minutes

Possibly related issues

• #1120: Addresses overlapping concerns about project-scoped pipeline and processing-service endpoints with proper permission scoping and access controls.

Suggested labels

backend, ml

Poem

🐰 The pipeline hops and bounds,
No endpoint needed found,
Async workers sing and play,
Register their pipes today!
Pull-mode dreams come into sight, ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'PSV2: endpoint to register pipelines' is concise, clear, and directly describes the main feature introduced—a new pipeline registration endpoint for V2 processing services.
Linked Issues check	✅ Passed	The PR successfully implements the core requirement from #1086: an API endpoint for V2 services to register pipelines. It makes endpoint_url nullable and includes comprehensive tests and documentation updates as required.
Out of Scope Changes check	✅ Passed	All changes are directly related to the pipeline registration feature. Model updates, migrations, tests, serializers, views, and permissions all support the core objective of enabling pipeline registration for async V2 services.
Description check	✅ Passed	The PR description is comprehensive and well-structured, covering summary, design approach, key files, and related issue reference.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[netlify]

✅ Deploy Preview for antenna-ssec canceled.

Name	Link
🔨 Latest commit	1724851
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-ssec/deploys/6993fd71eafc350008a63de9

[coderabbitai]

Actionable comments posted: 3

🤖 Fix all issues with AI agents

In `@ami/main/api/views.py`:
- Around line 240-262: The current logic in the view returns a 400 when a
ProcessingService (found via
ProcessingService.objects.filter(name=parsed.processing_service_name).first())
is already associated with the project, which prevents idempotent
re-registration; update the branch handling the existing processing_service so
that if project is already in processing_service.projects.all() you do not
return Response(status=400) but simply continue (no-op) and allow the endpoint
to proceed (e.g., log an info/debug message) — ensure you keep the
processing_service.projects.add(project)/save() when association is missing, and
remove the early return that blocks subsequent pipeline registration.

In `@ami/ml/schemas.py`:
- Around line 326-327: The endpoint_url annotation is optional but currently
still required because it has no default; update the schema so endpoint_url has
an explicit default (e.g., set endpoint_url to None or use Field(default=None)
if this is a Pydantic model) so parsing succeeds when senders omit it—modify the
endpoint_url declaration (near the latency: float field) to include the default
None.

In `@ami/ml/tasks.py`:
- Around line 112-114: The current check only skips when service.endpoint_url is
None but still allows empty strings; update the conditional around
service.endpoint_url in the loop (the block that calls logger.warning and
continue) to treat None, empty, or whitespace-only values as missing — e.g.,
ensure you test both falsy and stripped emptiness (safe-check to avoid calling
.strip() on None) so that logger.warning(f"Processing service {service} has no
endpoint URL, skipping.") is used and the loop continues for None/"", or
whitespace-only endpoint_url.

🧹 Nitpick comments (1)

ami/main/tests.py (1)

3457-3459: Consider disabling default fixtures for these API tests.
create_defaults=True builds extra related objects these tests don’t use; turning it off keeps setup lean.

♻️ Suggested change

-        self.project = Project.objects.create(name="Test Project", owner=self.user, create_defaults=True)
-        self.other_project = Project.objects.create(name="Other Project", owner=self.other_user, create_defaults=True)
+        self.project = Project.objects.create(name="Test Project", owner=self.user, create_defaults=False)
+        self.other_project = Project.objects.create(name="Other Project", owner=self.other_user, create_defaults=False)

[Copilot]

Pull request overview

This pull request introduces a new API endpoint for V2 ML processing services to register pipelines with projects in "pull mode" (without requiring an endpoint URL). The changes enable processing services to push pipeline configurations to Antenna rather than Antenna pulling them from a service endpoint.

Changes:

• Added a new POST endpoint /api/v2/projects/{id}/pipelines/ for pipeline registration
• Made ProcessingService.endpoint_url nullable to support pull-mode services
• Updated ProcessingService methods to handle null endpoint URLs gracefully

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
ami/main/api/views.py	Added new pipelines action to ProjectViewSet for receiving pipeline registrations from V2 services
ami/main/models.py	Added permission check for "pipelines" action requiring ProjectManager role
ami/ml/models/processing_service.py	Made endpoint_url nullable and updated get_status() and get_pipeline_configs() to handle null values
ami/ml/schemas.py	Made endpoint_url nullable in ProcessingServiceStatusResponse and added AsyncPipelineRegistrationRequest schema
ami/ml/migrations/0026_make_processing_service_endpoint_url_nullable.py	Database migration to make endpoint_url nullable
ami/ml/tasks.py	Added check to skip processing services without endpoint URLs in periodic status check
ami/ml/tests.py	Added comprehensive tests for null endpoint_url handling
ami/main/tests.py	Added test suite for the new pipelines API endpoint
requirements/base.txt	Added explanatory comment about psycopg binary vs non-binary versions

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@ami/ml/schemas.py`:
- Around line 339-345: Update the inaccurate docstring on
AsyncPipelineRegistrationRequest to reference AsyncPipelineRegistrationRequest
(not PipelineRegistrationResponse) and clearly state it represents the async
callback payload containing processing results; rename the output-oriented model
PipelineRegistrationResponse to a more generic name like
PipelineProcessingResult (or create a new PipelineProcessingResult model) and
adjust its fields (timestamp, success, error, pipelines_created,
algorithms_created) to be output/result-oriented (optional or excluded from
required input validation) so the
AsyncPipelineRegistrationRequest.pipeline_response type reflects a result object
rather than required caller-supplied output fields; update all references to
PipelineRegistrationResponse to the new name (or to the new model) including the
AsyncPipelineRegistrationRequest definition and related validation/serialization
logic.

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@ami/main/api/views.py`:
- Around line 232-233: Update the docstring in ami/main/api/views.py that
describes the AsyncPipelineRegistrationRequest payload: change the field name
from `pipelines_response` to the correct `pipeline_response` so it reads that
the list of PipelineConfigResponse objects is under the
`pipeline_response.pipelines` key; keep references to
AsyncPipelineRegistrationRequest and PipelineConfigResponse so readers can
locate the schema and adjust any adjacent wording to match the actual field
name.

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@ami/main/api/views.py`:
- Around line 226-227: The pipelines view method currently defines an unused
parameter pk which triggers ARG002; update the method signature for def
pipelines(self, request, pk=None): to silence the lint warning by marking the
unused argument with a noqa (e.g., def pipelines(self, request, pk=None):  #
noqa: ARG002) or remove/rename if appropriate; ensure the change references the
pipelines method so the decorator `@action`(detail=True, methods=["post"],
url_path="pipelines") remains unchanged.

[mihow]

@annavik it would be great to discuss the UX for pipeline registration this week while we have @carlosgjs

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@ami/main/api/views.py`:
- Around line 254-283: Wrap the sequence of DB writes and pipeline creation in a
single atomic transaction: import transaction from django.db and enclose
ProcessingService.objects.get_or_create(...), the
check/processing_service.projects.add(project), and the call to
processing_service.create_pipelines(...) inside a with transaction.atomic():
block so any failure in create_pipelines rolls back the get_or_create/add
changes; keep the existing logging and Response handling inside the block or
immediately after as appropriate.

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@ami/main/tests.py`:
- Around line 3539-3556: The test_list_pipelines test fails when
DEFAULT_PROCESSING_SERVICE_ENDPOINT is not set because
get_or_create_default_processing_service() returns None and no pipelines are
created; fix by either adding an
override_settings(DEFAULT_PROCESSING_SERVICE_ENDPOINT="http://ml_backend:2009/")
decorator to test_list_pipelines so create_defaults=True produces pipelines, or
explicitly create a ProcessingService and at least one Pipeline in setUp (use
ProcessingService and Pipeline model constructors and ensure
project_pipeline_configs__enabled=True for the Pipeline) so that
Pipeline.objects.filter(projects=self.project,
project_pipeline_configs__enabled=True) returns results.

In `@ami/ml/views.py`:
- Around line 248-265: In create(), guard against self.get_active_project()
returning None: after calling project = self.get_active_project(), check if
project is None and raise a DRF exception (e.g., NotFound or ValidationError)
with a clear message so processing_service.projects.add(project) never receives
None; keep the rest of the transaction logic
(ProcessingService.objects.get_or_create and
processing_service.create_pipelines) unchanged and return the response as
before.

🧹 Nitpick comments (4)

ami/users/roles.py (1)

194-196: Minor: CREATE_PROJECT_PIPELINE_CONFIG is redundant in the explicit set.

ProjectManager.permissions already includes MLDataManager.permissions (Line 162), which contains CREATE_PROJECT_PIPELINE_CONFIG (Line 147). The explicit addition on Line 194 is a no-op due to the set union. Removing it would reduce duplication, but this is purely cosmetic since set semantics make it harmless.

🧹 Optional cleanup

             Project.Permissions.CREATE_USER_PROJECT_MEMBERSHIP,
             Project.Permissions.UPDATE_USER_PROJECT_MEMBERSHIP,
             Project.Permissions.DELETE_USER_PROJECT_MEMBERSHIP,
-            Project.Permissions.CREATE_PROJECT_PIPELINE_CONFIG,
             Project.Permissions.UPDATE_PROJECT_PIPELINE_CONFIG,
             Project.Permissions.DELETE_PROJECT_PIPELINE_CONFIG,
         }

ami/ml/views.py (1)

205-221: Guard against get_active_project() returning None.

Although ProjectPipelineConfigPermission.has_permission already validates the project exists (returning False if not), get_active_project() is called again in get_queryset without a guard. If the permission class is ever bypassed (e.g., by a superclass or middleware change), filtering with projects=None would silently return an empty queryset instead of failing explicitly.

Consider adding a defensive check consistent with how other viewsets handle this:

🛡️ Optional defensive guard

     def get_queryset(self) -> QuerySet:
         project = self.get_active_project()
+        if not project:
+            return Pipeline.objects.none()
         return (
             Pipeline.objects.filter(projects=project, project_pipeline_configs__enabled=True)

ami/ml/serializers.py (1)

170-172: Add max_length=255 to processing_service_name to match the model constraint.

The ProcessingService.name field has max_length=255. Without matching this in the serializer's CharField(), validation fails at the database layer instead of the API layer, resulting in poor error feedback to users.

The default=[] for pipelines is acceptable—the create_pipelines() method handles empty pipeline lists gracefully by falling back to get_pipeline_configs() if needed.

ami/main/tests.py (1)

3470-3475: Test payload always has an empty pipelines list — no coverage of actual pipeline registration logic.

All POST tests use _get_test_payload which sends "pipelines": []. This means the pipeline creation/update paths within the view's create action (and the associated serializer/schema validation for pipeline entries) are never exercised. Consider adding at least one test that includes realistic pipeline data in the payload to verify the full registration flow.

[mihow]

@mohamedelabbas1996 Could we calll this CREATE_PROJECT_PIPELINE_REGISTRATION? or does the name need to exactly match the through model name?