Add layered MergePipeline for multi-source entity discovery

docs/config/discovery-options.rst

@@ -87,6 +87,98 @@ The ``min_topics_for_component`` parameter (default: 1) sets the minimum number
 of topics required before creating a component. This can filter out namespaces
 with only a few stray topics.
 
+Merge Pipeline Options (Hybrid Mode)
+-------------------------------------
+
+When using ``hybrid`` mode, the merge pipeline controls how entities from
+different discovery layers are combined. The ``merge_pipeline`` section
+configures gap-fill behavior for runtime-discovered entities.
+
+Gap-Fill Configuration
+^^^^^^^^^^^^^^^^^^^^^^
+
+In hybrid mode, the manifest is the source of truth. Runtime (heuristic) discovery
+fills gaps - entities that exist at runtime but aren't in the manifest. Gap-fill
+controls restrict what the runtime layer is allowed to create:
+
+.. code-block:: yaml
+
+   discovery:
+     merge_pipeline:
+       gap_fill:
+         allow_heuristic_areas: true
+         allow_heuristic_components: true
+         allow_heuristic_apps: true
+         allow_heuristic_functions: false
+         namespace_whitelist: []
+         namespace_blacklist: []
+
+.. list-table:: Gap-Fill Options
+   :header-rows: 1
+   :widths: 35 15 50
+
+   * - Parameter
+     - Default
+     - Description
+   * - ``allow_heuristic_areas``
+     - ``true``
+     - Allow runtime layer to create Area entities not in the manifest
+   * - ``allow_heuristic_components``
+     - ``true``
+     - Allow runtime layer to create Component entities not in the manifest
+   * - ``allow_heuristic_apps``
+     - ``true``
+     - Allow runtime layer to create App entities not in the manifest
+   * - ``allow_heuristic_functions``
+     - ``false``
+     - Allow runtime layer to create Function entities (rarely useful at runtime)
+   * - ``namespace_whitelist``
+     - ``[]``
+     - If non-empty, only allow gap-fill from these ROS 2 namespaces (Areas and Components only)
+   * - ``namespace_blacklist``
+     - ``[]``
+     - Exclude gap-fill from these ROS 2 namespaces (Areas and Components only)
+
+When both whitelist and blacklist are empty, all namespaces are eligible for gap-fill.
+If whitelist is non-empty, only listed namespaces pass. Blacklist is always applied.
+
+Namespace matching uses path-segment boundaries: ``/nav`` matches ``/nav`` and ``/nav/sub``
+but NOT ``/navigation``. Both lists only filter Areas and Components (which carry
+``namespace_path``). Apps and Functions are not namespace-filtered.
+
+
+Merge Policies
+^^^^^^^^^^^^^^
+
+Each discovery layer declares a ``MergePolicy`` per field group. When two layers
+provide the same entity (matched by ID), policies determine which values win:
+
+.. list-table:: Merge Policies
+   :header-rows: 1
+   :widths: 25 75
+
+   * - Policy
+     - Description
+   * - ``AUTHORITATIVE``
+     - This layer's value wins over lower-priority layers.
+       If two AUTHORITATIVE layers conflict, a warning is logged and the
+       higher-priority (earlier) layer wins.
+   * - ``ENRICHMENT``
+     - Fill empty fields from this layer. Non-empty target values are preserved.
+       Two ENRICHMENT layers merge additively (collections are unioned).
+   * - ``FALLBACK``
+     - Use only if no other layer provides the value. Two FALLBACK layers
+       merge additively (first non-empty fills gaps).
+
+**Built-in layer policies:**
+
+- **ManifestLayer** (priority 1): IDENTITY, HIERARCHY, METADATA are AUTHORITATIVE.
+  LIVE_DATA is ENRICHMENT (runtime topics/services take precedence).
+  STATUS is FALLBACK (manifest cannot know online state).
+- **RuntimeLayer** (priority 2): LIVE_DATA and STATUS are AUTHORITATIVE.
+  METADATA is ENRICHMENT. IDENTITY and HIERARCHY are FALLBACK.
+- **PluginLayer** (priority 3+): All field groups ENRICHMENT
+
 Configuration Example
 ---------------------
 
@@ -100,9 +192,6 @@ Complete YAML configuration for runtime discovery:
          mode: "runtime_only"
 
          runtime:
-           # Map nodes to Apps
-           expose_nodes_as_apps: true
-
            # Group Apps into Components by namespace
            create_synthetic_components: true
            grouping_strategy: "namespace"
@@ -112,6 +201,16 @@ Complete YAML configuration for runtime discovery:
            topic_only_policy: "create_component"
            min_topics_for_component: 2  # Require at least 2 topics
 
+         # Note: merge_pipeline settings only apply when mode is "hybrid"
+         merge_pipeline:
+           gap_fill:
+             allow_heuristic_areas: true
+             allow_heuristic_components: true
+             allow_heuristic_apps: true
+             allow_heuristic_functions: false
+             namespace_whitelist: []
+             namespace_blacklist: ["/rosout", "/parameter_events"]
+
 Command Line Override
 ---------------------
 
@@ -128,3 +227,5 @@ See Also
 
 - :doc:`manifest-schema` - Manifest-based configuration
 - :doc:`/tutorials/heuristic-apps` - Tutorial on runtime discovery
+- :doc:`/tutorials/manifest-discovery` - Hybrid mode tutorial
+- :doc:`/tutorials/plugin-system` - Plugin layer integration

docs/config/manifest-schema.rst

@@ -441,7 +441,9 @@ ros_binding Fields
 
 **Matching behavior:**
 
-1. **Exact match** (default): ``node_name`` and ``namespace`` must match exactly
+1. **Name and namespace match** (default): ``node_name`` must match exactly.
+   ``namespace`` uses path-segment-boundary matching: ``/nav`` matches ``/nav``
+   and ``/nav/sub`` but NOT ``/navigation``.
 2. **Wildcard namespace**: Set ``namespace: "*"`` to match node in any namespace
 3. **Topic namespace**: Match nodes by their published topic prefix
 

docs/tutorials/manifest-discovery.rst

@@ -119,9 +119,9 @@ Option 1: Using Launch File
                package='ros2_medkit_gateway',
                executable='gateway_node',
                parameters=[{
-                   'manifest.enabled': True,
-                   'manifest.file_path': '/path/to/system_manifest.yaml',
-                   'manifest.mode': 'hybrid',
+                   'discovery.mode': 'hybrid',
+                   'discovery.manifest_path': '/path/to/system_manifest.yaml',
+                   'discovery.manifest_strict_validation': True,
                }]
            )
        ])
@@ -137,18 +137,15 @@ Add to your ``gateway_params.yaml``:
      ros__parameters:
        # ... existing parameters ...
 
-       manifest:
-         # Enable manifest-based discovery
-         enabled: true
-
-         # Path to manifest YAML file
-         file_path: "/path/to/system_manifest.yaml"
-
+       discovery:
          # Discovery mode: "runtime_only", "hybrid", or "manifest_only"
          mode: "hybrid"
 
+         # Path to manifest YAML file (required for hybrid and manifest_only)
+         manifest_path: "/path/to/system_manifest.yaml"
+
          # Strict validation: fail on any validation error
-         strict_validation: true
+         manifest_strict_validation: true
 
 Then launch with:
 
@@ -163,9 +160,8 @@ Option 3: Command Line Parameters
 .. code-block:: bash
 
    ros2 run ros2_medkit_gateway gateway_node --ros-args \
-       -p manifest.enabled:=true \
-       -p manifest.file_path:=/path/to/system_manifest.yaml \
-       -p manifest.mode:=hybrid
+       -p discovery.mode:=hybrid \
+       -p discovery.manifest_path:=/path/to/system_manifest.yaml
 
 Verifying the Configuration
 ---------------------------
@@ -207,16 +203,67 @@ List functions:
 
    curl http://localhost:8080/api/v1/functions
 
-Understanding Runtime Linking
------------------------------
+Understanding Hybrid Mode
+-------------------------
+
+In hybrid mode, discovery uses a **merge pipeline** that combines entities from
+multiple discovery layers:
+
+1. **ManifestLayer** (highest priority) - entities from the YAML manifest
+2. **RuntimeLayer** - entities discovered via ROS 2 graph introspection
+3. **PluginLayers** (optional) - entities from gateway plugins
+
+The pipeline merges entities by ID. When the same entity appears in multiple layers,
+per-field-group merge policies determine which values win. See
+:doc:`/config/discovery-options` for details on merge policies and gap-fill configuration.
+
+After merging, the **RuntimeLinker** binds manifest apps to running ROS 2 nodes:
+
+1. **Discovery**: All layers produce entities
+2. **Merging**: Pipeline merges entities by ID, applying field-group policies
+3. **Linking**: For each manifest app, checks ``ros_binding`` configuration
+4. **Binding**: If match found, copies runtime resources (topics, services, actions)
+5. **Status**: Apps with matched nodes are marked ``is_online: true``
+
+Merge Report
+~~~~~~~~~~~~
+
+After each pipeline execution, the gateway produces a ``MergeReport`` available
+via the health endpoint (``GET /health``). The report includes:
 
-In hybrid mode, manifest apps are automatically linked to running ROS 2 nodes.
-The linking process:
+- Layer names and ordering
+- Total entity count, enrichment count
+- Conflict details (which layers disagreed on which field groups)
+- Cross-type ID collision warnings
+- Gap-fill filtering statistics
 
-1. **Discovery**: Gateway discovers running ROS 2 nodes
-2. **Matching**: For each manifest app, checks ``ros_binding`` configuration
-3. **Linking**: If match found, copies runtime resources (topics, services, actions)
-4. **Status**: Apps with matched nodes are marked ``is_online: true``
+In hybrid mode, the ``GET /health`` response includes full discovery diagnostics:
+
+.. code-block:: json
+
+   {
+     "discovery": {
+       "mode": "hybrid",
+       "strategy": "hybrid",
+       "pipeline": {
+         "layers": ["manifest", "runtime"],
+         "total_entities": 12,
+         "enriched_count": 8,
+         "conflict_count": 0,
+         "id_collisions": 0
+       },
+       "linking": {
+         "linked_count": 5,
+         "orphan_count": 1,
+         "binding_conflicts": 0,
+         "warnings": ["Orphan node: /unmanifested_node"]
+       }
+     }
+   }
+
+
+Runtime Linking
+~~~~~~~~~~~~~~~
 
 ROS Binding Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -237,7 +284,8 @@ The ``ros_binding`` section specifies how to match an app to a ROS 2 node:
 
 **Match Types:**
 
-- **Exact match** (default): Node name and namespace must match exactly
+- **Name and namespace match** (default): Node name must match exactly. Namespace uses
+  path-segment-boundary matching (``/nav`` matches ``/nav`` and ``/nav/sub`` but NOT ``/navigation``)
 - **Wildcard namespace**: Use ``namespace: "*"`` to match any namespace
 - **Topic namespace**: Match nodes by their topic prefix
 
@@ -360,6 +408,12 @@ in the manifest. The ``config.unmanifested_nodes`` setting controls this:
 - ``error``: Fail startup if orphan nodes detected
 - ``include_as_orphan``: Include with ``source: "orphan"``
 
+.. note::
+   In hybrid mode with gap-fill configuration (see :doc:`/config/discovery-options`),
+   namespace filtering controls which runtime entities enter the pipeline.
+   ``unmanifested_nodes`` controls how runtime nodes that passed gap-fill
+   but did not match any manifest app are handled by the RuntimeLinker.
+
 Hot Reloading
 -------------
 

docs/tutorials/migration-to-manifest.rst

@@ -306,9 +306,8 @@ Step 7: Test in Hybrid Mode
    .. code-block:: bash
 
       ros2 run ros2_medkit_gateway gateway_node --ros-args \
-          -p manifest.enabled:=true \
-          -p manifest.file_path:=/path/to/system_manifest.yaml \
-          -p manifest.mode:=hybrid
+          -p discovery.mode:=hybrid \
+          -p discovery.manifest_path:=/path/to/system_manifest.yaml
 
 3. **Check manifest status**:
 
@@ -378,9 +377,9 @@ Run validation:
 
    # Start gateway with strict validation
    ros2 run ros2_medkit_gateway gateway_node --ros-args \
-       -p manifest.enabled:=true \
-       -p manifest.file_path:=/path/to/system_manifest.yaml \
-       -p manifest.strict_validation:=true
+       -p discovery.mode:=hybrid \
+       -p discovery.manifest_path:=/path/to/system_manifest.yaml \
+       -p discovery.manifest_strict_validation:=true
 
 Common Issues
 -------------

docs/tutorials/plugin-system.rst

@@ -10,8 +10,9 @@ Overview
 Plugins implement the ``GatewayPlugin`` C++ base class plus one or more typed provider interfaces:
 
 - **UpdateProvider** - software update backend (CRUD, prepare/execute, automated, status)
-- **IntrospectionProvider** *(preview)* - enriches discovered entities with platform-specific metadata.
-  This interface is defined and can be implemented, but is not yet wired into the discovery cycle.
+- **IntrospectionProvider** - enriches discovered entities with platform-specific metadata
+  via the merge pipeline. In hybrid mode, each IntrospectionProvider is wrapped as a
+  ``PluginLayer`` and added to the pipeline with ENRICHMENT merge policy.
 
 A single plugin can implement multiple provider interfaces. For example, a "systemd" plugin
 could provide both introspection (discover systemd units) and updates (manage service restarts).
@@ -300,7 +301,12 @@ Multiple Plugins
 Multiple plugins can be loaded simultaneously:
 
 - **UpdateProvider**: Only one plugin's UpdateProvider is used (first in config order)
-- **IntrospectionProvider**: All plugins' results are merged *(preview - not yet wired)*
+- **IntrospectionProvider**: All plugins are added as PluginLayers to the merge pipeline.
+  Each plugin's entities are merged with ENRICHMENT policy - they fill empty fields but
+  never override manifest or runtime values. Plugins are added after all built-in layers,
+  and the pipeline is refreshed once after all plugins are registered (batch registration).
+  The ``introspect()`` method receives an ``IntrospectionInput`` populated with all entities
+  from previous layers (manifest + runtime), enabling context-aware metadata and discovery.
 - **Custom routes**: All plugins can register endpoints (use unique path prefixes)
 
 Error Handling

src/ros2_medkit_fault_manager/CMakeLists.txt

@@ -192,12 +192,14 @@ if(BUILD_TESTING)
     TARGET test_integration
     TIMEOUT 60
   )
+  set_tests_properties(test_integration PROPERTIES LABELS "integration")
 
   add_launch_test(
     test/test_rosbag_integration.test.py
     TARGET test_rosbag_integration
     TIMEOUT 90
   )
+  set_tests_properties(test_rosbag_integration PROPERTIES LABELS "integration")
 endif()
 
 ament_package()

src/ros2_medkit_gateway/CMakeLists.txt

@@ -92,6 +92,10 @@ add_library(gateway_lib STATIC
   src/discovery/discovery_manager.cpp
   src/discovery/runtime_discovery.cpp
   src/discovery/hybrid_discovery.cpp
+  src/discovery/merge_pipeline.cpp
+  src/discovery/layers/manifest_layer.cpp
+  src/discovery/layers/runtime_layer.cpp
+  src/discovery/layers/plugin_layer.cpp
   # Discovery models (with .cpp serialization)
   src/discovery/models/app.cpp
   src/discovery/models/function.cpp
@@ -184,6 +188,7 @@ target_precompile_headers(gateway_lib PRIVATE
   <httplib.h>
   <tl/expected.hpp>
 )
+set_target_properties(gateway_lib PROPERTIES POSITION_INDEPENDENT_CODE ON)
 
 # Gateway node executable
 add_executable(gateway_node src/main.cpp)
@@ -325,6 +330,10 @@ if(BUILD_TESTING)
   ament_add_gtest(test_runtime_linker test/test_runtime_linker.cpp)
   target_link_libraries(test_runtime_linker gateway_lib)
 
+  # Add merge pipeline tests
+  ament_add_gtest(test_merge_pipeline test/test_merge_pipeline.cpp)
+  target_link_libraries(test_merge_pipeline gateway_lib)
+
   # Add capability builder tests
   ament_add_gtest(test_capability_builder test/test_capability_builder.cpp)
   target_link_libraries(test_capability_builder gateway_lib)
@@ -488,6 +497,8 @@ if(BUILD_TESTING)
       test_plugin_manager
       test_log_manager
       test_log_handlers
+      test_merge_pipeline
+      test_runtime_linker
     )
     foreach(_target ${_test_targets})
       target_compile_options(${_target} PRIVATE --coverage -O0 -g)