Skip to content
60 changes: 60 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,66 @@
:undoc-members:
```

### Stage Manifest Internals

```{eval-rst}
.. automodule:: microplex_us.pipelines.stage_manifest_types
:members:
:undoc-members:

.. automodule:: microplex_us.pipelines.stage_manifest_builder
:members:
:undoc-members:

.. automodule:: microplex_us.pipelines.stage_manifest_io
:members:
:undoc-members:

.. automodule:: microplex_us.pipelines.stage_status
:members:
:undoc-members:

.. automodule:: microplex_us.pipelines.stage_metrics
:members:
:undoc-members:

.. automodule:: microplex_us.pipelines.stage_data_flow
:members:
:undoc-members:

.. automodule:: microplex_us.pipelines.stage_policyengine_artifacts
:members:
:undoc-members:

.. automodule:: microplex_us.pipelines.stage_validation_evidence
:members:
:undoc-members:
```

## Stage artifacts

```{eval-rst}
.. automodule:: microplex_us.pipelines.stage_artifacts
:members:
:undoc-members:
```

## Conditional readiness

```{eval-rst}
.. automodule:: microplex_us.pipelines.stage_readiness
:members:
:undoc-members:
```

## Stage run writer

```{eval-rst}
.. automodule:: microplex_us.pipelines.stage_run
:members:
:undoc-members:
```

## Artifact helpers

```{eval-rst}
Expand Down
41 changes: 38 additions & 3 deletions docs/stage-contracts.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,31 @@ The canonical stage registry lives in
expected inputs, outputs, artifacts, diagnostics, validation placeholders, and
resume mode.

Saved artifact bundles now include a `stage_manifest.json` sidecar. This file is
the machine-readable saved-run overlay for the stage taxonomy. It records the
Saved artifact bundles now include a `stage_manifest.json` derived artifact. This
file is the machine-readable saved-run overlay for the stage taxonomy. It records the
canonical stages, status for the current run, artifact paths, diagnostics owned
by each stage, and the current resume posture.

Each saved bundle also includes typed per-stage output manifests at
`stage_artifacts/manifests/<stage_id>.json`. These manifests are written through
`USStageRunWriter`, which validates each stage as a whole instead of updating
individual manifest keys directly. The manifest files live outside each stage's
payload directory so they do not change the content hash of reloadable stage
artifacts.

The registry exposes two seam layers:

- `inputs` and `outputs` are structured stage resources. They identify artifact,
config, manifest, runtime, and external-data dependencies with explicit keys.
- `consumes` and `produces` remain short human-readable summaries for diagrams
and documentation.

Artifact `required` means required for a complete canonical saved bundle. It is
separate from `resume_role`, which says whether an existing artifact is useful
for diagnostics, manual replay, manual resume, or post-artifact validation.
Partial bundles can therefore still expose a valid replay boundary while the
manifest honestly reports that the complete publication bundle is incomplete.

## Legacy run-contract IDs

Older run-contract summaries and dashboard payloads used operational labels
Expand All @@ -34,7 +54,7 @@ boundary artifacts where the pipeline already has stable outputs:
- Stage 7: `calibrated_data.parquet`, `targets.json`, and
`stage_artifacts/07_calibration/calibration_summary.json`
- Stage 8: `policyengine_us.h5`
- Stage 9: validation and benchmark evidence sidecars
- Stage 9: validation and benchmark evidence artifacts

The Stage 4 artifact is the scaffold-projected seed before donor integration. It
is a diagnostic and manual replay boundary, not an automatic conditional resume
Expand All @@ -44,6 +64,21 @@ Conditional execution is intentionally not implemented yet. The stage manifest
and artifacts are designed to make that possible later without changing the
saved-run contract again.

## Artifact inventory and readiness

Saved bundles also expose two Stage 8 diagnostic artifacts:

- `stage_artifacts/artifact_inventory.json` lists canonical stage artifacts,
whether each path exists, whether it was referenced by the run manifest, its
resume role, size/file counts, and content hashes.
- `stage_artifacts/conditional_readiness.json` summarizes which stage outputs
are available for manual replay, manual resume, post-artifact evidence, or
diagnostics only.

These reports are advisory. They do not skip or rerun stages, and they do not
silently accept stale artifacts. If a requested config is supplied to the
readiness builder, config mismatches are reported as `must_rerun`.

## Validation hooks

Each stage contract includes concise validation descriptors. These describe the
Expand Down
61 changes: 61 additions & 0 deletions src/microplex_us/pipelines/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -257,12 +257,38 @@ def _exports(module: str, names: tuple[str, ...]) -> dict[str, str]:
(
"USPipelineStageContract",
"USStageArtifactContract",
"USStageResourceContract",
"USStageValidationContract",
"config_keys_for_us_pipeline_stage",
"default_us_pipeline_stage_contracts",
"get_us_stage_artifact_contract",
"get_us_pipeline_stage_contract",
"resolve_us_stage_artifact_contract_path",
"serialize_us_pipeline_stage_contracts",
),
),
**_exports(
"microplex_us.pipelines.stage_artifacts",
(
"USCalibratedStageArtifacts",
"USCandidateCalibrationReplayArtifacts",
"USCandidateStageArtifacts",
"USDatasetAssemblyArtifacts",
"USPolicyEngineEntityStageArtifacts",
"USSeedScaffoldStageArtifacts",
"USStageArtifactInventory",
"build_us_stage_artifact_inventory",
"load_us_calibrated_stage_artifacts",
"load_us_candidate_calibration_replay_artifacts",
"load_us_candidate_stage_artifacts",
"load_us_dataset_assembly_artifacts",
"load_us_policyengine_entity_stage_artifacts",
"load_us_seed_scaffold_stage_artifacts",
"load_us_stage_artifact_inventory",
"resolve_us_stage_artifact_path_checked",
"write_us_stage_artifact_inventory",
),
),
**_exports(
"microplex_us.pipelines.stage_manifest",
(
Expand All @@ -285,6 +311,41 @@ def _exports(module: str, names: tuple[str, ...]) -> dict[str, str]:
"write_us_validation_evidence_manifest",
),
),
**_exports(
"microplex_us.pipelines.stage_readiness",
(
"USConditionalReadinessReport",
"USConditionalReadinessStageRecord",
"build_us_conditional_readiness_report",
"build_us_stage_reuse_key",
"load_us_conditional_readiness_report",
"write_us_conditional_readiness_report",
),
),
**_exports(
"microplex_us.pipelines.stage_run",
(
"USAuxiliaryArtifact",
"USArtifactRef",
"USCalibrationOutputs",
"USDatasetAssemblyOutputs",
"USDiagnosticOutput",
"USDonorSynthesisOutputs",
"USPolicyEngineEntityOutputs",
"USRunProfileOutputs",
"USSeedScaffoldOutputs",
"USSourceLoadingOutputs",
"USSourcePlanningOutputs",
"USStageInputOverride",
"USStageOutputManifest",
"USStageRunWriter",
"USValidationBenchmarkingOutputs",
"build_us_stage_output_manifests_from_artifact_manifest",
"parse_us_stage_input_override",
"resolve_us_manifest_or_contract_artifact_path",
"write_us_stage_run_manifests_from_artifact_manifest",
),
),
**_exports(
"microplex_us.pipelines.summarize_pe_native_family_drilldown",
(
Expand Down
Loading
Loading