Skip to content

Add pre-calibration eCPS export schema checks#138

Merged
MaxGhenis merged 1 commit into
mainfrom
codex/ecps-derived-export-fields-20260601
Jun 1, 2026
Merged

Add pre-calibration eCPS export schema checks#138
MaxGhenis merged 1 commit into
mainfrom
codex/ecps-derived-export-fields-20260601

Conversation

@MaxGhenis
Copy link
Copy Markdown
Contributor

Summary

  • derive the five eCPS-persisted inputs that the Gate-1 CPS-spine artifact was missing (fsla_overtime_premium, has_itin, has_tin, in_nyc, meets_ssi_disability_criteria)
  • add a schema-only export-column resolver so saved PE entity tables can be checked before calibration/materialization
  • add --entity-tables and direct-override support to microplex-us-check-export-columns
  • add an optional rebuild checkpoint flag to fail the pipeline at the post-imputation/pre-calibration stage when the eCPS export contract would fail

Validation

  • uv run --python 3.13 --extra dev --extra policyengine ruff check src/microplex_us/__init__.py src/microplex_us/policyengine/__init__.py src/microplex_us/policyengine/us.py src/microplex_us/pipelines/check_export_columns.py src/microplex_us/pipelines/us.py src/microplex_us/pipelines/pe_us_data_rebuild_checkpoint.py tests/policyengine/test_us.py tests/pipelines/test_check_export_columns.py
  • uv run --python 3.13 --extra dev --extra policyengine ruff format --check src/microplex_us/__init__.py src/microplex_us/policyengine/__init__.py src/microplex_us/policyengine/us.py src/microplex_us/pipelines/check_export_columns.py src/microplex_us/pipelines/us.py src/microplex_us/pipelines/pe_us_data_rebuild_checkpoint.py tests/policyengine/test_us.py tests/pipelines/test_check_export_columns.py
  • uv run --python 3.13 --extra dev --extra policyengine pytest tests/policyengine/test_us.py tests/pipelines/test_check_export_columns.py -q
  • Gate-1 post-imputation checkpoint schema check: 246/246 required columns present, 0 forbidden columns present
  • Gate-1 sidecar H5 export check: 246/246 required columns present, 0 forbidden columns present

Gate-1 comparison note

The fixed export surface passes the contract, but MP does not yet beat latest local eCPS on the primary sound comparison. Matched 41,314-household symmetric-refit holdout comparison against latest local eCPS gives MP loss 0.4294876907206762 vs eCPS 0.1726525197190867; eCPS refit recovery passed.

@MaxGhenis MaxGhenis merged commit 5bba3e1 into main Jun 1, 2026
5 checks passed
@MaxGhenis MaxGhenis deleted the codex/ecps-derived-export-fields-20260601 branch June 1, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant