Fix SPM and family overfragmentation in PE export

[MaxGhenis]

Context

After PR #79, Microplex tax-unit construction is close to the eCPS structural reference, but the sound eCPS replacement comparison still shows large losses. The next structural mismatch is SPM/family fragmentation in the PolicyEngine export.

Fresh matched-N comparison artifact after PR #79:

/Users/maxghenis/CosilicoAI/microplex-us/artifacts/small_asec_acs100k_household_coherent_20260529/sound_ecps_replacement_comparison/sound_ecps_replacement_comparison.json

Key result:

• candidate refit loss: 3.7243506116660963
• eCPS refit loss: 0.1726525197190867
• candidate holdout loss: 0.5319674877579285
• eCPS holdout loss: 0.02754433292858976

The tax-unit structure is now reasonable:

• candidate matched tax units: 54,034 on 41,314 households (1.308/HH)
• eCPS tax units: 55,264 on 41,314 households (1.338/HH)

But SPM/family structures remain overfragmented:

• candidate matched SPM units: 65,905 on 41,314 households (1.595/HH)
• eCPS SPM units: 43,134 on 41,314 households (1.044/HH)
• candidate matched families: 65,905 (1.595/HH)
• eCPS families: 46,222 (1.119/HH)

Current root cause:

• _assign_family_and_spm_units uses the same fallback split for family and SPM.
• It puts relationship {0, 1, 2} in one primary family/SPM unit and assigns every other person to their own family/SPM unit.
• The calibrated source parquet already has family_id and spm_unit_id with the same inflated count (159,331 on 100,000 households), so simply preserving those columns is not enough.

Current mitigation

PR #80 changes only the SPM fallback to one SPM unit per household while keeping the current family split unchanged:

#80

Lightweight structural probe with PR #80 logic:

/Users/maxghenis/CosilicoAI/microplex-us/artifacts/small_asec_acs100k_household_spm_20260529/structural_probe.json

• households: 100,000
• persons: 245,714
• tax units: 130,980 (1.3098/HH)
• SPM units: 100,000 (1.0/HH)
• families: 159,331 (1.59331/HH), intentionally unchanged

Desired direction

Do not hard-code eCPS structure as the target. Use Microplex architecture to construct coherent relational units from source relationships and donor structure:

• SPM units should be household-coherent unless richer SPM relationship detail supports a split.
• Family units need a separate, data-driven rule; current family=primary-plus-singletons is likely too fragmented.
• Preserve diagnostics in sidecars/gates, not in the model H5.
• Add structural gates for SPM/family units per household and singleton-unit shares, calibrated against source distributions and policy semantics rather than only eCPS.

Acceptance criteria

• Small ASEC+ACS100k PE export has plausible SPM/family unit counts and no impossible entity memberships.
• A sidecar reports household/person/tax-unit/SPM/family/marital counts, per-household ratios, singleton shares, and cross-household ID violations.
• The sound matched-N symmetric-refit eCPS comparison is rerun after SPM/family changes.
• The comparison report breaks out SNAP, Census/SPM poverty, IRS filing-status-sensitive cells, and protected target families so we can tell whether the structural change improved the intended surfaces.

[MaxGhenis]

PR #80 is merged: #80

Full matched-N symmetric comparison after household-level SPM fallback:

/Users/maxghenis/CosilicoAI/microplex-us/artifacts/small_asec_acs100k_household_spm_20260529/sound_ecps_replacement_comparison/sound_ecps_replacement_comparison.json

Summary versus the previous household-coherent tax-unit build:

• Candidate final native loss improved from 3.7243506116660963 to 3.3933854903630642.
• Candidate train loss improved from 3.1923831254826487 to 2.88954251907915.
• Candidate holdout loss improved from 0.5319674877579285 to 0.5038429754602355.
• eCPS still wins after symmetric refit: 0.16459032249699052, so this is progress but not replacement-ready.
• Matched-N / symmetric-refit / scoring-objective contract remains satisfied; n_targets_kept=2818.

Largest intended improvements:

• national_census_other candidate contribution improved by -0.17343410223592154 (0.5141781655373558 -> 0.3407440633014343).
• state_snap_households improved by -0.09879537248763577 (0.15583634511701627 -> 0.0570409726293805).
• state_snap_cost improved by -0.08619092423986258 (0.10615087892225486 -> 0.019959954682392277).
• Protected snap loss improved from 9.441313821487866e-05 to 2.7512328839444265e-05.

Regressions to track next:

• national_irs_other worsened slightly (+0.026715437186476176 candidate contribution).
• state_agi_distribution worsened slightly (+0.003468346222308516).

The remaining structural mismatch is family/marital construction, not SPM. PR #80 intentionally left families at 159,331 on 100,000 households (1.59331/HH) while setting SPM units to 100,000 (1.0/HH). Next work should make family units more data-driven and add the sidecar/gate described in this issue.

[MaxGhenis]

Follow-up PR: #83

It uses CPS family_relationship as a family-membership signal for PE family-unit construction while keeping SPM and tax-unit construction unchanged.

Measured on the existing calibrated small ASEC+ACS100k artifact, without recalibration:

• families: 159,331 -> 109,234
• families per household: 1.59331 -> 1.09234
• family singleton share: 0.6567 -> 0.3795
• cross-household family count: 0
• tax units unchanged: 130,980
• SPM units unchanged: 100,000
• marital units unchanged: 220,843
• filing-status counts unchanged

Probe artifact: /Users/maxghenis/CosilicoAI/microplex-us/artifacts/small_asec_acs100k_family_relationship_units_20260529/structural_probe.json

Caveat: this fixes the family overfragmentation mechanism but may now be slightly below eCPS family count, so the next validation should rematerialize the H5 and rerun the sound matched-N eCPS comparison before marking the replacement path improved on loss.

[MaxGhenis]

Validation after PR #83 merge on current main (ccc9939):

Artifacts:

• Rematerialized H5: /Users/maxghenis/CosilicoAI/microplex-us/artifacts/small_asec_acs100k_family_relationship_units_20260529/materialized_current_main/policyengine_us.h5
• Sound comparison JSON: /Users/maxghenis/CosilicoAI/microplex-us/artifacts/small_asec_acs100k_family_relationship_units_20260529/sound_ecps_replacement_comparison/sound_ecps_replacement_comparison.json

Structural comparison, matched to 41,314 households:

• Microplex tax units/HH: 1.3079; eCPS: 1.3377
• Microplex SPM units/HH: 1.0000; eCPS: 1.0441
• Microplex families/HH: 1.0928; eCPS: 1.1188
• Microplex family singleton share: 0.3787; eCPS: 0.4148
• Cross-household family units: 0 for both

Loss result: unchanged from the post-SPM comparison.

• Microplex final native loss: 3.3933854903630642
• eCPS final native loss: 0.16459032249699052
• Microplex train/holdout: 2.88954251907915 / 0.5038429754602355
• eCPS train/holdout: 0.1372832454307687 / 0.027307076904983546

Interpretation: PR #83 fixes the family overfragmentation mechanism and puts family counts near eCPS, but current PE-native broad loss is dominated by other target families and does not move. Next replacement blocker is therefore not structural family/SPM coherence; it is the large non-family target loss, especially national_irs_other and related target-definition/calibration issues.