2561 hi l2 update how full spin helio frame maps are produced #2689

subagonsouth · 2026-02-10T16:08:52Z

Change Summary

Overview

Sorry for how big this got...
This PR updates Hi L2 processing to properly handle full-spin heliocentric frame maps by generating ram and anti-ram maps separately, applying Compton-Getting corrections to each independently, and then combining them.

Updated Files

imap_processing/ena_maps/ena_maps.py
imap_processing/tests/ena_maps/test_ena_maps.py
imap_processing/ena_maps/utils/corrections.py
- Added keyword input bool to keep systematic error from getting updated if not desired. Hi does not want systematic uncertainty to go through the flux correction.
imap_processing/tests/ena_maps/test_corrections.py
imap_processing/hi/hi_l2.py
- create_sky_map_from_psets() now returns dict[str, RectangularSkyMap] instead of a single map
  - For helio-frame (hf) full-spin maps: returns {"ram": ..., "anti": ...}
  - For all other maps: returns {spin_phase: ...} (single entry)
- Added new combine_maps() function to merge ram and anti-ram maps after CG correction
- Added cleanup_intermediate_variables() to remove temporary calculation variables (bg_rate, energy_sc, ena_signal_rates, ena_signal_rate_stat_unc) from final output
imap_processing/tests/hi/test_hi_l2.py

Closes: #2561

Copilot

Pull request overview

This PR updates IMAP-Hi L2 processing to correctly generate full-spin heliocentric-frame (HF) sky maps by building separate ram and anti-ram maps, applying Compton-Getting (CG) corrections independently, and then combining them.

Changes:

Refactors Hi L2 map generation so HF full-spin maps are produced as separate {"ram": ..., "anti": ...} maps and combined after CG correction.
Adds update_sys_err flag to interpolate_map_flux_to_helio_frame so Hi can avoid updating systematic uncertainty during flux interpolation.
Updates naming from bg_rates/bg_rates_unc to bg_rate/bg_rate_sys_err and expands unit/integration test coverage for the new pipeline pieces.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`imap_processing/hi/hi_l2.py`	Refactors Hi L2 pipeline into multi-map creation + per-map processing + combine step; adds cleanup helpers and map combination logic.
`imap_processing/ena_maps/utils/corrections.py`	Adds `update_sys_err` parameter to optionally skip updating systematic error during heliocentric-frame interpolation.
`imap_processing/ena_maps/ena_maps.py`	Updates HiPointingSet variable rename mapping (`bg_rate`, `bg_rate_sys_err`) and reduces az/el update logging verbosity.
`imap_processing/tests/hi/test_hi_l2.py`	Updates integration/unit tests for new APIs and adds extensive tests for PSET processing, rate/intensity pipeline, cleanup, and combining maps.
`imap_processing/tests/ena_maps/test_corrections.py`	Adds a test ensuring systematic uncertainty is unchanged when `update_sys_err=False`.
`imap_processing/tests/ena_maps/test_ena_maps.py`	Updates expected HiPointingSet renamed variable names.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

imap_processing/hi/hi_l2.py

Copilot · 2026-02-10T16:14:47Z

imap_processing/hi/hi_l2.py

+        for spin_phase, map in output_maps.items():
+            # Project (bin) the PSET variables into the map pixels
+            directional_mask = get_pset_directional_mask(pset_processed, spin_phase)
+            map.project_pset_values_to_map(
+                hi_pset, list(vars_to_bin), pset_valid_mask=directional_mask


Minor: using map as a loop variable shadows the built-in map() function (also used in the isinstance(map, ...) comprehension earlier). Renaming it (e.g., sky_map) would reduce confusion during debugging.

greglucas · 2026-02-10T18:05:51Z

imap_processing/hi/hi_l2.py

+            for var in vars_to_exposure_time_average:
+                map.data_1d[var] /= map.data_1d["exposure_factor"]


Can you do this with xarray's list indexing syntax instead?

Suggested change

for var in vars_to_exposure_time_average:

map.data_1d[var] /= map.data_1d["exposure_factor"]

map.data_1d[vars_to_exposure_time_average] /= map.data_1d["exposure_factor"]

Oh, cool... I didn't know you could do that. I'll test it out.

greglucas · 2026-02-10T18:08:21Z

imap_processing/hi/hi_l2.py

+    # If we are making a full-spin, helio-frame map, we need to make one ram
+    # and one anti-ram map that get combined at the final L2 step.


This reminds me of Lo needing to make an extra "Oxygen" map for sputtering calculations and needing to carry it through to some extent as well. I'm wondering if this dictionary-like mapping should be carried over to the Lo calculations as well to keep things consistent or if there is another way of organizing this multi-map projections work across all instruments.

Just noting this as a comment, nothing to do here.

Yeah, good point. I was toying with the idea of generating a dataset that had yet another dimension added for "spin_phase" which could later be combined. I think that some of these things would come easier if we did a refactor of the mapping code to leverage the xarray custom accessor functions.

greglucas · 2026-02-10T18:17:33Z

imap_processing/hi/hi_l2.py

+    ram_ds = sky_maps["ram"].data_1d
+    anti_ds = sky_maps["anti"].data_1d


I'm not sure if this would be useful to have more general, but I think a lot of the math below could be generalized to something where you are working with a list of datasets input (not just ram/anti, but maybe combining multiple 7-day maps or something like that in the future...)

datasets = [ds.data_1d for ds in sky_maps.values()] weights = [xr.where(ds["ena_intensity_stat_uncert"] > 0, 1 / ds["ena_intensity_stat_uncert"] ** 2, 0) for ds in datasets] ...

Again, not needed here and I'm just writing it down for a quick note. It should be another ticket if this would even be useful.

…te sys_err vars or not. Set update_sys_err to false for Hi L2

…ning into full-spin helio-frame map Return dictionary of SkyMaps from create_sky_map_from_psets Loop over SkyMaps for when calculating rates and intensities Add function for combining data in multiple SkyMaps Add function for cleaning up intermediate variables

Add test coverage for NotImplementedError

Simplify two tests by using existing fixture

subagonsouth requested a review from Copilot February 10, 2026 16:08

subagonsouth self-assigned this Feb 10, 2026

subagonsouth added this to IMAP Feb 10, 2026

Copilot started reviewing on behalf of subagonsouth February 10, 2026 16:09 View session

Copilot AI reviewed Feb 10, 2026

View reviewed changes

subagonsouth force-pushed the 2561-hi-l2---update-how-full-spin-helio-frame-maps-are-produced branch from 5505752 to 85e321d Compare February 10, 2026 17:16

subagonsouth requested review from greglucas and lacoak21 February 10, 2026 18:02

greglucas approved these changes Feb 10, 2026

View reviewed changes

subagonsouth added 6 commits February 10, 2026 15:19

Change log message to debug

4663ee1

Add option in interpolate_map_flux_to_helio_frame for whether to upda…

b898ac9

…te sys_err vars or not. Set update_sys_err to false for Hi L2

Fix mypy issue

b3fa024

Add test coverage for NotImplementedError

Temporary fix for handling of obs_date

4dbf00d

Systematic error should be combined using exposure factor weighting

b6f1076

Simplify two tests by using existing fixture

subagonsouth force-pushed the 2561-hi-l2---update-how-full-spin-helio-frame-maps-are-produced branch from 2841a54 to b6f1076 Compare February 10, 2026 22:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2561 hi l2 update how full spin helio frame maps are produced #2689

2561 hi l2 update how full spin helio frame maps are produced #2689

subagonsouth commented Feb 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 10, 2026

Uh oh!

greglucas Feb 10, 2026

Uh oh!

subagonsouth Feb 10, 2026

Uh oh!

greglucas Feb 10, 2026

Uh oh!

subagonsouth Feb 10, 2026

Uh oh!

greglucas Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		for var in vars_to_exposure_time_average:
		map.data_1d[var] /= map.data_1d["exposure_factor"]

	for var in vars_to_exposure_time_average:
	map.data_1d[var] /= map.data_1d["exposure_factor"]
	map.data_1d[vars_to_exposure_time_average] /= map.data_1d["exposure_factor"]

		# If we are making a full-spin, helio-frame map, we need to make one ram
		# and one anti-ram map that get combined at the final L2 step.

		ram_ds = sky_maps["ram"].data_1d
		anti_ds = sky_maps["anti"].data_1d

2561 hi l2 update how full spin helio frame maps are produced #2689

Are you sure you want to change the base?

2561 hi l2 update how full spin helio frame maps are produced #2689

Conversation

subagonsouth commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Summary

Overview

Updated Files

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

greglucas Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

subagonsouth Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

greglucas Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

subagonsouth Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

greglucas Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

subagonsouth commented Feb 10, 2026 •

edited

Loading