Add capacity exports for electricity, hydrogen, gas, and heat networks

[aaccensi]

Context

The hourly curve exports allow users to download the load per participant per hour for electricity, hydrogen, network gas, and heat networks. However, there was no way to download the installed and peak capacity of each participant. This set of PRs adds 4 new CSV exports, one per carrier group, with that information.

Implemented changes

• Added ParticipantCapacitiesCSVSerializer module with the capacity CSV logic (key, installed_capacity, peak_capacity per participant)
• Added MeritCapacitiesCSVSerializer (inherits MeritCSVSerializer) as a carrier-agnostic base for merit-order capacity exports
• Added ElectricityCSVSerializer and ElectricityCapacitiesCSVSerializer as named subclasses for the electricity hourly and capacity exports respectively
• Added HeatNetworkParticipantCapacitiesCSVSerializer combining lt/mt/ht into a single file, consistent with HeatNetworkCSVSerializer
• Added ReconciliationCapacitiesCSVSerializer (inherits ReconciliationCSVSerializer) for hydrogen and network gas capacity exports
• Exposed carrier on CausalityCurvesCSVSerializer::Adapter via attr_reader
• Added 4 new controller actions and routes: electricity_capacities, hydrogen_capacities, network_gas_capacities, heat_network_capacities
• Renamed existing hourly curve exports:
  • merit_order → electricity_profiles
  • heat_network → heat_network_profiles
  • hydrogen → hydrogen_profiles
  • network_gas → network_gas_profiles
• Kept old route names as backwards-compatible aliases
• Added spec for ParticipantCapacitiesCSVSerializer
• Removed deprecated production_parameters export (action, serializer, route, and spec)
• Added hydrogen_based_input_capacity fallback to Qernel::NodeApi::Cost::input_capacity and update the spec fixtures so node test don't break given this change

Related

Related

Set of 2 pull requests:

• etengine: Add capacity exports for electricity, hydrogen, gas, and heat networks #1767 [This one]
• etmodel: Add capacity exports for electricity, hydrogen, gas, and heat networks etmodel#4724
• documentation: Update curve export documentation to reflect renamed endpoints and ne… documentation#329

Checklist

• I have tested these changes
• I have updated documentation as needed
• I have tagged the relevant people for review

[noracato]

Super clean. Smartly solved with the ParticipantCapacitiesCSVSerializer module. Nice work!

[mabijkerk]

Shouldn't we just name this electricity_capacities?

[aaccensi]

I followed the internal naming convention of the existing curves we were creating a capacities file for, The "Electricity load curves" just happens to be named "merit_order" internally.
I see your point but not sure is worth it since then the right thing would be to also change the original curve file as well for consistency.

[mabijkerk]

Yes, I agree. I would be up for renaming them all for consistency:

Hourly download	Capacity downloads
electricity_profiles	electricity_capacities
network_gas_profiles	network_gas_capacities
hydrogen_profiles	hydrogen_capacities
heat_network_profiles	heat_network_capacities

What do you think @aaccensi @noracato? We should check implications throughout the model, for example for PyETM.

[aaccensi]

Does this mean it is safe to rename all the existing file names as show below? I seem to recall that one time I changed a data export file name, it had to be reverted so it would not affect existing workflows.

• merit_order.csv to electricity_profiles.csv
• heat_network.csv to heat_network_profiles.csv
• network_gas.csv to network_gas_profiles.csv
• hydrogen.csv to hydrogen_profiles.csv

[noracato]

It will break workflows, like you say @aaccensi if we replace the routes.

We can add a route instead, so it is backwards compatible for API users

get :merit_order,  to: 'curves#electricity_profiles' # backwards compatible
get :electricity_profiles,  to: 'curves#electricity_profiles',  as: :electricity_profiles_download

[mabijkerk]

Smart!

[mabijkerk]

Items to discuss:

• If a node has no explicit installed capacity set, it will return 0, whereas the peak capacity may be > 0. For the energy_hydrogen_steam_methane_reformer_dispatchable has hydrogen_output_capacity but no network_gas_input_capacity. This may be counterintuitive, how should we approach this?
• The agriculture_flexibility_p2h_hydrogen_electricity node shows up with both input and output in the electricity, probably because its type is flex, but it only consumes electricity. How should we approach this?
• Peak capacity can exceed installed capacity in some cases. @aaccensi needs to provide an example.

[aaccensi]

The controller actions and CSV filenames have been renamed to use consistent suffixes:

• merit_order → electricity_profiles
• heat_network → heat_network_profiles
• hydrogen → hydrogen_profiles
• network_gas → network_gas_profiles

Note: Old endpoints kept as backwards-compatible aliases.

[noracato]

Awesome. Let's add the new names to the docs as well, and inform @louispt1 to ask if there is any impact on PyETM

[aaccensi]

Awesome. Let's add the new names to the docs as well, and inform @louispt1 to ask if there is any impact on PyETM

If I do not recall wrongly, PyETM did pulled curve files by calling to /scenarios/{id}/curves/{curve_name}.csv with the curve_names defined somewhere in the excel file. That said, since we left the old routes pointing at the updated controller actions names they will work with both the old file curve_names and the new ones.
I reckon though that in order to keep pyetm up with this somewhere in the templates or documentation these curve_names shouls be updated right? What do you think @louispt1?

[louispt1]

Ooh I think some people will be quite happy with these changes (especially merit_order --> electricity_profiles). I will update the examples in the documentation for pyetm. There are some hardcoded aliases in pyetm that maps some of these carriers that will need to be updated. And the ongoing problem that new endpoints are not currently 'auto-discovered' by pyetm.

I will make the relevant pyetm changes. One question I have about the backwards compatible endpoints is how long we will support them for, or if we have some way of telling users or API clients that these endpoints will be deprecated after X time?

[aaccensi]

• Peak capacity can exceed installed capacity in some cases. @aaccensi needs to provide an example.

Explanation:

installed_capacity is the nameplate capacity — calculated from input_capacity * number_of_units * conversion, representing rated capacity. peak_capacity is the observed maximum hourly load from the actual simulation curve — the highest MW value seen in any single hour.

For storage nodes (like energy_heat_network_storage_ht_steam_hot_water), peak output can exceed installed capacity because the input_capacity used to calculate installed capacity may represent the charging side, while the peak discharge in the curve reflects a different operational limit. More generally, the two numbers measure different things and are not constrained to have one always exceed the other. It's meaningful data showing the node operated beyond its nominal installed capacity at peak — not a bug.

[mabijkerk]

Following discussion:

To do for @aaccensi:

• Data-export for each carrier installed capacity for input should refer to the eletricity_based_input_capacity, network_gas_based_input_capacity etc.
• This should be added network_gas_based_input_capacity
• This should fix the use case where the SMR does have installed hydrogen output capacity but it does not have network gas installed capacity
• It does not yet solve the use can of the network gas backup export having peak input capacity but no installed input capacity

To do for @mabijkerk:

• Test the new approach from @aaccensi
• Write down all remaining debts
• Flag this as a project within the reinvestment calculation model theme: should revise these scattered capacity methods into a structured separate node api module

[aaccensi]

The installed capacity calculation simply uses the existing Qernel::NodeApi::Cost::input_capacity which uses a simplistic approach and although works correctly for the vast majority of nodes, we know is not a totally correct method. The method causes some nodes to show zero (or incorrect) installed capacities on a few cases they should not (depending on the definitions on the node). Fixing this is postponed as it is not trivial to change how NodeApi does the capacity calculations.

[mabijkerk]

I would have preferred to wait with the merging of these PRs. There were still some things I would like to have checked first.