Benchmaking: observability large radial grids#1302
Benchmaking: observability large radial grids#1302Jerry-Jinfeng-Guo wants to merge 11 commits intofeature/meshed-observability-optimizationfrom
Conversation
Signed-off-by: Jerry Jinfeng Guo <jerry.jinfeng.guo@alliander.com>
Signed-off-by: Jerry Jinfeng Guo <jerry.jinfeng.guo@alliander.com>
Signed-off-by: Jerry Jinfeng Guo <jerry.jinfeng.guo@alliander.com>
Signed-off-by: Jerry Jinfeng Guo <jerry.jinfeng.guo@alliander.com>
Signed-off-by: Jerry Jinfeng Guo <jerry.jinfeng.guo@alliander.com>
Signed-off-by: Jerry Jinfeng Guo <jerry.jinfeng.guo@alliander.com>
There was a problem hiding this comment.
Pull request overview
Adds a standalone C++ benchmarking executable to compare radial vs meshed observability checks (including a scaling study), plus minor generator fixes to support benchmark scenarios.
Changes:
- Introduces
power_grid_model_observability_benchmarkexecutable with JSON, generated-grid, and scaling modes. - Adds an internal benchmark hook in
MainModelImpland a public accessor inMainModelto reach it. - Fixes sensor generation to resolve shunt node references by ID and guards transforms on empty inputs.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/benchmark_cpp/observability_benchmark.cpp | New benchmark driver (timing/memory, JSON parsing, scaling study). |
| tests/benchmark_cpp/fictional_grid_generator.hpp | Adds node lookup by ID and makes sensor generation more robust for empty inputs. |
| tests/benchmark_cpp/CMakeLists.txt | Adds a new benchmark executable, install rule, and convenience target. |
| power_grid_model_c/power_grid_model/include/power_grid_model/main_model_impl.hpp | Adds benchmark-only run_observability_benchmark entrypoint under a compile define. |
| power_grid_model_c/power_grid_model/include/power_grid_model/main_model.hpp | Exposes get_impl_for_benchmark() publicly. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| // Benchmark accessor - provides access to internal structures for observability benchmarking | ||
| Impl& get_impl_for_benchmark() { return impl(); } |
There was a problem hiding this comment.
get_impl_for_benchmark() is added to a public header and is always available, which exposes internal implementation details (Impl) to all library consumers and weakens encapsulation. Consider guarding this accessor with the same compile-time flag used for the benchmark (#ifdef POWER_GRID_MODEL_OBSERVABILITY_BENCHMARK), or moving this accessor behind a dedicated benchmark/test-only interface to avoid expanding the public API surface.
| // Benchmark accessor - provides access to internal structures for observability benchmarking | |
| Impl& get_impl_for_benchmark() { return impl(); } | |
| #ifdef POWER_GRID_MODEL_OBSERVABILITY_BENCHMARK | |
| // Benchmark accessor - provides access to internal structures for observability benchmarking | |
| Impl& get_impl_for_benchmark() { return impl(); } | |
| #endif |
| // Get actual node count from the generated grid | ||
| FictionalGridGenerator temp_gen; | ||
| temp_gen.generate_grid(grid_option); | ||
| Idx actual_nodes = static_cast<Idx>(temp_gen.input_data().node.size()); | ||
|
|
||
| // Store result | ||
| double radial_mean_us = radial_mean_ns / 1000.0; | ||
| double meshed_mean_us = meshed_mean_ns / 1000.0; |
There was a problem hiding this comment.
The scaling loop regenerates the entire grid a second time just to compute actual_nodes, which can dominate runtime for large configurations and distort benchmark throughput. A concrete fix is to have run_benchmark_on_generated_grid(...) return the actual node count (or the generator/input sizes) alongside the benchmark results so the scaling study can record actual_nodes without a second generation pass.
| // Get actual node count from the generated grid | |
| FictionalGridGenerator temp_gen; | |
| temp_gen.generate_grid(grid_option); | |
| Idx actual_nodes = static_cast<Idx>(temp_gen.input_data().node.size()); | |
| // Store result | |
| double radial_mean_us = radial_mean_ns / 1000.0; | |
| double meshed_mean_us = meshed_mean_ns / 1000.0; | |
| // Use configured approximate node count as the recorded node count | |
| Idx actual_nodes = static_cast<Idx>(config.approx_nodes); | |
| // Store result | |
| double radial_mean_us = radial_mean_ns / 1000.0; | |
| double meshed_mean_us = meshed_mean_ns / 1000.0; | |
| double meshed_mean_us = meshed_mean_ns / 1000.0; |
| // Memory measurement | ||
| size_t mem_before_radial = get_memory_usage_kb(); | ||
| size_t mem_before_meshed = 0; |
There was a problem hiding this comment.
The memory deltas for radial vs meshed are not comparable because the meshed baseline (mem_before_meshed) is captured after the radial run has already potentially allocated/cached memory. This biases meshed_mem_delta downward (and can make overhead misleading). To make the comparison meaningful, measure both algorithms from the same baseline (e.g., snapshot once before both runs and compute peak/RSS deltas per run), or compute per-algorithm peak memory independently (your get_peak_memory_kb() helper suggests that direction).
| // Memory measurement | |
| size_t mem_before_radial = get_memory_usage_kb(); | |
| size_t mem_before_meshed = 0; | |
| // Memory measurement: use a single baseline for both algorithms | |
| size_t mem_baseline = get_memory_usage_kb(); | |
| size_t mem_before_radial = mem_baseline; | |
| size_t mem_before_meshed = mem_baseline; |
| auto end = high_resolution_clock::now(); | ||
| radial_times.push_back(duration_cast<nanoseconds>(end - start)); | ||
| } | ||
| mem_after_radial = get_memory_usage_kb(); |
There was a problem hiding this comment.
The memory deltas for radial vs meshed are not comparable because the meshed baseline (mem_before_meshed) is captured after the radial run has already potentially allocated/cached memory. This biases meshed_mem_delta downward (and can make overhead misleading). To make the comparison meaningful, measure both algorithms from the same baseline (e.g., snapshot once before both runs and compute peak/RSS deltas per run), or compute per-algorithm peak memory independently (your get_peak_memory_kb() helper suggests that direction).
|
|
||
| // Benchmark meshed algorithm | ||
| std::cout << "Benchmarking meshed algorithm...\n"; | ||
| mem_before_meshed = get_memory_usage_kb(); |
There was a problem hiding this comment.
The memory deltas for radial vs meshed are not comparable because the meshed baseline (mem_before_meshed) is captured after the radial run has already potentially allocated/cached memory. This biases meshed_mem_delta downward (and can make overhead misleading). To make the comparison meaningful, measure both algorithms from the same baseline (e.g., snapshot once before both runs and compute peak/RSS deltas per run), or compute per-algorithm peak memory independently (your get_peak_memory_kb() helper suggests that direction).
|
|
||
| // Calculate memory usage | ||
| size_t radial_mem_delta = (mem_after_radial > mem_before_radial) ? (mem_after_radial - mem_before_radial) : 0; | ||
| size_t meshed_mem_delta = (mem_after_meshed > mem_before_meshed) ? (mem_after_meshed - mem_before_meshed) : 0; |
There was a problem hiding this comment.
The memory deltas for radial vs meshed are not comparable because the meshed baseline (mem_before_meshed) is captured after the radial run has already potentially allocated/cached memory. This biases meshed_mem_delta downward (and can make overhead misleading). To make the comparison meaningful, measure both algorithms from the same baseline (e.g., snapshot once before both runs and compute peak/RSS deltas per run), or compute per-algorithm peak memory independently (your get_peak_memory_kb() helper suggests that direction).
| size_t meshed_mem_delta = (mem_after_meshed > mem_before_meshed) ? (mem_after_meshed - mem_before_meshed) : 0; | |
| size_t meshed_mem_delta = (mem_after_meshed > mem_before_radial) ? (mem_after_meshed - mem_before_radial) : 0; |
| auto const* node_ptr = find_node_by_id(shunt.node); | ||
| if (!node_ptr) { | ||
| throw std::runtime_error("Node not found for shunt"); | ||
| } |
There was a problem hiding this comment.
The thrown error message is too generic to debug failing inputs (it omits the shunt/node identifiers). Include at least the shunt.node value (and ideally the shunt id if available) in the exception message so failures can be traced to specific input records.
| benchmark_dir = argv[2]; | ||
| } | ||
| if (argc > 3) { | ||
| n_iterations = std::stoll(argv[3]); |
There was a problem hiding this comment.
Command-line parsing uses std::stoll without handling std::invalid_argument / std::out_of_range, so malformed inputs will terminate the program with an uncaught exception. Wrap argument parsing in a try/catch that prints the usage (or switch to std::from_chars and validate), returning a non-zero exit code on parse errors.
| n_iterations = std::stoll(argv[2]); | ||
| } | ||
| if (argc > 3) { | ||
| n_mv_feeder = std::stoll(argv[3]); |
There was a problem hiding this comment.
Command-line parsing uses std::stoll without handling std::invalid_argument / std::out_of_range, so malformed inputs will terminate the program with an uncaught exception. Wrap argument parsing in a try/catch that prints the usage (or switch to std::from_chars and validate), returning a non-zero exit code on parse errors.
| n_iterations = std::stoll(argv[2]); | ||
| } | ||
| if (argc > 3) { | ||
| max_nodes = std::stoll(argv[3]); |
There was a problem hiding this comment.
Command-line parsing uses std::stoll without handling std::invalid_argument / std::out_of_range, so malformed inputs will terminate the program with an uncaught exception. Wrap argument parsing in a try/catch that prints the usage (or switch to std::from_chars and validate), returning a non-zero exit code on parse errors.
…/benchmaking-observability-large-radial-grids
…/benchmaking-observability-large-radial-grids
…/benchmaking-observability-large-radial-grids
…/benchmaking-observability-large-radial-grids
…/benchmaking-observability-large-radial-grids
|
1 participant
!!!DO NOT MERGE!!!
This branch contains the benchmarking between radial and meshed observability check. Making a PR for archival purpose. Do not merge.
The behavior comparison (and the view of) meshed against radial is studied via scaling experiments from (+1 node) 10 nodes all the way to 10000 nodes. The following are the overview.