feat(UTILS): Add dfextensions packages and perfmonitor#2180

Closed

miranov25 wants to merge 132 commits intoAliceO2Group:masterfrom

miranov25:feature/groupby-optimization

Commits on May 4, 2025

make diff of time series
miranov25
committed

Commits on May 29, 2025

`AliasDataFrame` is a small utility that extends `pandas.DataFrame` functionality by enabling:
miranov25
committed

Commits on May 31, 2025

adding perfmonitor
miranov25
committed
adding PerfromanceLogger extracted from calibration code
miranov25
committed
supressing linter warning
miranov25
committed

Commits on Jun 1, 2025

Add dtype support and alias dependency graph to AliasDataFrame
miranov25
committed
Add support for dtype persistence and alias filtering in save/load
miranov25
committed
Save aliases directly to pyarrow metadata
miranov25
committed

Commits on Jun 2, 2025

add FormulaLinearModel.py used for the dEdx and distortion calibration
miranov25
committed
add FormulaLinearModel.py used for the dEdx and distortion calibration
miranov25
committed
special treatment for constants - should be enver materialized but used
miranov25
committed
special treatment for constants
miranov25
committed
special treatment for constants
miranov25
committed

Commits on Jun 3, 2025

`Add ROOT SetAlias export and Python-to-ROOT AST translation for aliases`
miranov25
committed

Commits on Jun 4, 2025

Add dependency-aware alias materialization with optional cleanup and verbosity
miranov25
committed

Commits on Jun 9, 2025

Extended usnit test for the sub_frames
miranov25
committed
Add extended unit tests for AliasDataFrame including lazy join and error handling
miranov25
committed
fixed - Circular dependency detection
miranov25
committed
fixing all unit test - except oth the automatic materialization
miranov25
committed
fixing automatic materialization test + working in the distrtion calibration test
miranov25
committed
fixing circular depndency test - all test are OK now
miranov25
committed

Commits on Jun 10, 2025

adding unit test for the export_import tree
miranov25
committed
add failing test for export/import of the subframes
miranov25
committed
make test_export_tree_read_tree_with_subframe already OK
miranov25
committed

Commits on Jun 11, 2025

adding metadata to all trees
miranov25
committed
Updated documentation
miranov25
committed
AliasDataFrame: add index-based subframe join and robust error handling
miranov25
committed

Commits on Jun 12, 2025

Commits on Jun 13, 2025

* removing pylint warning
miranov25
committed
* adding __init__.py
miranov25
committed
* adding protection for infinite recursion
miranov25
committed
pylint fix
miranov25
committed

Commits on Jun 23, 2025

adding test for the logger
miranov25
committed

Commits on Jun 25, 2025

adding conversions to the function list
miranov25
committed

Commits on Jun 27, 2025

adding chunksize and compression as argument
miranov25
committed
adding chunksize and compression as argument
miranov25
committed

Commits on Aug 14, 2025

adding df drawing interface similar to the tree::Draw
miranov25
committed

Commits on Oct 11, 2025

docs(quantile_fit_nd): add v3.1 Δq-centered ND quantile fitting spec
miranov25
committed
Commit latest working version of AliasDataFrame
miranov25
committed
Commit latest working version of perfoemance_logger.py
miranov25
committed
Commit latest working version of groupby_regression.py
miranov25
committed
feat(DataFrameUtils): Enhance docstrings and error handling for scatter plots
miranov25
committed
feat(dfextensions): add ND quantile fitting (Δq-centered) + tests & bench
miranov25
committed
test(dfextensions): fix quantile ND tests vs synthetic truth; add robust edge expectations
miranov25
committed
fix(quantile_fit_nd): exclude q_center from nuisance axes; silence single-groupby warning
miranov25
committed
fix(dfextensions/quantile_fit_nd): evaluator axis bug + window-local b_min + stable inversion
miranov25
committed
tests(quantile_fit_nd): snapshot pre-fix state with rich diagnostics (exclude IDE files)
miranov25
committed
fix(quantile_fit_nd): do not floor degenerate Δq windows; keep NaN and record reason
miranov25
committed
tests(quantile_fit_nd): handle Poisson via randomized PIT pre-processing
miranov25
committed
docs(quantile_fit_nd): add Discrete Inputs policy and utilities
miranov25
committed
bench(quantile_fit_nd): correct scaling assertions — α_b≈−0.5, α_rt≈0.0
miranov25
committed
docs(quantile_fit_nd): add contextLLM.md (cold-start guide + policies)
miranov25
committed
docs(quantile_fit_nd): add contextLLM.md (cold-start guide + policies)
miranov25
committed
Forgottend commit of refernce test and bench log
miranov25
committed

Commits on Oct 22, 2025

Commits on Oct 23, 2025

feat(bench): integrate class-level diagnostics summary into benchmark report
miranov25
committed
docs(restartContext): record diagnostics integration and real-data validation
miranov25
committed
docs(restartContext): record diagnostics integration and real-data validation
miranov25
committed
use faster compression by default
miranov25
committed
removed accidentally comitted code
miranov25
committed
feat(groupby_regression): add make_parallel_fit_fast (Phase 3 NumPy baseline)
miranov25
committed
feat(groupby): Phase 3 v4 (Numba) — 33–36× faster than v2 on tiny groups
miranov25
committed

Commits on Oct 24, 2025

feat(groupby): Phase 3 v4 (Numba) — 33–36× faster than v2 on tiny groups
miranov25
committed
feat(groupby_regression): finalize v4 diagnostics + 200× speedup
miranov25
committed
test(v2/v3/v4): add verbose diagnostics for multi-target layout
miranov25
committed

Commits on Oct 25, 2025

remove junk files
miranov25
committed
feat: create groupby_regression package structure
miranov25
committed
refactor: move files to package structure (preserve history)
miranov25
committed
refactor: update imports for package structure
miranov25
committed
test: add cross-validation tests with realistic tolerances
miranov25
committed
fix: handle suffix in benchmark column names
miranov25
committed
bench: limit quick mode to 2k groups, add performance warnings
miranov25
committed
fix: remove relative import in fallback path (bench_groupby_regression.py)
miranov25
committed
benchmarks: add optimized-only benchmark for GroupByRegressor (v2/v3/v4 engines)
miranov25
committed
bench: add optimized-only benchmark (v2/v3/v4)
miranov25
committed
bench: add optimized-only benchmark (v2/v3/v4)
miranov25
committed
bench: add optimized benchmark + visualization
miranov25
committed
cleanup: remove .bak files from repository
miranov25
committed
add: .gitignore for Python project
miranov25
committed
cleanup: remove diff.txt and prevent re-adding
miranov25
committed
cleanup: remove diff.txt and prevent re-adding
miranov25
committed
cleanup: remove diff.txt and prevent re-adding
miranov25
committed
refactor: remove obsolete optimized/ directory
miranov25
committed
refactor: complete migration to new package structure
miranov25
committed
rm diff.txt
miranov25
committed
cleanup: finalize package structure cleanup
miranov25
committed

Commits on Oct 26, 2025

docs: finalize README.md for v2.0.0 release
miranov25
committed
docs: finalize README.md for v2.0.0 release
miranov25
committed

Commits on Oct 27, 2025

docs: apply Section 1 corrections (scope, measurements)
miranov25
committed
docs: Section 2 cleanup and focus
miranov25
committed
docs: Section 5 accuracy corrections
miranov25
committed
git add docs/SLIDING_WINDOW_SPEC_DRAFT.md
miranov25
committed
Complete Section 6 specification with all reviewer feedback
miranov25
committed

Commits on Oct 28, 2025

fixing bencmark path
miranov25
committed
docs: Add Phase 7 implementation plan with synthetic data spec
miranov25
committed
test version 1
miranov25
committed
cd ~/alicesw/O2DPG/UTILS/dfextensions/groupby_regression
miranov25
committed
feat: Add realistic TPC distortion synthetic data and validation
miranov25
committed

Commits on Nov 8, 2025

cd ~/alicesw/O2DPG/UTILS/dfextensions/groupby_regression
miranov25
committed

Commits on Nov 9, 2025

Add column compression support to AliasDataFrame
miranov25
committed
Fix precision measurement in AliasDataFrame compression
miranov25
committed
Add .pylintrc to suppress non-functional pylint warningsRetry
miranov25
committed
Add bidirectional atan2/arctan2 support for ROOT compatibility
miranov25
committed
Remove IDE and system files from tracking, update .gitignore
miranov25
committed
Remove IDE and system files from tracking, update .gitignore
miranov25
committed
Remove .idea/ directory from git tracking
miranov25
committed
Remove .idea/ from git tracking and add to .gitignore
miranov25
committed

Commits on Nov 10, 2025

Add selective compression mode (Pattern 2) to AliasDataFrame
miranov25
committed
Refactor: Move AliasDataFrame to subdirectory
miranov25
committed
"Refactor: Move AliasDataFrame to subdirectory
miranov25
committed
Docs: Organize documentation structure
miranov25
committed
Style: Fix pylint issues
miranov25
committed
Style: Fix pylint issues in AliasDataFrame
miranov25
committed
Style: Fix pylint issues in groupby_regression
miranov25
committed
Style: Verify pylint scores in quantile_fit_nd All files already ≥9.0 ✅
miranov25
committed
Refactor: Reorganize root utilities into subdirectories
miranov25
committed
Style: Fix pylint issues in perfmonitor
miranov25
committed
Fix: Update imports after reorganization
miranov25
committed
fix: Resolve 4 pylint errors
miranov25
committed
fix: Resolve 4 pylint CI errors
miranov25
committed
fix: Skip aspirational formula validation test
miranov25
committed
fix: Skip formula validation tests and resolve pylint errors
miranov25
committed