[WIP] release(4.0.0): OpenSTEF 4.0.0 tracking branch.#637
Open
[WIP] release(4.0.0): OpenSTEF 4.0.0 tracking branch.#637
Conversation
Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* feature(STEF-2297): Initial commit. Cleared OpenSTEF 3.0 code. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2297): Updated readme to exclude 3.0 specific parts. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2297): Created directory structure for the 4.0. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2297): Set python version to 3.12 Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* feature(STEF-2297): Initial commit. Cleared OpenSTEF 3.0 code. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2297): Updated readme to exclude 3.0 specific parts. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2297): Created directory structure for the 4.0. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Configured Reuse tool. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Created reuse fix tool. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Added initial ruff config. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Configured ruff, poe, pytest and pyright. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Configured pyproject-fmt. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Added example scripts for ci testing. Added example integration test. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Added support for doctests. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* feature(STEF-2297): Initial commit. Cleared OpenSTEF 3.0 code. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2297): Updated readme to exclude 3.0 specific parts. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2297): Created directory structure for the 4.0. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Configured Reuse tool. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Created reuse fix tool. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Added initial ruff config. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Configured ruff, poe, pytest and pyright. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Configured pyproject-fmt. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Added example scripts for ci testing. Added example integration test. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2298): Added support for doctests. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Added added github action for REUSE compliance check. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Added added github quality checks. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Fixed reuse compliance. in gha. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Changed triggers for gha checks. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Changed triggers for gha checks. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Narrowed check gha permissions. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Updated github action. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Added coverage checking. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Added coverage checking. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Added release / dev build job. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Updated checks workflow. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Updated checks workflow. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Updated checks workflow. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Improved version pinning. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Added extra and default dependencies for openstef root package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Added extra and default dependencies for openstef root package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Enabled dev release publishing. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Disabled dev release publishing until mvp is ready. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2299): Style fixes. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* feature(STEF-2300): Changed python dev version. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2300): Added docs setup. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2300): Fixed styles. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2300): Added improved docs and api documentation. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2300): Added v4 docs publishing. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2300): Added v4 docs publishing. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2300): Added v4 docs publishing. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2300): Added v4 docs publishing. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2300): Added new docs. Added contributing guide. Updated readme. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2300): Added new docs. Added contributing guide. Updated readme. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2300): Updated issue template links and slack channel mentions. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(STEF-2300): Updated docs gha trigger to remove test branch. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
Collaborator
Author
Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
…added documentation on how to do logging. (#643) * feature(#629): Added openstef compatibility package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#629): Added logging configuration and documentation on how to use logging in this package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#629): Added license gitignore exclusion for vscode stuff. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * Update packages/openstef-compatibility/README.md Co-authored-by: Lars Schilders <123180911+lschilders@users.noreply.github.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> * change: Updated docs. Removed vscode example config. * change: Added all modules to api reference. Removed context lib import from init. * feature: Sign commit. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> Co-authored-by: Lars Schilders <123180911+lschilders@users.noreply.github.com>
…red code. (#645) * feature(#628): Added openstef core package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added interfaces and implementations for TimeSeries datasets and their accessors. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added tests for versioned timeseries. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added moved transforms. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added documentation to datasets module. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * Update packages/openstef-core/tests/unit/datasets/test_versioned_timeseries_accessors.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feature(#628): Added openstef core package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added interfaces and implementations for TimeSeries datasets and their accessors. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added tests for versioned timeseries. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added moved transforms. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added documentation to datasets module. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Improved reference docs. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* feature(#628): Migrated openstef beam backtesting. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Migrated openstef beam backtesting. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Migrated metrics to openstef beam. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Migrated metrics and metric provider. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Migrated evaluation code. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Migrated analysis module to openstef beam. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Migrated benchmarking module to openstef beam. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added module level docs. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added docs for backtesting, evaluation and core. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added docs for analysis module. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Documented openstef beam benchmark. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added docs for benchmarking module. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Updated docstrings to avoid references to model interface and be more general regarding forecast targets. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
…rsioned parts to avoid O(n^2) problem when doing cross dataset operations. (#657) * feature(#628): Started on versioned timeseries dataset. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Started on versioned timeseries dataset. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Started on versioned timeseries dataset. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Updated rest of the code to use new versioned dataset type. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Updated rest of the code to use new versioned dataset type. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Fixed doctests. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Improved docs. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Updated docsting in mixin for select function and inherited it from the dataset part. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Updated tests and removed unnecessary docstrings. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Updated tests and removed unnecessary docstrings. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* Update CITATION.cff (#642) * Update CITATION.cff order citation according to DOI on zenodo Signed-off-by: Frank Kreuwel <frank.kreuwel@alliander.com> * Update CITATION.cff Signed-off-by: Frank Kreuwel <frank.kreuwel@alliander.com> * fix: yaml formatting Signed-off-by: Bart Pleiter <bart.pleiter@alliander.com> --------- Signed-off-by: Frank Kreuwel <frank.kreuwel@alliander.com> Signed-off-by: Bart Pleiter <bart.pleiter@alliander.com> Co-authored-by: Bart Pleiter <bart.pleiter@alliander.com> * feature(#629): Added openstef compatibility package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added openstef core package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added interfaces and implementations for TimeSeries datasets and their accessors. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added tests for versioned timeseries. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added moved transforms. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added documentation to datasets module. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * Update packages/openstef-core/tests/unit/datasets/test_versioned_timeseries_accessors.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> * add cylic features Signed-off-by: lschilders <lars.schilders@alliander.com> * update cylic features Signed-off-by: lschilders <lars.schilders@alliander.com> * add cylic features tests Signed-off-by: lschilders <lars.schilders@alliander.com> * add timeOfDay cyclic features Signed-off-by: lschilders <lars.schilders@alliander.com> * configurable cyclic features Signed-off-by: lschilders <lars.schilders@alliander.com> * configurable cyclic features Signed-off-by: lschilders <lars.schilders@alliander.com> * add rolling aggregate features Signed-off-by: lschilders <lars.schilders@alliander.com> * add rolling aggregate features tests Signed-off-by: lschilders <lars.schilders@alliander.com> * add rolling aggregate features tests Signed-off-by: lschilders <lars.schilders@alliander.com> * add datetime features and tests Signed-off-by: lschilders <lars.schilders@alliander.com> * add feature clipper and tests Signed-off-by: lschilders <lars.schilders@alliander.com> * added scaling transform Signed-off-by: lschilders <lars.schilders@alliander.com> * added scaling transform tests Signed-off-by: lschilders <lars.schilders@alliander.com> * fixed all poe tasks Signed-off-by: lschilders <lars.schilders@alliander.com> * add daylight features Signed-off-by: lschilders <lars.schilders@alliander.com> * import or skip pvlib Signed-off-by: lschilders <lars.schilders@alliander.com> * remove icalendar dependency Signed-off-by: lschilders <lars.schilders@alliander.com> * fix test formatting * install all packages in workflow * fix test formatting * feature(#626): Missing value imputation (#649) * Initial implementation for missing values handling feature engineering Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add to pyproject Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Added logic to remove rows with trailing nan for a feature. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add tests. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Restore pyproject.toml and uv.lock from origin/feature/626-feature-engineering Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix linting, formatting Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * More linting and formatting Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix typing Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Review comments Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Pyright Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Changes based on review comments Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix tests and warnings Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix warnings Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix fixtures. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Clean up Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add warning assert to test Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix attribute type Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix linting Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Reset uv.lock Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Config as part of init Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Remove type ignores Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Extend transform from baseconfig Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix linting Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Validate for fill value when constant strategy. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Use default value for missing_value Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Move imputer to private attribute. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix warning Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> --------- Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * initial commit weather transforms Signed-off-by: lschilders <lars.schilders@alliander.com> * fix type errors * fix most type and formatting errors * fix all errors except for weather transforms * fix radiation_derived_features tests * vectorize feature clipper computation * rename scaler to scaler_transform * rename rolling aggregate features * formatting Signed-off-by: lschilders <lars.schilders@alliander.com> * feature(#628): Update feature naming. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Updated forecasting transforms. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Refactored temporal transforms. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Refactored temporal transforms. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Fixed tests and doctests. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Updated docstring to include the right unit. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Updated doctests to include output examples. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Frank Kreuwel <frank.kreuwel@alliander.com> Signed-off-by: Bart Pleiter <bart.pleiter@alliander.com> Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> Signed-off-by: lschilders <lars.schilders@alliander.com> Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> Co-authored-by: Frank Kreuwel <frank.kreuwel@alliander.com> Co-authored-by: Bart Pleiter <bart.pleiter@alliander.com> Co-authored-by: Egor Dmitriev <egor.dmitriev@alliander.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Marnix van Lieshout <47454346+MvLieshout@users.noreply.github.com>
* Update CITATION.cff (#642) * Update CITATION.cff order citation according to DOI on zenodo Signed-off-by: Frank Kreuwel <frank.kreuwel@alliander.com> * Update CITATION.cff Signed-off-by: Frank Kreuwel <frank.kreuwel@alliander.com> * fix: yaml formatting Signed-off-by: Bart Pleiter <bart.pleiter@alliander.com> --------- Signed-off-by: Frank Kreuwel <frank.kreuwel@alliander.com> Signed-off-by: Bart Pleiter <bart.pleiter@alliander.com> Co-authored-by: Bart Pleiter <bart.pleiter@alliander.com> * feature(#629): Added openstef compatibility package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added openstef core package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added interfaces and implementations for TimeSeries datasets and their accessors. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added tests for versioned timeseries. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added moved transforms. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added documentation to datasets module. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * Update packages/openstef-core/tests/unit/datasets/test_versioned_timeseries_accessors.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> * add cylic features Signed-off-by: lschilders <lars.schilders@alliander.com> * update cylic features Signed-off-by: lschilders <lars.schilders@alliander.com> * add cylic features tests Signed-off-by: lschilders <lars.schilders@alliander.com> * add timeOfDay cyclic features Signed-off-by: lschilders <lars.schilders@alliander.com> * configurable cyclic features Signed-off-by: lschilders <lars.schilders@alliander.com> * configurable cyclic features Signed-off-by: lschilders <lars.schilders@alliander.com> * add rolling aggregate features Signed-off-by: lschilders <lars.schilders@alliander.com> * add rolling aggregate features tests Signed-off-by: lschilders <lars.schilders@alliander.com> * add rolling aggregate features tests Signed-off-by: lschilders <lars.schilders@alliander.com> * add datetime features and tests Signed-off-by: lschilders <lars.schilders@alliander.com> * add feature clipper and tests Signed-off-by: lschilders <lars.schilders@alliander.com> * added scaling transform Signed-off-by: lschilders <lars.schilders@alliander.com> * added scaling transform tests Signed-off-by: lschilders <lars.schilders@alliander.com> * fixed all poe tasks Signed-off-by: lschilders <lars.schilders@alliander.com> * add daylight features Signed-off-by: lschilders <lars.schilders@alliander.com> * import or skip pvlib Signed-off-by: lschilders <lars.schilders@alliander.com> * remove icalendar dependency Signed-off-by: lschilders <lars.schilders@alliander.com> * fix test formatting * install all packages in workflow * fix test formatting * feature(#626): Missing value imputation (#649) * Initial implementation for missing values handling feature engineering Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add to pyproject Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Added logic to remove rows with trailing nan for a feature. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add tests. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Restore pyproject.toml and uv.lock from origin/feature/626-feature-engineering Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix linting, formatting Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * More linting and formatting Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix typing Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Review comments Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Pyright Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Changes based on review comments Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix tests and warnings Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix warnings Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix fixtures. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Clean up Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add warning assert to test Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix attribute type Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix linting Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Reset uv.lock Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Config as part of init Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Remove type ignores Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Extend transform from baseconfig Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix linting Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Validate for fill value when constant strategy. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Use default value for missing_value Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Move imputer to private attribute. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix warning Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> --------- Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * initial commit weather transforms Signed-off-by: lschilders <lars.schilders@alliander.com> * fix type errors * fix most type and formatting errors * fix all errors except for weather transforms * fix radiation_derived_features tests * Init setup Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * vectorize feature clipper computation * rename scaler to scaler_transform * rename rolling aggregate features * formatting Signed-off-by: lschilders <lars.schilders@alliander.com> * Added sufficient complete check. Added tests. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Remove comment Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * feature(#628): Update feature naming. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Updated forecasting transforms. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * Fix docs Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * More testcases Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Raise error and optionally run on transform Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add override Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Raise error or return original dataframe Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix based on review Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Minor fixes Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Remove state from completeness check Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> --------- Signed-off-by: Frank Kreuwel <frank.kreuwel@alliander.com> Signed-off-by: Bart Pleiter <bart.pleiter@alliander.com> Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> Signed-off-by: lschilders <lars.schilders@alliander.com> Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> Co-authored-by: Frank Kreuwel <frank.kreuwel@alliander.com> Co-authored-by: Bart Pleiter <bart.pleiter@alliander.com> Co-authored-by: Egor Dmitriev <egor.dmitriev@alliander.com> Co-authored-by: lschilders <lars.schilders@alliander.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update CITATION.cff (#642) * Update CITATION.cff order citation according to DOI on zenodo Signed-off-by: Frank Kreuwel <frank.kreuwel@alliander.com> * Update CITATION.cff Signed-off-by: Frank Kreuwel <frank.kreuwel@alliander.com> * fix: yaml formatting Signed-off-by: Bart Pleiter <bart.pleiter@alliander.com> --------- Signed-off-by: Frank Kreuwel <frank.kreuwel@alliander.com> Signed-off-by: Bart Pleiter <bart.pleiter@alliander.com> Co-authored-by: Bart Pleiter <bart.pleiter@alliander.com> * feature(#629): Added openstef compatibility package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added openstef core package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added interfaces and implementations for TimeSeries datasets and their accessors. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added tests for versioned timeseries. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added moved transforms. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added documentation to datasets module. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * Update packages/openstef-core/tests/unit/datasets/test_versioned_timeseries_accessors.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> * add cylic features Signed-off-by: lschilders <lars.schilders@alliander.com> * update cylic features Signed-off-by: lschilders <lars.schilders@alliander.com> * add cylic features tests Signed-off-by: lschilders <lars.schilders@alliander.com> * add timeOfDay cyclic features Signed-off-by: lschilders <lars.schilders@alliander.com> * configurable cyclic features Signed-off-by: lschilders <lars.schilders@alliander.com> * configurable cyclic features Signed-off-by: lschilders <lars.schilders@alliander.com> * add rolling aggregate features Signed-off-by: lschilders <lars.schilders@alliander.com> * add rolling aggregate features tests Signed-off-by: lschilders <lars.schilders@alliander.com> * add rolling aggregate features tests Signed-off-by: lschilders <lars.schilders@alliander.com> * add datetime features and tests Signed-off-by: lschilders <lars.schilders@alliander.com> * add feature clipper and tests Signed-off-by: lschilders <lars.schilders@alliander.com> * added scaling transform Signed-off-by: lschilders <lars.schilders@alliander.com> * added scaling transform tests Signed-off-by: lschilders <lars.schilders@alliander.com> * fixed all poe tasks Signed-off-by: lschilders <lars.schilders@alliander.com> * add daylight features Signed-off-by: lschilders <lars.schilders@alliander.com> * import or skip pvlib Signed-off-by: lschilders <lars.schilders@alliander.com> * remove icalendar dependency Signed-off-by: lschilders <lars.schilders@alliander.com> * fix test formatting * install all packages in workflow * fix test formatting * feature(#626): Missing value imputation (#649) * Initial implementation for missing values handling feature engineering Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add to pyproject Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Added logic to remove rows with trailing nan for a feature. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add tests. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Restore pyproject.toml and uv.lock from origin/feature/626-feature-engineering Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix linting, formatting Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * More linting and formatting Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix typing Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Review comments Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Pyright Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Changes based on review comments Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix tests and warnings Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix warnings Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix fixtures. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Clean up Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add warning assert to test Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix attribute type Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix linting Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Reset uv.lock Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Config as part of init Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Remove type ignores Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Extend transform from baseconfig Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix linting Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Validate for fill value when constant strategy. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Use default value for missing_value Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Move imputer to private attribute. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix warning Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> --------- Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * initial commit weather transforms Signed-off-by: lschilders <lars.schilders@alliander.com> * fix type errors * Add flatliner check transform Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * fix most type and formatting errors * fix all errors except for weather transforms * fix radiation_derived_features tests * Remove redundant comment Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix import statement Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * vectorize feature clipper computation * rename scaler to scaler_transform * rename rolling aggregate features * formatting Signed-off-by: lschilders <lars.schilders@alliander.com> * Add absolute tolerance arg. Update docstrings and tests. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * feature(#628): Update feature naming. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * Improve docstring Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * feature(#628): Updated forecasting transforms. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * Added flatliner detection to transform Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add option to give custom load column name Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix order of exceptions Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add overrides Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> --------- Signed-off-by: Frank Kreuwel <frank.kreuwel@alliander.com> Signed-off-by: Bart Pleiter <bart.pleiter@alliander.com> Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> Signed-off-by: lschilders <lars.schilders@alliander.com> Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> Co-authored-by: Frank Kreuwel <frank.kreuwel@alliander.com> Co-authored-by: Bart Pleiter <bart.pleiter@alliander.com> Co-authored-by: Egor Dmitriev <egor.dmitriev@alliander.com> Co-authored-by: lschilders <lars.schilders@alliander.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feature(#628): Moved transforms to the openstef-models package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Moved transforms to the openstef-models package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Moved transforms to the openstef-models package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* feature(#628): Moved transforms to the openstef-models package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Moved transforms to the openstef-models package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Moved transforms to the openstef-models package. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Updated package deps. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* feature(#628): Added feature transform pipeline. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Improved doc example. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * Update packages/openstef-models/tests/unit/feature_engineering/validation_transforms/test_flatliner_check.py Co-authored-by: Marnix van Lieshout <47454346+MvLieshout@users.noreply.github.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> * feature(#628): Fixed doctest for FeaturPipeline. Added repr to leadtime. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Improved test cases for feature pipeline. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Improved test cases for horizon splitting. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> Co-authored-by: Marnix van Lieshout <47454346+MvLieshout@users.noreply.github.com>
…t part can have duplicate timestamps in index. (#665) * feature(#628): Added feature transform pipeline. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Improved doc example. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Added lag transform. Fixed bug where timeseries dataset part can have duplicate timestamps in index. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#628): Clarified test doc. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* Add file Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add calculations with references Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Fix quality checks Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Make stateless, fix doctest Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Rename to atmosphere derived features Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Rename test file Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add hardcoded values Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add more info and DOI Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Remove redundant comment Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add more info to docstring Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> --------- Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com>
* Init commit Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Clean up Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Improve docstrings Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Improve naming Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> --------- Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com>
…by fixing github links to docs and disabling source button. (#667) * docs(#656): Updated code style guide regarding types. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * docs(#661): Fixed links to edit files on github. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* feature(#627): Initial model interface implementation. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#627): Documented model interface mixins and updated documentation guidelines with new Invariant section. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#627): Added tests for combine_horizon_forecasts. Updated docstrings / and tests for mixins. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2391): Added tests for constant median forecaster and documentation to validated datasets. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#627): Updated code style guide. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2391): Added docs for forecaster implementations. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2391): Added docs for forecaster implementations. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2391): Fixed tests so fitting is more adaptable for multi horizon adapter. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#627): Minor fixes after feedback. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
…torage, callbacks. Refactor by reorganizing transforms. (#674) * refactor(#633): Moved feature engineering into transforms. Favoring a more flat structure for different operations on datasets. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Added storage, callback and workflow interfaces. Added an example and implemented example storage. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Added tests for new components. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Added docstings to the new pipelines. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Added docstings to the storage. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Renamed postprocessing pipeline to ForecastTransformPipeline. Refactored holiday features transform to avoid patching since it often breaks tests due to minor changes. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * Update packages/openstef-models/src/openstef_models/models/forecasting_model.py Co-authored-by: Lars Schilders <123180911+lschilders@users.noreply.github.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> * feature(#633): Updated dependency groups. Implemented is_fitted. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> Co-authored-by: Lars Schilders <123180911+lschilders@users.noreply.github.com>
* add rIQD and relative pinball loss metrics Signed-off-by: MentReeze <ment.reeze@alliander.com> * docstring fix Co-authored-by: Lars Schilders <123180911+lschilders@users.noreply.github.com> Signed-off-by: MentReeze <ment.reeze@alliander.com> * add unit test for symmetric quantile logic of rIQD provider Signed-off-by: MentReeze <ment.reeze@alliander.com> --------- Signed-off-by: MentReeze <ment.reeze@alliander.com> Co-authored-by: Lars Schilders <123180911+lschilders@users.noreply.github.com>
…n data (#676) * Add the option to plot timeseries without showing interpolated lines where gaps in data Signed-off-by: MentReeze <ment.reeze@alliander.com> * Fix ruff formatting issues Signed-off-by: MentReeze <ment.reeze@alliander.com> --------- Signed-off-by: MentReeze <ment.reeze@alliander.com>
…ded split component pipeline. (#677) * feature(#633): Moved model mixins into a single namespace inside models. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Updated transform interfaces. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Refactored feature pipeline and transform base. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Added component splitting. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Refactored transforms, predictors, and statefulness. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Improved doctests and docstrings. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Improved docstrings. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Added tests voor multi horizon transform adapter. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Added support for validation set during training. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#633): Changed dataset to data naming for consistency in forecasting model. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* feature(#687): Implemented xgboost model. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#687): Added tests for loss objectives. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#687): Added docstrings for xgboost implementation. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#687): Improved tests. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * Update packages/openstef-models/src/openstef_models/models/forecasting/xgboost_forecaster.py Co-authored-by: Lars Schilders <123180911+lschilders@users.noreply.github.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> * feature(#687): Documented model state. Moved input_data filtering into forecast input dataset. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> Co-authored-by: Lars Schilders <123180911+lschilders@users.noreply.github.com>
* feature: add sample_weight_column to ForecastInputDataset * add doctest example * add code and config for gblinear * feature(#687): Implemented xgboost model. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#687): Added tests for loss objectives. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#687): Added docstrings for xgboost implementation. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feature(#687): Improved tests. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * refactor gblinear and tests * Update packages/openstef-models/src/openstef_models/models/forecasting/xgboost_forecaster.py Co-authored-by: Lars Schilders <123180911+lschilders@users.noreply.github.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> * feature(#687): Documented model state. Moved input_data filtering into forecast input dataset. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * align GBLinear implementation with XGBoost and add booster type checking * add GBLinear implementation * revert changes to types * revert changes to pyproject * revert changes to pyproject * add doctest SKIP --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egordmitriev2@gmail.com> Co-authored-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* refactor(STEF-2702): push polymorphism into models, eliminate isinstance protocols - SubsetMetric.to_flat_dict() for metrics self-serialization - ModelFitResult.metrics_to_flat_dict() with component_fit_results override - BaseForecastingModel: component_hyperparams, get_explainable_components() - ForecastCombiner extends ExplainableForecaster (gains plot_feature_importances) - EnsembleForecastingModel/EnsembleModelFitResult override all polymorphic methods - MLflow callback: delete 3 protocols, 5 isinstance checks -> polymorphic calls - Fix penalty bug: lower_is_better now multiplies (was dividing) * refactor(STEF-2702): extract normalize_to_unit_sum, dedup feature importances - Add normalize_to_unit_sum() pipe-compatible utility in openstef_core.utils.pandas - Replace duplicated normalization logic in 4 forecasters (xgboost, gblinear, lgbm, lgbmlinear) - All now use weights_df.pipe(normalize_to_unit_sum) * fix: make AvailableAt.__str__() Windows-safe by removing colon Change DnTHH:MM format to DnTHHMM (e.g. D-1T0600 instead of D-1T06:00). Colons are illegal in Windows file paths, breaking benchmark output directories. from_string() now accepts both formats for backward compatibility. * feat(STEF-2702): make openstef-models/meta optional for beam baselines * refactor(STEF-2702): replace workflow factory with template + model_copy - Replace workflow_factory Callable with workflow_template field - Add with_run_name() method for type-safe deep copy - Add kind discriminator to ForecastingWorkflowConfig (single) and EnsembleForecastingWorkflowConfig (ensemble) - Simplify factory: use kind-based narrowing, raise MissingExtraError - Delete WorkflowCreationContext (no longer needed) - Remove both pyright: ignore[reportArgumentType] suppressions * test(STEF-2702): add template pattern smoke tests for OpenSTEF4BacktestForecaster - test_fit_does_not_mutate_template: verifies template immutability after fit - test_fit_then_predict_returns_forecast: e2e smoke test for fit→predict path * chore(STEF-2702): add openstef-meta to release pipeline - Add missing [build-system] section to openstef-meta pyproject.toml - Add openstef-meta to poe version task (version bump + root pinning) - Fix baselines extra version range: >=4.0.0.dev0,<5 (was >=0.0.1,<1) - Add openstef-meta to root [all] optional extra - Add openstef-meta to licensecheck ignore_packages * refactor(STEF-2702): extend test utility, address review comments - Add include_atmosphere/price/available_at options to create_synthetic_forecasting_dataset for realistic test data - Simplify test fixture to use the shared utility - Remove verbose parenthetical comments per review feedback
…ty (#826) - Add normalize_tracking_uri() helper that converts local file paths to file:/// URIs, handling relative paths, absolute POSIX paths, and Windows drive-letter paths (e.g. D:\mlflow) - Apply normalization in MLFlowStorage.model_post_init() so all users get correct URIs regardless of how they construct tracking_uri - Add parametrized tests covering 8 URI variants including a mock-based Windows drive letter test
Signed-off-by: lschilders <lars.schilders@alliander.com>
* Add run_name. Set default weights. Add confidence interval applicator postprocessing. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add lgbm extra for openstef-models dep in openstef-meta Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * No sample weights standard for lgbm Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> --------- Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com>
* Add run_name. Set default weights. Add confidence interval applicator postprocessing. Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Add lgbm extra for openstef-models dep in openstef-meta Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * No sample weights standard for lgbm Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * transform -> fit_transform Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Split postprocessing into common, per-forecaster and combiner Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> --------- Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com>
* feature: add apply() to AvailableAt and tz-aware option * feature: refactor AvailableAt to use day_offset and time_of_day rather than lag_from_day * feature: (de)serialization support for tz-aware AvailableAt * feature: improved docstring * feature: add tests for apply_index Signed-off-by: lschilders <lars.schilders@alliander.com> --------- Signed-off-by: lschilders <lars.schilders@alliander.com>
Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com>
…833) * fix: timezone handling for offset in AvailableAt apply() (matching apply_index()) * feature: add new metric NMAE (normalized MAE) * add MAEProvider * make allow_nan configurable * refactored NMAE into rMAE with norm_value Signed-off-by: lschilders <lars.schilders@alliander.com> * formatting Signed-off-by: lschilders <lars.schilders@alliander.com> * add allow_nan to rMAE --------- Signed-off-by: lschilders <lars.schilders@alliander.com>
* Update workflows Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> * Revert doc change Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> --------- Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com>
* fix(STEF-2854): handle InsufficientlyCompleteError during backtest training OpenSTEF4BacktestForecaster.fit() now catches InsufficientlyCompleteError alongside FlatlinerDetectedError. When a training window has insufficient non-NaN data, the training event is skipped and the previous model is retained instead of crashing the entire target backtest. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * test(STEF-2854): replace mock with real NaN data in insufficient-data test Use all-NaN load data with model_reuse_enable=False to trigger InsufficientlyCompleteError naturally instead of patching workflow.fit. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2854): return None from predict() when no model fitted When the first fit fails due to InsufficientlyCompleteError, _workflow stays None. predict() now returns None (like flatliner) instead of raising NotFittedError, letting the benchmark pipeline skip gracefully. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2854): make WindowedMetricVisualization robust to missing data Skip runs/targets with no windowed metrics instead of raising ValueError. Returns an HTML placeholder when all items in a visualization are empty. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2854): fix combiner label/weight shape mismatch and Quantile serialization Two fixes: 1. learned_weights_combiner.py: Filter labels to match combined_data index after inner join drops rows from additional_features. Fixes ValueError: operands could not be broadcast together. 2. types.py: Add Pydantic serializer to Quantile to suppress PydanticSerializationUnexpectedValue warnings. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2854): add Pydantic serializer to Quantile to suppress warnings Quantile.__get_pydantic_core_schema__ only defined a validator but no serializer. When Quantile values appear as dict keys in a union type (e.g., QuantileOrGlobal = Quantile | Literal['global']), Pydantic emits PydanticSerializationUnexpectedValue warnings during model_dump_json(). Add a plain_serializer_function_ser_schema(float) so Pydantic knows how to serialize Quantile as a float, preventing the warning. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2854): raise InsufficientlyCompleteError for empty datasets in train/test split chronological_train_test_split crashed with IndexError when the dataset had fewer than 2 unique timestamps. This happens during ensemble backtest when a base forecaster's preprocessed data is empty. Now raises InsufficientlyCompleteError which is caught by the backtest harness. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2854): raise InsufficientlyCompleteError on empty combiner data after inner join Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feat(STEF-2854): add strict parameter to BenchmarkComparisonPipeline.run() Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2854): use 'global' subdirectory for RUN_AND_GROUP analysis scope The RUN_AND_GROUP scope was saving directly to the base analysis dir, making has_analysis_output fail to locate it and colliding with group-level outputs. Store in a 'global' subdirectory to match the group-level pattern (group_name/global). Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2854): renormalize ensemble weights when base model predictions are NaN When a base model cannot predict certain timestamps (e.g. gblinear limited to 2-day weather horizon while lgbm predicts 7 days), the combiner must redistribute the missing model's weight proportionally to the remaining models. Previously, pandas sum(axis=1, skipna=True) silently dropped the NaN model's weight contribution, causing predictions to be systematically scaled down by ~35% for timestamps beyond the weather horizon. Now weights are reindexed to match predictions, zeroed where predictions are NaN, and the weighted sum is divided by the available weight total. When all models are NaN, the result is 0 (matching prior behavior). Includes regression test with seeded data verifying no NaN propagation and no systematic downscaling. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * refactor(STEF-2854): extract nan_aware_weighted_mean helper Extract NaN-aware weight renormalization into a reusable helper in openstef_core.utils.pandas and use it in learned_weights_combiner. Removes type: ignore comments from _predict_quantile. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feat(STEF-2854): add skip_analysis param to BenchmarkPipeline.run() Allows skipping per-target and global analysis steps when running benchmarks. Useful when analysis will be run separately later via the comparison pipeline. Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * feat(STEF-2854): add filterings override to AnalysisConfig Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2854): resolve ruff lint warnings Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> * fix(STEF-2854): resolve pyright type errors in modified files Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com> --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
* add shifter transform and tests Signed-off-by: MentReeze <ment.reeze@alliander.com> * added shifter to forecasting workflow Signed-off-by: MentReeze <ment.reeze@alliander.com> * changed averaging to aggregated, simplified edge handling and removed cast to datetimeindex Signed-off-by: MentReeze <ment.reeze@alliander.com> * changed to list of shifts Signed-off-by: MentReeze <ment.reeze@alliander.com> * small fix: removed redundant list() Signed-off-by: MentReeze <ment.reeze@alliander.com> --------- Signed-off-by: MentReeze <ment.reeze@alliander.com>
## Summary Adds end-to-end examples showing how to create and run custom BEAM benchmarks, plus a Windows compatibility fix. ## Custom Benchmark Examples | File | Purpose | |---|---| | `example_baseline.py` | Minimal forecaster implementing `BacktestForecasterMixin` — predicts a constant median | | `example_benchmark.py` | Custom target provider extending `SimpleTargetProvider` with metrics and pipeline assembly | | `run_liander2024_benchmark.py` | Runs the example baseline + GBLinear on the built-in Liander 2024 dataset | | `run_benchmark.py` | Runs the example baseline + GBLinear using the custom benchmark pipeline | | `README.md` | Beginner-friendly guide with setup, usage, and "create your own" instructions | All files include inline comments explaining concepts for newcomers (quantiles, `RestrictedHorizonVersionedTimeSeries`, config parameters, metrics, etc.). ## Windows Compatibility Fix **`AvailableAt.__str__()`** now produces `D-1T0600` instead of `D-1T06:00`. Colons are illegal in Windows file/directory names, which broke benchmark output paths like `benchmark_results/analysis/mv_feeder/OS Edam/D-1T06:00/`. - `from_string()` accepts both `D-1T06:00` (legacy) and `D-1T0600` (new) for backward compatibility - All existing unit tests pass (core + beam) --------- Signed-off-by: Egor Dmitriev <egor.dmitriev@alliander.com>
## Summary This PR extends the current clipping transform with optional NaN handling for out-of-range values. ## Changes - Add `outlier_action` parameter: - `clip` (default): existing behavior - `nan`: replace out-of-range values with NaN - Rename `Clipper` to `OutlierHandler` - Update unit tests for both clip and NaN behavior ## Motivation For some applications, clipping many outliers may distort the input distribution by concentrating values at the learned bounds. In these cases, it is preferable to replace out-of-range values with `NaN` so downstream models can ignore or specially handle them. ## Base The branch was originally created from `v4.0.0.a30` and rebased onto `release/v4.0.0`. --------- Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com> Co-authored-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com>
…855) Signed-off-by: MentReeze <ment.reeze@alliander.com> Co-authored-by: Lars Schilders <123180911+lschilders@users.noreply.github.com>
…856) This pull request significantly improves the precision and robustness of the `Quantile` type in `openstef_core/types.py` by switching to `Decimal` arithmetic for all conversions and string formatting, and by adding new utility methods. Comprehensive unit tests have been added to ensure correct behavior and to cover edge cases. **Precision improvements and new features for `Quantile`:** * Switched all internal arithmetic in `Quantile` to use `Decimal`, ensuring that values like `Quantile(0.999)` round-trip exactly through `format()`, `parse()`, `to_percentile()`, and `from_percentile()` without floating-point rounding errors. [[1]](diffhunk://#diff-a5fdd3bb0fdb7e7fda4a88fd6ed02992f205ca1d03765b5be5ebf87867b04239R17) [[2]](diffhunk://#diff-a5fdd3bb0fdb7e7fda4a88fd6ed02992f205ca1d03765b5be5ebf87867b04239R301-R304) [[3]](diffhunk://#diff-a5fdd3bb0fdb7e7fda4a88fd6ed02992f205ca1d03765b5be5ebf87867b04239R355-R379) * Added new methods: `from_percentile`, `to_percentile`, and `inverse`, providing convenient and precise conversions between quantiles and percentiles, and to compute the complementary quantile. * Improved the `format` and `parse` methods to handle arbitrary decimal precision and to use `Decimal` for exact string conversions. [[1]](diffhunk://#diff-a5fdd3bb0fdb7e7fda4a88fd6ed02992f205ca1d03765b5be5ebf87867b04239R355-R379) [[2]](diffhunk://#diff-a5fdd3bb0fdb7e7fda4a88fd6ed02992f205ca1d03765b5be5ebf87867b04239L377-R394) * Updated the quantile string validation regex to support up to three digits and arbitrary decimal places, e.g., `quantile_P99.99`. **Testing and validation:** * Added comprehensive unit tests for all new and updated `Quantile` methods, including round-trip conversions, edge cases, error handling, and the new `inverse` method.
Add nan_features field to ForecastingWorkflowConfig and EnsembleForecastingWorkflowConfig. When set, a second OutlierHandler with outlier_action='nan' replaces out-of-range values with NaN instead of clipping them to bounds. Default is NONE (no features). --------- Signed-off-by: Marnix van Lieshout <marnix.van.lieshout@alliander.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.