0.24.0

Highlights

• Cubed can now use any Zarr store for intermediate storage.
• Memory limits are now checked when building the plan rather than when array API operations are called, which allows you to visualize which operations exceed allowed memory.

What's Changed

• Make itemsize support other Array API implementations by @tomwhite in #804
• Make count_nonzero work with array-api-strict by @tomwhite in #806
• Use cubed version of normalize_chunks in diff by @tomwhite in #807
• Storage array rename by @tomwhite in #808
• Remove comment about take_along_axis not being in array API by @tomwhite in #809
• Explicitly use Beam Direct Runner in tests since default is now Prism by @tomwhite in #815
• Don't use len on array by @tomwhite in #816
• Allow an arbitrary Zarr store to be used for intermediate storage by @tomwhite in #801
• Refactor plan to store output array names by @tomwhite in #819
• Misc doc improvements by @tomwhite in #822
• Add callback for when an operation has ended by @tomwhite in #823
• Use new zarrs entrypoint by @tomwhite in #824
• Remove Python 3.11 conditions since it's now the minimum Python version by @tomwhite in #825
• Improve task naming for Coiled executor by @tomwhite in #811
• Add plan function by @tomwhite in #827
• Plan stats by @tomwhite in #828
• Allow int dtypes in mean (and std, var) by @tomwhite in #831
• Add isin by @tomwhite in #832
• Support allow_rechunk=True in case no rechunking is necessary by @tomwhite in #834
• Improve detection of array names in visualize by @tomwhite in #835
• Check memory when building plan by @tomwhite in #836
• Pass executor options to default threads executor by @tomwhite in #839
• Don't count virtual arrays towards max_total_source_arrays by @tomwhite in #838
• Improve equality comparison for executors by @tomwhite in #840
• Clarifying Donfig Reference in Configuration Documentation by @neilSchroeder in #844
• Visualize plans that exceed memory by @neilSchroeder in #846

New Contributors

• @neilSchroeder made their first contribution in #844

Full Changelog: 0.23.0...0.24.0

0.23.0

Highlights

Cubed now runs on GPUs, supports Python array API standard 2024.12, can use obstore, and can write to regions of Zarr arrays.

What's Changed

• Update to use latest array-api-tests tag by @tomwhite in #765
• Ensure chunks are square for eye by @tomwhite in #763
• Make fsspec a test only dependency by @tomwhite in #766
• Add instructions for running examples on Ray cluster on AWS by @tomwhite in #769
• Basic support for device arrays by @TomAugspurger in #772
• Misc dtype fixes by @tomwhite in #771
• Reduce fsspec usage in tests by @tomwhite in #776
• Revert modal test change back to running on fsspec by @tomwhite in #777
• Array API 2024 additions by @tomwhite in #779
• Implement diff by @tomwhite in #781
• Exclude Zarr 3.1.2 due to zarr-developers/zarr-python#3406 by @tomwhite in #787
• Update slides - August 2025 by @tomwhite in #786
• Separate store and array name for intermediate storage by @tomwhite in #785
• Factor out dtype-related normalization functions by @tomwhite in #788
• Fix Modal CI by @tomwhite in #790
• Add support for zarrs-python by @tomwhite in #789
• Docs for Zarr storage backends by @tomwhite in #791
• Add Cubed logo by @tomwhite in #792
• Add option to run examples on Lithops AWS lambda with obstore by @tomwhite in #778
• Allow Modal secret name to be specified in config by @tomwhite in #793
• Add cumulative_prod by @tomwhite in #795
• Add support for complex floating-point data types in mean by @tomwhite in #796
• Run Modal tests on S3 using obstore by @tomwhite in #794
• Misc array API improvements by @tomwhite in #798
• Declare support for Array API 2024.12 by @tomwhite in #797
• Support region(s) in to_zarr and store by @tomwhite in #799
• Use Cubed version of array assertion functions in tests by @tomwhite in #802

New Contributors

• @TomAugspurger made their first contribution in #772

Full Changelog: 0.22.0...0.23.0

0.22.0

What's Changed

• Remove skipna=False from the Xarray examples by @tomwhite in #706
• Spark executor by @songhan89 in #704
• Update executor docs with new experimental executors by @tomwhite in #705
• Icechunk example by @tomwhite in #708 and #709
• Remove new_axis from map_overlap kwargs by @tomwhite in #710
• Make mypy ignore vendored libraries by @tomwhite in #716
• Fix doc build by @tomwhite in #717
• Drop Python 3.10 by @tomwhite in #718
• Add scan by @dcherian in #531
• Change list of output arrays in BlockwiseSpec to a map by @tomwhite in #724
• Simplify resume implementation by taking out of runtime by @tomwhite in #725
• Mem utilization tests fail on zarr-python v3 by @tomwhite in #697
• Minor improvements to rechunk_new by @tomwhite in #732
• Use general_blockwise for scan, rather than map_direct by @tomwhite in #726
• Run array API tests on Zarr Python 2 and 3 by @tomwhite in #733
• Add MemoryVisualizationCallback by @tomwhite in #734
• Ruff by @tomwhite in #735
• Explicitly skip test_resume with tensorstore by @tomwhite in #736
• Allow use of obstore as a Zarr store by @tomwhite in #723
• Fix cardinality of output_dtypes in apply_gufunc by @tomwhite in #738
• Fix chunking where indexing an array returns an empty array by @tomwhite in #739
• Add a blocks property to Cubed arrays to allow block-level indexing by @tomwhite in #737
• Migrate to Icechunk 1.0 ForkSessions by @tomwhite in #742
• Use new rechunk implementation by default by @tomwhite in #741
• Remove zarray and zarray_maybe_lazy properties from Cubed Array by @tomwhite in #743
• Move index to indexing.py module by @tomwhite in #744
• Don't measure coverage for new executors by @tomwhite in #745
• Remove Spec from getting started doc by @tomwhite in #746
• Fix successors_unordered by @tomwhite in #747
• Exclude array API test that attempts in-place mutation by @tomwhite in #754
• Only run Zarr v2 CI tests on default local executors by @tomwhite in #759
• Update lithops aws image by @tomwhite in #755
• Fix indexing zero-sized dimension by @tomwhite in #761
• Run lithops CI on ubuntu-22.04 by @tomwhite in #762

New Contributors

• @songhan89 made their first contribution in #704
• @dcherian made their first contribution in #531

Full Changelog: 0.21.0...0.22.0

0.21.0

What's Changed

• Icechunk store by @tomwhite in #633
• Move retry logic to lithops by @tomwhite in #673
• Coverage improvements by @tomwhite in #674
• Add scalar support to where by @tomwhite in #679
• Add PendingDeprecationWarning for map_direct by @tomwhite in #682
• Add a rechunking example by @tomwhite in #681
• Remove Lithops retries tests since this is now tested upstream by @tomwhite in #683
• Using default_dtypes instead of hard-coding dtypes by @alxmrs in #666
• Array API status update for draft 2024 spec by @tomwhite in #684
• Refactor asyncio executors to share common code by @tomwhite in #685
• Add scalar support to binary element-wise ops by @tomwhite in #686
• Fix icechunk tests to opt-in to pickling of Session by @tomwhite in #688
• Move non-standardized functions to separate package by @tomwhite in #690
• Add Ray executor by @tomwhite in #687
• Allow nan functions to work with xarray by @tomwhite in #670
• Make callbacks context managers by @tomwhite in #691
• Adding pre-commit (for linting) to test dependencies by @alxmrs in #671
• Support float32 in random number generation by @tomwhite in #693
• Fix for Zarr 3.0.3 change to OrthogonalIndexer internals from Zarr #2784 by @tomwhite in #696
• Modal 1.0 changes by @tomwhite in #695
• Show nchunks on visualization by @tomwhite in #698
• Always set region for Modal executor by @tomwhite in #699
• Improve memory model by explicitly modelling buffer copies by @tomwhite in #701
• Multistage rechunking by @tomwhite in #700
• Remove duplicate code from new rechunk implementation by @tomwhite in #702
• Use extra buffer copies for cloud stores like S3 by @tomwhite in #703

Full Changelog: 0.20.0...0.21.0

0.20.0

What's Changed

• Include task result in callbacks by @tomwhite in #632
• Pin dask and distributed to < 2024.12.0 by @tomwhite in #639
• Fix Modal deprecation (show progress) and pip package by @tomwhite in #634
• Remove test 'extras' and run lithops tests separately by @tomwhite in #640
• Array API tests fixes by @tomwhite in #636
• New tagline by @tomwhite in #641
• Add tile by @tomwhite in #644
• Add searchsorted by @tomwhite in #647
• Declare Array API 2023.12 support by @tomwhite in #651
• Change chunk_shape back to chunks for Zarr v3 by @tomwhite in #656
• Document use of cubed.config by @tomwhite in #654
• Make async executors work in Jupyter notebooks by @tomwhite in #661
• Add Zarr and Xarray examples to docs by @tomwhite in #655
• Fix tooltips for SVGs generated by visualize() in Jupyter notebooks by @tomwhite in #663
• Add note to contributing guide for developers to have graphviz installed by @alxmrs in #665
• Incorrect dtypes in map_selection by @tomwhite in #669

Full Changelog: 0.19.0...0.20.0

0.19.0

What's Changed

• Support block_id for general_blockwise functions by @tomwhite in #593
• Run tests on Python 3.13 by @tomwhite in #503
• Fix dtypes for QR by @tomwhite in #594
• Change usages of map_direct in QR to general_blockwise by @tomwhite in #597
• Implement var and std using a numerically stable parallel algorithm by @tomwhite in #596
• Add map_selection primitive to replace map_direct by @tomwhite in #595
• Use unstack from array-api-compat by @tomwhite in #602
• Increase allowed_mem default to 2GB by @tomwhite in #599
• Implement structured arrays as Zarr group of arrays by @tomwhite in #603
• Implement int array indexing using map_selection by @tomwhite in #604
• Implement merge_chunks using map_selection by @tomwhite in #605
• Free memory earlier by changing function scope by @tomwhite in #606
• Fix groupby tests running on tensorstore by @tomwhite in #608
• Implement concat using general_blockwise by @tomwhite in #607
• Speed up test collection by @tomwhite in #609
• Update GH actions to macos-13 test runner and upload-artifact@v4 by @tomwhite in #611
• Implement repeat by @tomwhite in #610
• Map nested improvements by @tomwhite in #613
• SVD by @tomwhite in #598
• Unify chunks for concat, and check preconditions by @tomwhite in #616
• Defer to merge_chunks in special cases of rechunk by @tomwhite in #612
• Document map_selection core operation by @tomwhite in #617
• Change default max_total_num_input_blocks to 10 by @tomwhite in #615
• Update slides - November 2024 by @tomwhite in #620
• Change use_backups default to True only on cloud stores by @tomwhite in #619
• Various minor doc improvements by @tomwhite in #625
• Make threads the default executor by @tomwhite in #621
• Rewrite Cubed intro docs by @tomwhite in #622
• More doc improvements: simplify getting started page by @tomwhite in #629
• Add examples to docs by @tomwhite in #630

Full Changelog: 0.18.0...0.19.0

0.18.0

What's Changed

• Multiple outputs by @tomwhite in #419
• Support use of multiprocessing start methods other than "spawn" (e.g. "dragon") by @applio in #554
• Tensorstore compression fix by @tomwhite in #574
• Implement unstack using multiple outputs by @tomwhite in #575
• Handle more memray allocation types by @tomwhite in #578
• Drop Python 3.9 by @tomwhite in #576
• QR decomposition by @tomwhite in #577
• Expose the linalg namespace and include in status page by @tomwhite in #581
• Fix outer result dtype by @tomwhite in #582
• Add new 2023.12 elemwise functions by @tomwhite in #583
• Bring vecdot implementation in line with the one in array-api-compat by @tomwhite in #402
• Zarr v3 now requires at least Python 3.11 by @tomwhite in #585
• Debug failing straggler test by @tomwhite in #580
• Set object codec for object arrays by @tomwhite in #573
• Fix for flaky straggler test by @tomwhite in #587
• Use general_blockwise in case of chunk-aligned selections in index by @tomwhite in #586
• Remove deprecated code by @tomwhite in #588
• Documentation small fixes by @tomwhite in #591
• Add page on optimization to user guide by @tomwhite in #397
• Remove old reduction implementation by @tomwhite in #589
• Add array API inspection utilities by @tomwhite in #592

New Contributors

• @applio made their first contribution in #554

Full Changelog: 0.17.0...0.18.0

0.17.0

What's Changed

• Skip fewer array API tests by @tomwhite in #547
• Rechunk where dict has missing axes by @tomwhite in #546
• Adds logging [INFO] to TimelineVisualizationCallback and HistoryCallback by @norlandrhagen in #548
• More lenient dtype support by @tomwhite in #550
• Cast inputs to Cubed arrays in apply_ufunc by @tomwhite in #551
• Fix bug where newaxis with full slices doesn't add new axes by @tomwhite in #559
• Memray integration by @tomwhite in #558
• Add chunkmem to VirtualFullArray, which was missed in #447 by @tomwhite in #560
• Fix Array API Tests by @tomwhite in #562
• Memray callback by @tomwhite in #561
• Introduce FinalizedPlan by @tomwhite in #563
• Materialize arrays optimizer bugfix by @tomwhite in #564
• Align tqdm progress bars by @tomwhite in #567
• Coerce args to map_blocks to arrays by @tomwhite in #566
• Fail during planning if map_blocks drop_axis is for a chunked dimension by @tomwhite in #569
• Fix Rich progress bar flickering by limiting refreshes by @tomwhite in #571
• Allow Zarr compression to be set for intermediate files by @tomwhite in #572

Full Changelog: 0.16.0...0.17.0

0.16.0

What's Changed

• Improve documentation around Array API support by @tomwhite in #477
• Speed up tests by @JosepSampe in #478
• Add normalize_shape since it is not in Zarr's public API by @tomwhite in #479
• Implement virtual array indexing using ndindex by @tomwhite in #441
• Ndindex indexing by @tomwhite in #481
• Pluggable backend storage by @tomwhite in #480
• Ensure integer arrays used for indexing are NumPy arrays by @tomwhite in #485
• Indicate that we support both 2021.12 and 2022.12 versions of the Array API by @tomwhite in #487
• TensorStore storage by @tomwhite in #483
• Update slides - June 2024 by @tomwhite in #489
• Fail with NotImplementedError if backend Zarr is missing nchunks_initialized by @tomwhite in #491
• Zarr v3 by @tomwhite in #484
• Remove print for local executors by @tomwhite in #506
• document what needs to be installed to run climatology anomaly example by @rbavery in #501
• Document how to run examples on local machine using processes by @tomwhite in #507
• Check allowed_mem does not exceed total on machine for processes executor by @tomwhite in #517
• Warn if peak mem exceeds allowed_mem by @tomwhite in #516
• Remove max_tasks_per_child=1 limitation from processes executor by @tomwhite in #515
• Zarr Python v3 updates by @tomwhite in #523
• Modal progress fix by @tomwhite in #522
• Delete local context directories on exit by @tomwhite in #524
• Update demo notebooks by @tomwhite in #525
• Update CUBED_CONFIG path in example README.md by @thodson-usgs in #530
• Implement flip by @tomwhite in #528
• Add better test for local paths in delete_on_exit by @tomwhite in #532
• Deprecate cubed.extensions and move to cubed.diagnostics by @tomwhite in #533
• Turning on JAX tests for all PRs. by @alxmrs in #537
• Improve set-up instructions for examples by @tomwhite in #534
• Configuration documentation by @tomwhite in #535
• Adding compile_function as execute option. by @alxmrs in #536
• Diagnostics documentation by @tomwhite in #540

New Contributors

• @JosepSampe made their first contribution in #478
• @rbavery made their first contribution in #501
• @thodson-usgs made their first contribution in #530
• @alxmrs made their first contribution in #537

Full Changelog: 0.15.0...0.16.0

0.15.0

What's Changed

• Test on Python 3.12 by @tomwhite in #453
• Test NumPy 2 by @tomwhite in #454
• Expose Lithops wait_dur_sec to speed up tests by @tomwhite in #456
• Simplify executor naming by @tomwhite in #457
• Treat warnings as errors when running tests by @tomwhite in #458
• Re-interpret 'fusable' to mean with predecessor operations by @tomwhite in #460
• Don't include hidden arrays when counting max_total_source_arrays by @tomwhite in #463
• Limited implementation of map_overlap by @tomwhite in #462
• Limited implementation of pad by @tomwhite in #461
• Implement roll by @tomwhite in #465
• Fix map_overlap in case of single chunk in overlap dimension by @tomwhite in #466
• Fix case of Lithops 3.4.0 hanging by @tomwhite in #471
• Fix unify_chunks to return regular chunks in all cases by @tomwhite in #470
• Improve test coverage by @tomwhite in #472
• Change development status to beta by @tomwhite in #474
• Top level array namespace by @tomwhite in #473
• Add notebooks for Pangeo examples by @tomwhite in #476

Full Changelog: 0.14.3...0.15.0