`fixed_size_list<T>[0]` parquet round-trips fail: writes fine, read-back raises `ArrowInvalid`

[ikrommyd]

Describe the bug, including details regarding any error messages, version, and platform.

A fixed_size_list<T> array with list_size=0 can be written to Parquet successfully but fails to read back with

ArrowInvalid: Expected all lists to be of size=0 but index 1 had size=1

Note also that the high-level constructor pa.FixedSizeListArray.from_arrays explicitly rejects list_size=0 with "list_size needs to be a strict positive integer", but the type can still be constructed via Array.from_buffers, and pq.write_table accepts it without complaint. The API is inconsistent: either the type should be rejected everywhere, or it should round-trip correctly.

inner = pa.array([], type=pa.float32())  # 0 child values
fsl = pa.FixedSizeListArray.from_arrays(inner, list_size=0)

raises

ArrowInvalid: list_size needs to be a strict positive integer

Reproducer

import pyarrow as pa
import pyarrow.parquet as pq

fsl = pa.Array.from_buffers(
    pa.list_(pa.float32(), 0),
    length=50,
    buffers=[None],
    children=[pa.array([], type=pa.float32())],
)
table = pa.table({"x": fsl})

pq.write_table(table, "tmp.parquet")   # OK
pq.read_table("tmp.parquet")           # ArrowInvalid

Component(s)

Parquet, Python

[rok]

Thanks for reporting this @ikrommyd !
Would you be open to submitting a PR for this?

[ikrommyd]

Thanks for reporting this @ikrommyd ! Would you be open to submitting a PR for this?

Should this raise an error during writing or do you expect a PR to fix the reading?

[rok]

What is your preference @ikrommyd ?
Since we put a verification check for pa.FixedSizeListArray.from_arrays(inner, list_size=0) in place I imagine we would prefer that behavior.
cc @pitrou

[ikrommyd]

Well the way I encountered this problem is through awkward array which does allow a 50 * 0 * float32 shape like so (using the semi-public ak.contents classes which are the array internals)

In [1]: import awkward as ak
   ...: import numpy as np
   ...:
   ...: layout = ak.contents.RegularArray(
   ...:     ak.contents.NumpyArray(np.zeros(0, dtype=np.float32)),
   ...:     size=0,
   ...:     zeros_length=50,
   ...: )
   ...:
   ...: arr = ak.Array(layout)
   ...: print(arr)
   ...: print(arr.type)
[[], [], [], [], [], [], [], [], [], ..., [], [], [], [], [], [], [], [], []]
50 * 0 * float32

I was trying to save this to a parquet file which awkward does through arrow exactly in the same way my above reproducer does (extensionarray=False is to not allow the pyarrow extension types which awkward defines)

In [2]: ak.to_arrow(layout, extensionarray=False)
Out[2]:
<pyarrow.lib.FixedSizeListArray object at 0x109f8d6c0>
[
  [],
  [],
  ...
  [],
  []
]

In [3]: ak.to_arrow_table(layout, extensionarray=False)
Out[3]:
pyarrow.Table
: fixed_size_list<item: float not null>[0] not null
  child 0, item: float not null
----
: [[[],[],...,[],[]]]

In [5]: ak.to_parquet(layout, "tmp.parquet", extensionarray=False)
Out[5]:
<pyarrow._parquet.FileMetaData object at 0x10a09fb00>
  created_by: parquet-cpp-arrow version 23.0.1
  num_columns: 1
  num_rows: 50
  num_row_groups: 1
  format_version: 2.6
  serialized_size: 0

Then I couldn't read this parquet file with the error mentioned above

ArrowInvalid: list_size needs to be a strict positive integer

So I'm wondering, is there something fundamentally wrong with list_size=0? If so, pyarrow should error when one tries to save it to parquet and we should also error in awkward array. Otherwise, if it's fine, It's the reading that should be fixed IMO.