GH-48978: [Python] test failures on pandas 3.0 for fastparquet and for zoneinfo w/o pytz

[tadeja]

Rationale for this change

Closes #48978

What changes are included in this PR?

Update to parquet/test_basic.py test_fastparquet_cross_compatibility for fastparquet string and categorical dtype differences causing failure Attribute "dtype" are different
Update to test_pandas.py‎ test_timestamp_as_object_non_nanosecond for failure ValueError: fromutc: dt.tzinfo is not self.

Are these changes tested?

Yes. Initially tested locally with pandas upgraded to 3.0 as CI was still running with pandas 2.3.3 cached.

Are there any user-facing changes?

No.

• GitHub Issue: [Python] test failures on pandas 3.0 for fastparquet and for zoneinfo w/o pytz #48978

[github-actions]

⚠️ GitHub issue #48978 has been automatically assigned in GitHub to PR creator.

[tadeja]

Failures replicated here on CI job AMD64 Conda Python 3.13 Pandas latest
with temporary CI modification as follows:
pip install --upgrade pandas fastparquet
pip uninstall -q -y pytz

[tadeja]

Proposed fixes complete previously failing tests with success, you can see it is with pandas 3.0, with fastparquet and without pytz here:
AMD64 Conda Python 3.13 Pandas latest succeeded

• fastparquet 2025.12.0 pypi_0 pypi
• pandas 3.0.0 pypi_0 pypi
• ...
• = 7784 passed, 283 skipped, 22 xfailed, 2 xpassed, 54 warnings in 241.86s (0:04:01) =

[h-vetinari]

I backported this to the conda-forge feedstock in conda-forge/pyarrow-feedstock#169, and can confirm that it works! Thanks!

[AlenkaF]

Thank you for the fix @tadeja! I am adding my comments bellow.

[AlenkaF]

Would it make sense to use integers for categories until fastparquet fully supports pandas 3.0? That way we can also check the dtype roundtrip here.

[tadeja]

Ok, @AlenkaF here's the new approach with
"f": pd.Categorical([5, 6, 7]),

This fails the test at

tm.assert_frame_equal(table_fp.to_pandas(), df_for_fp, check_dtype=False)
  pyarrow/tests/parquet/test_basic.py:769:
...
...
E           AssertionError: Categorical Expected type <class 'pandas.Categorical'>, found <class 'numpy.ndarray'> instead

so we have to add another check_categorical=False on line 769
as already done before at line 756 for the same reason.

[tadeja]

@AlenkaF, @rok this now includes comments and # TODO: once fastparquet supports pandas 3.0 dtypes revert string/categorical test - also according to #48978 (comment)

[rok]

Just a comment suggestion.

[rok]

Timezone part looks good. Suggesting a more readable fastparquet workaround. Proposed changes are not tested.

[rok]

Would this work instead? Since underlying type of categorical is not int it feels like it should?

Suggested change

	# TODO: once fastparquet supports pandas 3 dtypes revert string and categorical
	# tests by removing `check_dtype=False` and `check_categorical=False` so type
	# equality is asserted again
	tm.assert_frame_equal(df, df_fp, check_dtype=False, check_categorical=False)
	# TODO: once fastparquet supports pandas 3's pd.StringDtype() remove casting
	expected_types = {"a": pd.StringDtype()}
	tm.assert_frame_equal(df, df_fp.astype(expected_types))

[rok]

Perhaps we can avoid creating a new dataframe and only non-destructively cast at write-time?

Suggested change

	# fastparquet can't write pandas 3.0 StringDtype
	df_for_fp = df.copy()
	df_for_fp['a'] = df_for_fp['a'].astype(object)
	fp.write(file_fastparquet, df_for_fp)
	# fastparquet can't write pandas 3.0 StringDtype
	fp_compatible_types = {"a": object}
	fp.write(file_fastparquet, df.astype(fp_compatible_types))

[rok]

Maybe:

Suggested change

	tm.assert_frame_equal(table_fp.to_pandas(), df_for_fp, check_dtype=False,
	check_categorical=False)
	expected_types = {"a": pd.StringDtype(), "f": object}
	tm.assert_frame_equal(table_fp.to_pandas(), df.astype(expected_types))