Allow writing Variant to files and test parquet-variant IO

[AdamGS]

Summary

Integrate Variant better into the compressor so that it can be used with the default write strategy.

[codspeed-hq]

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 1235 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	chunked_varbinview_canonical_into[(100, 100)]	273.1 µs	307.8 µs	-11.27%
⚡	Simulation	chunked_varbinview_opt_canonical_into[(1000, 10)]	224.9 µs	187.5 µs	+19.9%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing adamg/variant-to-file (93cc893) with develop (19a1fb3)}

hold on a sec

[robert3005]

Why do you explicitly enumare the slots here instead of allowing the array to decide? Shouldn't this be

self.compress(variant_array.core_storage(), exec_ctx)
variant_array
                    .shredded()
                    .map(|arr| self.compress(arr, exec_ctx))
                    .transpose()?

[AdamGS]

IIRC because core_storage is potentially recursive, you end up with a stack overflow here.

[AdamGS]

canonicalization for variant does something similar? this structure introduces some complexity for this sort of stuff.

[robert3005]

core_storage must be variant but I think shredded does not.

[AdamGS]

pushed something, WDYT?

[robert3005]

I think you want similar fix in the canonical code path

[AdamGS]

I agree, I have some other fixes there in the works as part of a separate PR.

[AdamGS]

@robert3005 I've fixed the canonicalization issue if you want to take another look.