[C++] Segmentation fault in pyarrow.compute.if_else: arguments have string type but the result only fits in large_string type

[kccqzy]

Describe the bug, including details regarding any error messages, version, and platform.

Run the following Python code:

import pyarrow
t = pyarrow.repeat('RANDOM_STR', 10**8)
f = pyarrow.repeat('RANDOM_STR_LARGE', 10**8)
m = pyarrow.repeat(False, 10**8)
pyarrow.compute.if_else(m, t, f)

This segfaults almost immediately on my computer.

The error message is Fatal Python error: Segmentation fault.

The cause of the issue that the arguments have string type but the result does not fit in the string type. If one were to cast the argument from string to large_string prior to calling if_else then it works.

This should either raise a proper exception in Python or silently do the cast for the user. It should not segfault.

Component(s)

Python

[kccqzy]

Stack trace:

Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
__memmove_evex_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:383
383             VMOVU   %XMM0, (%rdi)
(gdb) bt
#0  __memmove_evex_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:383
#1  0x00007fffd9c259dc in arrow::compute::internal::(anonymous namespace)::ResolveIfElseExec<arrow::BinaryType, std::integral_constant<bool, true> >::Exec(arrow::compute::KernelContext*, arrow::compute::ExecSpan const&, arrow::compute::ExecResult*) ()
#2  0x00007fffd9fa128f in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::Execute(arrow::compute::ExecBatch const&, arrow::compute::detail::ExecListener*) () 
#3  0x00007fffd9fb991b in arrow::compute::detail::FunctionExecutorImpl::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, long) () 
#4  0x00007fffd9f87539 in arrow::compute::(anonymous namespace)::ExecuteInternal(arrow::compute::Function const&, std::vector<arrow::Datum, std::allocator<arrow::Datum> >, long, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) ()
#5  0x00007fffd9f87847 in arrow::compute::Function::Execute(std::vector<arrow::Datum, std::allocator<arrow::Datum> > const&, arrow::compute::FunctionOptions const*, arrow::compute::ExecContext*) const () 

[vanshaj2023]

Hi @kccqzy
I'd like to work on this issue.

Implementation Approach:

The segfault occurs in pyarrow.compute.if_else when the result exceeds string type capacity (2GB limit) but arguments are string type. I plan to:

1. Add validation in the if_else kernel to detect when output size would exceed string type limits
2. Automatically promote string → large_string when necessary, similar to how other Arrow kernels handle type promotion
3. Add proper error handling to raise ArrowInvalid exception instead of segfaulting if auto-promotion isn't feasible

I'll look into the C++ implementation of the if_else kernel (likely in cpp/src/arrow/compute/kernels/scalar_if_else.cc) to add size checks and type promotion logic.

Questions:

• Does this approach sound correct?
• Should we auto-promote to large_string or raise an exception and let users explicitly cast?
• Any specific areas of the codebase I should focus on?

Could you please assign this issue to me if the approach looks good?

Thanks!

[pitrou]

Thanks for the report @kccqzy ! Some checks are definitely missing.

@vanshaj2023 I've said it on another issue, but please don't post such AI-generated messages on the tracker. The more you do that, the less you'll get heard. Does that sound ok?

[pitrou]

cc @zanmato1984

[Anakin100100]

I'd like to work on this