PyArrow StringDtype / StringArray fallback policy (original) (raw)

In #35169 / #42597 we discussed the desired behavior of PyArrow-backed StringArray when a certain method is not implemented in pyarrow.compute.

For string methods like str_normalize, which aren't currently implemented in pyarrow.compute, I believe we (silently) cast from Pyarrow[string] to an object-dtype ndarray of Python str objects at

. That's going to be slow and more than doubles the memory usage of the array.

These kinds of performance cliffs are difficult for users to debug. I don't think we should do that conversion on behalf of the user. If something isn't implemented yet, then I think we should raise with a message saying they should convert to string[python] dtype first.

If we don't want to raise, we could emit a PerformanceWarning, similar to what we do for SparseArray when converting to dense.