PyArrow StringDtype / StringArray fallback policy

[TomAugspurger]

In #35169 / #42597 we discussed the desired behavior of PyArrow-backed StringArray when a certain method is not implemented in pyarrow.compute.

For string methods like str_normalize, which aren't currently implemented in pyarrow.compute, I believe we (silently) cast from Pyarrow[string] to an object-dtype ndarray of Python str objects at

pandas/pandas/core/arrays/string_.py

Lines 527 to 528 in edd5af7

	mask = isna(self)
	arr = np.asarray(self)

. That's going to be slow and more than doubles the memory usage of the array.

These kinds of performance cliffs are difficult for users to debug. I don't think we should do that conversion on behalf of the user. If something isn't implemented yet, then I think we should raise with a message saying they should convert to string[python] dtype first.

If we don't want to raise, we could emit a PerformanceWarning, similar to what we do for SparseArray when converting to dense.

[simonjayhawkins]

maybe also consider a nopython=True/False setting (pandas option, dtype constructor argument and/or array attribute)

[TomAugspurger]

Yep, those would be worth exploring. Global options are a bit harder for dask, since it's a bit hard to propagate global options to multiple processes (it'd be easier if we had a config system like Dask's). A per- StringArray / StringDtype instance option might also be useful, though we'd need to propagate that setting through operations which sounds a bit hard.

[mroeschke]

I think the nicest behavior for the users would be for operations to "just work" with a fallback method, so I would be partial to raise a PerformanceWarning when the fallback happens. I think the big assumption here is that the fallback method can always produce the same result as the pyarrow compute method.

The easiest thing for us to do is probably raise telling user to convert, but one inconvenience for users is that they to revert their conversions to string[python] in their code base if they ever upgrade pyarrow to access more performant methods.

So I think for users

• User's codebase calls foo() and gets PerformanceWarning and a result
• User upgrades Pyarrow
• User's codebase calls foo() and gets a performant result

is a better experience than

• User's codebase calls foo() and gets NotImplementedError and told to convert to string[python]
• User's codebase calls .astype("string[python]").foo() and gets a result
• User upgrades Pyarrow
• User needs to remove .astype("string[python]") from their codebase to get a performant result