ENH: use native filesystem (if available) for read_parquet with pyarrow engine by mroeschke · Pull Request #51609 · pandas-dev/pandas (original) (raw)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I was under the impression that ArrowFSWrapper would be interpreted as an pyarrow filesystem since that's the backend from https://arrow.apache.org/docs/python/filesystems.html#using-arrow-filesystems-with-fsspec)

That section is for if you want to use a pyarrow filesystem in an fsspec context (for example for passing it to a function that expects an fsspec filesystem, or to use the richer API it provives). But in this case we are not in such a context, since we are passing the filesystem to a pyarrow function.

It doesn't really say anything about then using that wrapper in a pyarrow context, but as mentioned above, we can certainly unwrap this inside pyarrow, avoiding the confusion / wrong expectation you just had (but so currently we don't do that yet).