BUG: Chained urls (allowed by fsspec) not recognized by is_fsspec_url (original) (raw)

Pandas version checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandas.
• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd from pandas.io.common import is_fsspec_url import fsspec

This fsspec trick chains a cache fs with an s3 file system using "::".

See https://filesystem-spec.readthedocs.io/en/latest/features.html#url-chaining.

fn = "filecache::s3://ncei-wcsd-archive/data/processed/SH1305/18kHz/SaKe2013-D20130523-T080854_to_SaKe2013-D20130523-T085643.csv"

with fsspec.open(fn, storage_options={"s3": {"anon": True}}) as f: foo = pd.read_csv(f)

print(foo.shape)

But this url isn't recognized as a valid fsspec url by pandas...

print(pd.io.common.is_fsspec_url(fn))

...and so attempts to use chained url directly fail:

bar = pd.read_csv("filecache::s3://ncei-wcsd-archive/data/processed/SH1305/18kHz/" "SaKe2013-D20130523-T080854_to_SaKe2013-D20130523-T085643.csv", storage_options={"s3": {"anon": True}, "filecache":{"cache_storage":"/tmp/cache"}})

Issue Description

fsspec allows one to pass "chained" urls (useful, e.g., for caching), for example "filecache::https://example.com/my_file". However, since commit eeff2b0 meant to address issue #36271 supplying urls of this form to e.g., pd.read_csv has failed.

Expected Behavior

Pandas should pass the chained url to fsspec. In the example code, pd.Dataframes foo and bar should be identical.

Installed Versions

INSTALLED VERSIONS ------------------ commit : 87cfe4epython : 3.9.12.final.0 python-bits : 64 OS : Linux OS-release : 5.10.131-19115-g2e2fb0ed324d Version : #1 SMP PREEMPT Mon Sep 12 18:55:51 PDT 2022 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.5.0
numpy : 1.23.3
pytz : 2022.4
dateutil : 2.8.2
setuptools : 58.1.0
pip : 22.2.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.5.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : 2022.8.2
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : 2022.8.2
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None