ENH: Pandas 2.0 with pyarrow engine add the argument like 'skip_bad_lines=True' · Issue #54480 · pandas-dev/pandas (original) (raw)

Feature Type

• Adding new functionality to pandas
• Changing existing functionality in pandas
• Removing existing functionality in pandas

Problem Description

When I use Pandas 2.0 with pyarrow engine to read CSV file, if the file get some bad lines, it will raise a error such as "pyarrow.lib.ArrowInvalid: CSV parse error: Expected 1 columns, got 2: ".

Then I added the argument '', it will raise another error "TypeError: read_csv() got an unexpected keyword argument 'skip_bad_lines'"

So,can you add the argument like 'skip_bad_lines=True' with pyarrow engine.

df_arrow = pd.read_csv(r"VBAP.CSV", engine='pyarrow', dtype_backend='pyarrow')
df_arrow = pd.read_csv(r"VBAP.CSV", engine='pyarrow', dtype_backend='pyarrow', skip_bad_lines=True)

Feature Description

df_arrow = pd.read_csv(r"VBAP.CSV", engine='pyarrow', dtype_backend='pyarrow', skip_bad_lines=True)

Alternative Solutions

df_arrow = pd.read_csv(r"VBAP.CSV", engine='pyarrow', dtype_backend='pyarrow', skip_bad_lines=True)

Additional Context

No response