BUG: Python Engine does not respect chunksize when skiprows is specified as a nonempty list. · Issue #56323 · pandas-dev/pandas (original) (raw)

Pandas version checks

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

from io import StringIO

import pandas as pd

import pandas as pd csv = """col_a 0 1 2 3 4 5 6 7 8 9"""

text_file_reader = pd.read_csv(StringIO(csv), engine='python', skiprows=[1, 2, 3, 7, 10], chunksize=2) with text_file_reader as reader: for chunk in iter(reader): print(chunk)

Issue Description

Pandas returns a chunk of size 3 even though read_csv requests chunks of at most size=2.

Note: this behavior only occurs when both skiprows and chunksize are specified and skiprows is a non-empty list.

Expected Behavior

   col_a
0      3
1      4
   col_a
2      5
3      7
   col_a
4      8

The python engine provides chunks of at most size=2 when the parameter is specified with a nonempty list skiprows (mirroring c engine behavior).

Installed Versions

INSTALLED VERSIONS

commit : 0e2277b
python : 3.10.13.final.0
python-bits : 64
OS : Darwin
OS-release : 23.0.0
Version : Darwin Kernel Version 23.0.0: Thu Aug 17 21:24:46 PDT 2023; root:xnu-10002.1.11~3/RELEASE_ARM64_T6020
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8

pandas : 2.2.0.dev0+536.g0e2277b443
numpy : 1.24.4
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 68.2.2
pip : 23.3.1
Cython : 3.0.5
pytest : 7.4.3
hypothesis : 6.88.3
sphinx : 6.2.1
blosc : None
feather : None
xlsxwriter : 3.1.9
lxml.etree : 4.9.3
html5lib : 1.1
pymysql : 1.4.6
psycopg2 : 2.9.7
jinja2 : 3.1.2
IPython : 8.17.2
pandas_datareader : None
bs4 : 4.12.2
bottleneck : 1.3.7
dataframe-api-compat: None
fastparquet : 2023.10.1
fsspec : 2023.10.0
gcsfs : 2023.10.0
matplotlib : 3.7.3
numba : 0.57.1
numexpr : 2.8.7
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 14.0.0
pyreadstat : 1.2.4
python-calamine : None
pyxlsb : 1.0.10
s3fs : 2023.10.0
scipy : 1.11.3
sqlalchemy : 2.0.23
tables : 3.9.1
tabulate : 0.9.0
xarray : 2023.10.1
xlrd : 2.0.1
zstandard : 0.22.0
tzdata : 2023.3
qtpy : None
pyqt5 : None