pandas (original) (raw)

Pandas version checks

I have checked that this issue has not already been reported.
I have confirmed this issue exists on the latest version of pandas.
I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

A major bottleneck of Categorical.from_codes is the validation step where 'codes' is checked if all of its values are in range see source code.
The issue is that the check cannot be disabled. However, it is probably a common use-case for 'Categorical.from_codes' to be utilized as a raw constructor, i.e. to construct a categorical where it is known that the codes are already in a valid range. This check causes a significant slowdown for large datasets.

So it would be good to have a parameter which can be used to disable the validity checking (it is probably not a good idea to remove the check entirely). Ideally one would also make the following coecion step disableable (see)

One could add parameters copy and verify_integrity to Categorial.from_codes much in the same way as they are in pandas.MultiIndex constructor. As Categorical and MultiIndex are very similar semantically this would make sense IMO.

Background: I have a 17 GB dataset with a lot of categorical columns, where loading (with memory mapping) takes ~27 sec with present pandas, but without the Categorical.from_codes validation only 2 sec (I edited pandas source code for that). So more than a factor of 10 faster. Validation is not needed in this case as correctness is ensured when saing the data.

Example:

import mmap
import numpy
import time
import pandas

with open('codes.bin', 'w+b') as f:
    f.truncate(1024*1024*1024)
    map = mmap.mmap(f.fileno(), 1024*1024*1024, access=mmap.ACCESS_READ)
    codes = numpy.frombuffer(map, dtype=numpy.int8)
    codes.flags.writeable = False


start = time.time()
c = pandas.Categorical.from_codes(codes, categories=['a', 'b', 'c'] )
print( f"from_codes time {time.time()-start} sec" )

del c
del codes

map.close()

0.8 s present pandas (with check)
0.006 s (without 'code' check, this if disabled/removed )

Installed Versions

INSTALLED VERSIONS

commit : 2e218d1
python : 3.9.7.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : de_DE.cp1252

pandas : 1.5.3
numpy : 1.24.1
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 66.1.1
pip : 22.1.2
Cython : 0.29.33
pytest : 6.2.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None

Prior Performance