ENH: Make maybe_convert_object respect dtype itemsize by rhshadrach · Pull Request #40908 · pandas-dev/pandas (original) (raw)
This adds support for e.g. float32 NumPy dtypes to maybe_convert_object. If any non-NumPy scalar is hit, the behavior is the same as master. This is my first foray into the NumPy C-API, so any tips are appreciated. In particular, I couldn't figure out how to use the C API to do the cast:
result = result.astype(result.dtype.kind + str(itemsize))
Not sure if there should also be specific logic for EAs/nullable types.
before after ratio
[7d4757b4] [c1288962]
<maybe_convert_object_itemsize~12> <maybe_convert_object_itemsize>
+ 768±8μs 1.01±0ms 1.32 ctors.SeriesConstructors.time_series_constructor(<class 'list'>, False, 'float')
+ 779±9μs 1.02±0ms 1.31 ctors.SeriesConstructors.time_series_constructor(<class 'list'>, True, 'float')
+ 579±5μs 710±20μs 1.23 arithmetic.NumericInferOps.time_divide(<class 'numpy.int8'>)
+ 608±30μs 715±20μs 1.18 arithmetic.NumericInferOps.time_divide(<class 'numpy.uint8'>)
+ 3.18±0.2μs 3.69±0.3μs 1.16 index_cached_properties.IndexCache.time_engine('UInt64Index')
+ 1.58±0.01ms 1.83±0.01ms 1.16 ctors.SeriesConstructors.time_series_constructor(<function arr_dict at 0x7f5964729820>, False, 'float')
+ 3.30±0.2μs 3.80±0.5μs 1.15 index_cached_properties.IndexCache.time_engine('TimedeltaIndex')
+ 943±80ns 1.08±0.1μs 1.15 index_cached_properties.IndexCache.time_inferred_type('Float64Index')
+ 1.73±0.01ms 1.98±0ms 1.14 ctors.SeriesConstructors.time_series_constructor(<class 'list'>, True, 'int')
+ 1.72±0.01ms 1.96±0ms 1.14 ctors.SeriesConstructors.time_series_constructor(<class 'list'>, False, 'int')
+ 5.19±0.03μs 5.93±0.3μs 1.14 tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1, 6000, datetime.timezone(datetime.timedelta(seconds=3600)))
+ 1.65±0.01ms 1.89±0.01ms 1.14 ctors.SeriesConstructors.time_series_constructor(<function arr_dict at 0x7f5964729820>, True, 'float')
+ 6.06±0.06μs 6.91±0.1μs 1.14 tslibs.tz_convert.TimeTZConvert.time_tz_convert_from_utc(1, datetime.timezone(datetime.timedelta(seconds=3600)))
+ 7.46±0.04μs 8.47±0.5μs 1.14 tslibs.offsets.OffestDatetimeArithmetic.time_subtract(<BusinessDay>)
+ 1.15±0.09μs 1.27±0.08μs 1.11 index_cached_properties.IndexCache.time_values('UInt64Index')
- 11.7±1ms 10.5±0.1ms 0.90 algos.isin.IsinAlmostFullWithRandomInt.time_isin(<class 'numpy.object_'>, 18, 'outside')
- 641±30ns 577±20ns 0.90 index_cached_properties.IndexCache.time_is_monotonic('RangeIndex')
- 641±20ns 576±20ns 0.90 index_cached_properties.IndexCache.time_shape('Int64Index')
- 3.37±0.03ms 3.02±0.02ms 0.90 timeseries.ResampleSeries.time_resample('period', '5min', 'ohlc')
- 787±20ns 702±20ns 0.89 index_cached_properties.IndexCache.time_is_monotonic_decreasing('RangeIndex')
- 1.25±0.04μs 1.12±0.04μs 0.89 index_cached_properties.IndexCache.time_is_all_dates('Int64Index')
- 1.50±0.1ms 1.33±0.02ms 0.88 dtypes.SelectDtypes.time_select_dtype_string_exclude('float32')
- 1.19±0.09μs 1.05±0.08μs 0.88 index_cached_properties.IndexCache.time_inferred_type('UInt64Index')
- 57.6±1μs 50.3±0.2μs 0.87 frame_methods.Dtypes.time_frame_dtypes
- 168±3μs 146±0.6μs 0.87 algos.isin.IsinWithArangeSorted.time_isin(<class 'numpy.uint64'>, 8000)
- 552±20ns 480±20ns 0.87 index_cached_properties.IndexCache.time_is_monotonic_increasing('RangeIndex')
- 1.50±0.1ms 1.30±0ms 0.87 dtypes.SelectDtypes.time_select_dtype_string_exclude('complex64')
- 1.89±0ms 1.64±0.01ms 0.87 period.DataFramePeriodColumn.time_set_index
- 11.0±0.2μs 9.50±0.08μs 0.87 period.Indexing.time_series_loc
- 110±20ms 93.1±0.06ms 0.85 algos.isin.IsInLongSeriesLookUpDominates.time_isin('float32', 1000, 'random_hits')
- 392±30ns 330±10ns 0.84 index_cached_properties.IndexCache.time_inferred_type('RangeIndex')
- 4.45±0.4ms 3.68±0.02ms 0.83 algorithms.Factorize.time_factorize(True, True, 'Int64')
- 9.60±0.8ms 7.75±0.01ms 0.81 algorithms.Factorize.time_factorize(True, False, 'string')
- 11.2±0.8ms 8.90±0.1ms 0.80 algos.isin.IsinAlmostFullWithRandomInt.time_isin(<class 'numpy.object_'>, 18, 'inside')
- 615±30ns 484±30ns 0.79 index_cached_properties.IndexCache.time_is_monotonic_increasing('Int64Index')
- 73.8±20ms 55.4±0.5ms 0.75 algos.isin.IsInLongSeriesLookUpDominates.time_isin('float32', 1000, 'monotone_hits')
- 180±0.1μs 128±0.5μs 0.71 indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.Int8Engine'>, <class 'numpy.int8'>), 'monotonic_incr')
- 128±40ms 90.8±0.03ms 0.71 algos.isin.IsInLongSeriesLookUpDominates.time_isin('object', 5, 'monotone_hits')
- 186±0.3μs 131±2μs 0.70 indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.Int16Engine'>, <class 'numpy.int16'>), 'monotonic_incr')
- 1.72±0.4μs 1.17±0.1μs 0.68 index_cached_properties.IndexCache.time_inferred_type('TimedeltaIndex')
- 216±0.8μs 140±3μs 0.65 indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.UInt32Engine'>, <class 'numpy.uint32'>), 'monotonic_incr')
- 1.55±0.01ms 972±7μs 0.63 algos.isin.IsinWithArangeSorted.time_isin(<class 'numpy.int64'>, 100000)
SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE DECREASED.
np.array of python integers:
127 ms ± 80.9 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) <--- PR
120 ms ± 155 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) <--- master
np.array of int32:
179 ms ± 1.21 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) <--- PR
133 ms ± 274 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) <--- master
np.array of int32 with last one a python int:
178 ms ± 1.71 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) <--- PR
133 ms ± 106 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) <--- master