PERF: do DataFrame.op(series, axis=0) blockwise by jbrockmendel · Pull Request #31296 · pandas-dev/pandas (original) (raw)

Also fixes the same bug as #31271 (with the same test ported), so if this is accepted that will be closeable.

~2000x speedups for very-wide mixed-dtype cases.

arr = np.arange(10**6).reshape(100, -1)    
df = pd.DataFrame(arr)
df["C"] = 1.0
ser = df[0]

In [11]: %timeit df.eq(ser, axis=0)
1.58 s ± 20 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <-- master
595 µs ± 79.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  <-- PR

In [12]: %timeit df.add(ser, axis=0)
2.06 s ± 52.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <-- master                                                                                                                   
1.04 ms ± 5.07 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- PR