ENH: RangeIndex._shallow_copy can return RangeIndex by mroeschke · Pull Request #47557 · pandas-dev/pandas (original) (raw)
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Conversation10 Commits11 Checks0 Files changed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
[ Show hidden characters]({{ revealButtonHref }})
- closes Index is being materialized in pd.concat when axis=1 #46675 (Replace xxxx with the Github issue number)
- closes PERF: Index.union is materializing even when it is not needed when sort=False #43885
- Tests added and passed if fixing a bug or adding a new feature
- All code checks passed.
- Added an entry in the latest
doc/source/whatsnew/v1.5.0.rstfile if fixing a bug or adding a new feature.
Related to the Index class or subclasses
label
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
asv's prove out the change in perf?
asv's prove out the change in perf?
The memory perf will just be evident in the result
In [1]: >>> idx1 = pd.RangeIndex(0, 100_000)
...: >>> idx2 = pd.RangeIndex(0, 50_000)
# main
In [2]: idx1.union(idx2, sort=False).memory_usage()
Out[2]: 800000
# PR
In [2]: idx1.union(idx2, sort=False).memory_usage()
Out[2]: 128
idx1.union
no objection to this PR, but ideally this case would be caught in _union to prevent ever materializing the ndarray
idx1.union
no objection to this PR, but ideally this case would be caught in _union to prevent ever materializing the ndarray
Added logic to do this in 63e7894, but I think the change in _shallow_copy is still needed because concat goes though there
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and asv's still hold here (after latest changes)?
and asv's still hold here (after latest changes)?
% asv continuous -f 1.1 upstream/main HEAD -b index_object.Range
· Creating environments.........................
· Discovering benchmarks.
·· Uninstalling from conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.
·· Building 498b94f7 <enh/range_shallowcopy> for conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...........................................................................
·· Installing 498b94f7 <enh/range_shallowcopy> into conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..
· Running 20 total benchmarks (2 commits * 1 environments * 10 benchmarks)
[ 0.00%] · For pandas commit 8e6ca289 <main> (round 1/2):
[ 0.00%] ·· Building for conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.........................................................................
[ 0.00%] ·· Benchmarking conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 2.50%] ··· Running (index_object.Range.time_get_loc_dec--)..........
[ 25.00%] · For pandas commit 498b94f7 <enh/range_shallowcopy> (round 1/2):
[ 25.00%] ·· Building for conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..
[ 25.00%] ·· Benchmarking conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 27.50%] ··· Running (index_object.Range.time_get_loc_dec--)..........
[ 50.00%] · For pandas commit 498b94f7 <enh/range_shallowcopy> (round 2/2):
[ 50.00%] ·· Benchmarking conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 52.50%] ··· index_object.Range.time_get_loc_dec 554±100ns
[ 55.00%] ··· index_object.Range.time_get_loc_inc 509±50ns
[ 57.50%] ··· index_object.Range.time_iter_dec 14.4±2ms
[ 60.00%] ··· index_object.Range.time_iter_inc 15.9±1ms
[ 62.50%] ··· index_object.Range.time_max 1.20±0.09μs
[ 65.00%] ··· index_object.Range.time_max_trivial 1.10±0.2μs
[ 67.50%] ··· index_object.Range.time_min 1.24±0.1μs
[ 70.00%] ··· index_object.Range.time_min_trivial 1.48±0.3μs
[ 72.50%] ··· index_object.Range.time_sort_values_asc 2.73±0.2μs
[ 75.00%] ··· index_object.Range.time_sort_values_des 4.48±2μs
[ 75.00%] · For pandas commit 8e6ca289 <main> (round 2/2):
[ 75.00%] ·· Building for conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[ 75.00%] ·· Benchmarking conda-py3.8-Cython0.29.30-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pyarrow-pytables-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 77.50%] ··· index_object.Range.time_get_loc_dec 550±60ns
[ 80.00%] ··· index_object.Range.time_get_loc_inc 506±100ns
[ 82.50%] ··· index_object.Range.time_iter_dec 12.8±1ms
[ 85.00%] ··· index_object.Range.time_iter_inc 13.9±3ms
[ 87.50%] ··· index_object.Range.time_max 1.34±0.2μs
[ 90.00%] ··· index_object.Range.time_max_trivial 1.13±0.06μs
[ 92.50%] ··· index_object.Range.time_min 1.19±0.3μs
[ 95.00%] ··· index_object.Range.time_min_trivial 1.09±0.08μs
[ 97.50%] ··· index_object.Range.time_sort_values_asc 3.03±0.8μs
[100.00%] ··· index_object.Range.time_sort_values_des 5.22±0.5μs
BENCHMARKS NOT SIGNIFICANTLY CHANGED.
yehoshuadimarsky pushed a commit to yehoshuadimarsky/pandas that referenced this pull request
Labels
Related to the Index class or subclasses