BUG: Series.where behaves inconsistent · Issue #39595 · pandas-dev/pandas (original) (raw)

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd df1 = pd.DataFrame([1, 2, 3, 4], columns=['w']) df2 = pd.DataFrame([1, 2, 3, 4], columns=['w']) c1 = df1['w'].where(df1['w'] < 4, 4) c2 = df2['w'].where(df2['w'] < 5) c1 *= 2 c2 *= 2 assert (c1 == c2).all() assert (df1['w'] == df2['w']).all() # df2 got changed, so assertion fails Traceback (most recent call last): File "", line 1, in AssertionError

df1 w 0 1 1 2 2 3 3 4 df2 w 0 2 1 4 2 6 3 8

Problem description

The behavior is inconsistent: in case .where is used to select all elements in the series, a reference to the original data is returned instead of a copy. This way it is possible to modify the original dataframe/series - but only when all elements are selected.
In the example code, c1 is identical to c2, however changing c2 (which is created by selecting all elements) changes the parent dataframe.

Prior to 1.1.4 .where always returned a copy: checked 0.25.1, 1.1.0, 1.1.1, 1.1.2, 1.1.3 and all of them created a copy. This has been changed in 1.1.4, and 1.1.5 as well as 1.2.1 still has this behaviour.

Expected Output

There is no assertion, i.e. .where returns a copy consistently.

Please note that the documentation is unclear on this, it might worth explicitly mentioning whether or not copying is expected.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 67a3d42
python : 3.7.9.final.0
python-bits : 64
OS : Linux
OS-release : 4.20.13-1.el7.elrepo.x86_64
Version : #1 SMP Wed Feb 27 10:02:05 EST 2019
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.4
numpy : 1.19.4
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.4
setuptools : 47.3.1.post20201218
Cython : 0.29.21
pytest : 5.4.3
hypothesis : 5.30.0
sphinx : 3.0.3
blosc : None
feather : None
xlsxwriter : 1.2.9
lxml.etree : 4.5.2
html5lib : 1.1
pymysql : 0.10.1
psycopg2 : 2.8.6 (dt dec pq3 ext lo64)
jinja2 : 2.11.2
IPython : 7.17.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : 1.3.2
fsspec : 0.8.3
fastparquet : None
gcsfs : None
matplotlib : 3.3.2
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.5
pandas_gbq : None
pyarrow : 2.0.0
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.4
sqlalchemy : 1.3.20
tables : 3.6.1
tabulate : 0.8.7
xarray : 0.15.1
xlrd : 1.2.0
xlwt : 1.3.0
numba : 0.51.2