What’s new in 1.2.0 (December 26, 2020) — pandas 3.0.0rc0+33.g1fd184de2a documentation (original) (raw)

These are the changes in pandas 1.2.0. See Release notes for a full changelog including other versions of pandas.

Warning

The xlwt package for writing old-style .xlsexcel files is no longer maintained. The xlrd package is now only for reading old-style .xls files.

Previously, the default argument engine=None to read_excel()would result in using the xlrd engine in many cases, including new Excel 2007+ (.xlsx) files. If openpyxl is installed, many of these cases will now default to using the openpyxl engine. See the read_excel() documentation for more details.

Thus, it is strongly encouraged to install openpyxl to read Excel 2007+ (.xlsx) files.**Please do not report issues when using ``xlrd`` to read ``.xlsx`` files.**This is no longer supported, switch to using openpyxl instead.

Attempting to use the xlwt engine will raise a FutureWarningunless the option io.excel.xls.writer is set to "xlwt". While this option is now deprecated and will also raise a FutureWarning, it can be globally set and the warning suppressed. Users are recommended to write .xlsx files using the openpyxl engine instead.

Enhancements#

Optionally disallow duplicate labels#

Series and DataFrame can now be created with allows_duplicate_labels=False flag to control whether the index or columns can contain duplicate labels (GH 28394). This can be used to prevent accidental introduction of duplicate labels, which can affect downstream operations.

By default, duplicates continue to be allowed.

In [1]: pd.Series([1, 2], index=['a', 'a']) Out[1]: a 1 a 2 Length: 2, dtype: int64

In [2]: pd.Series([1, 2], index=['a', 'a']).set_flags(allows_duplicate_labels=False) ... DuplicateLabelError: Index has duplicates. positions label a [0, 1]

pandas will propagate the allows_duplicate_labels property through many operations.

In [3]: a = ( ...: pd.Series([1, 2], index=['a', 'b']) ...: .set_flags(allows_duplicate_labels=False) ...: )

In [4]: a Out[4]: a 1 b 2 Length: 2, dtype: int64

An operation introducing duplicates

In [5]: a.reindex(['a', 'b', 'a']) ... DuplicateLabelError: Index has duplicates. positions label a [0, 2]

[1 rows x 1 columns]

Warning

This is an experimental feature. Currently, many methods fail to propagate the allows_duplicate_labels value. In future versions it is expected that every method taking or returning one or more DataFrame or Series objects will propagate allows_duplicate_labels.

See Duplicate Labels for more.

The allows_duplicate_labels flag is stored in the new DataFrame.flagsattribute. This stores global attributes that apply to the pandas object. This differs from DataFrame.attrs, which stores information that applies to the dataset.

Passing arguments to fsspec backends#

Many read/write functions have acquired the storage_options optional argument, to pass a dictionary of parameters to the storage backend. This allows, for example, for passing credentials to S3 and GCS storage. The details of what parameters can be passed to which backends can be found in the documentation of the individual storage backends (detailed from the fsspec docs forbuiltin implementations and linked to external ones). See Section Reading/writing remote files.

GH 35655 added fsspec support (including storage_options) for reading excel files.

Support for binary file handles in to_csv#

to_csv() supports file handles in binary mode (GH 19827 and GH 35058) with encoding (GH 13068 and GH 23854) and compression (GH 22555). If pandas does not automatically detect whether the file handle is opened in binary or text mode, it is necessary to provide mode="wb".

For example:

In [1]: import io

In [2]: data = pd.DataFrame([0, 1, 2])

In [3]: buffer = io.BytesIO()

In [4]: data.to_csv(buffer, encoding="utf-8", compression="gzip")

Support for short caption and table position in to_latex#

DataFrame.to_latex() now allows one to specify a floating table position (GH 35281) and a short caption (GH 36267).

The keyword position has been added to set the position.

In [5]: data = pd.DataFrame({'a': [1, 2], 'b': [3, 4]})

In [6]: table = data.to_latex(position='ht')

In [7]: print(table) \begin{table}[ht] \begin{tabular}{lrr} \toprule & a & b \ \midrule 0 & 1 & 3 \ 1 & 2 & 4 \ \bottomrule \end{tabular} \end{table}

Usage of the keyword caption has been extended. Besides taking a single string as an argument, one can optionally provide a tuple (full_caption, short_caption)to add a short caption macro.

In [8]: data = pd.DataFrame({'a': [1, 2], 'b': [3, 4]})

In [9]: table = data.to_latex(caption=('the full long caption', 'short caption'))

In [10]: print(table) \begin{table} \caption[short caption]{the full long caption} \begin{tabular}{lrr} \toprule & a & b \ \midrule 0 & 1 & 3 \ 1 & 2 & 4 \ \bottomrule \end{tabular} \end{table}

Change in default floating precision for read_csv and read_table#

For the C parsing engine, the methods read_csv() and read_table() previously defaulted to a parser that could read floating point numbers slightly incorrectly with respect to the last bit in precision. The option floating_precision="high" has always been available to avoid this issue. Beginning with this version, the default is now to use the more accurate parser by makingfloating_precision=None correspond to the high precision parser, and the new optionfloating_precision="legacy" to use the legacy parser. The change to using the higher precision parser by default should have no impact on performance. (GH 17154)

Experimental nullable data types for float data#

We’ve added Float32Dtype / Float64Dtype and FloatingArray. These are extension data types dedicated to floating point data that can hold thepd.NA missing value indicator (GH 32265, GH 34307).

While the default float data type already supports missing values using np.nan, these new data types use pd.NA (and its corresponding behavior) as the missing value indicator, in line with the already existing nullable integerand boolean data types.

One example where the behavior of np.nan and pd.NA is different is comparison operations:

the default NumPy float64 dtype

In [11]: s1 = pd.Series([1.5, None])

In [12]: s1 Out[12]: 0 1.5 1 NaN dtype: float64

In [13]: s1 > 1 Out[13]: 0 True 1 False dtype: bool

the new nullable float64 dtype

In [14]: s2 = pd.Series([1.5, None], dtype="Float64")

In [15]: s2 Out[15]: 0 1.5 1 dtype: Float64

In [16]: s2 > 1 Out[16]: 0 True 1 dtype: boolean

See the NA semantics doc section for more details on the behavior when using the pd.NA missing value indicator.

As shown above, the dtype can be specified using the “Float64” or “Float32” string (capitalized to distinguish it from the default “float64” data type). Alternatively, you can also use the dtype object:

In [17]: pd.Series([1.5, None], dtype=pd.Float32Dtype()) Out[17]: 0 1.5 1 dtype: Float32

Operations with the existing integer or boolean nullable data types that give float results will now also use the nullable floating data types (GH 38178).

Warning

Experimental: the new floating data types are currently experimental, and their behavior or API may still change without warning. Especially the behavior regarding NaN (distinct from NA missing values) is subject to change.

Index/column name preservation when aggregating#

When aggregating using concat() or the DataFrame constructor, pandas will now attempt to preserve index and column names whenever possible (GH 35847). In the case where all inputs share a common name, this name will be assigned to the result. When the input names do not all agree, the result will be unnamed. Here is an example where the index name is preserved:

In [18]: idx = pd.Index(range(5), name='abc')

In [19]: ser = pd.Series(range(5, 10), index=idx)

In [20]: pd.concat({'x': ser[1:], 'y': ser[:-1]}, axis=1) Out[20]: x y abc
1 6.0 6.0 2 7.0 7.0 3 8.0 8.0 4 9.0 NaN 0 NaN 5.0

The same is true for MultiIndex, but the logic is applied separately on a level-by-level basis.

GroupBy supports EWM operations directly#

DataFrameGroupBy now supports exponentially weighted window operations directly (GH 16037).

In [21]: df = pd.DataFrame({'A': ['a', 'b', 'a', 'b'], 'B': range(4)})

In [22]: df Out[22]: A B 0 a 0 1 b 1 2 a 2 3 b 3

In [23]: df.groupby('A').ewm(com=1.0).mean() Out[23]: B A
a 0 0.000000 2 1.333333 b 1 1.000000 3 2.333333

Additionally mean supports execution via Numba with the engine and engine_kwargs arguments. Numba must be installed as an optional dependency to use this feature.

Other enhancements#

Notable bug fixes#

These are bug fixes that might have notable behavior changes.

Consistency of DataFrame Reductions#

DataFrame.any() and DataFrame.all() with bool_only=True now determines whether to exclude object-dtype columns on a column-by-column basis, instead of checking if all object-dtype columns can be considered boolean.

This prevents pathological behavior where applying the reduction on a subset of columns could result in a larger Series result. See (GH 37799).

In [24]: df = pd.DataFrame({"A": ["foo", "bar"], "B": [True, False]}, dtype=object)

In [25]: df["C"] = pd.Series([True, True])

Previous behavior:

In [5]: df.all(bool_only=True) Out[5]: C True dtype: bool

In [6]: df[["B", "C"]].all(bool_only=True) Out[6]: B False C True dtype: bool

New behavior:

In [26]: In [5]: df.all(bool_only=True) Out[26]: C True dtype: bool

In [27]: In [6]: df[["B", "C"]].all(bool_only=True) Out[27]: C True dtype: bool

Other DataFrame reductions with numeric_only=None will also avoid this pathological behavior (GH 37827):

In [28]: df = pd.DataFrame({"A": [0, 1, 2], "B": ["a", "b", "c"]}, dtype=object)

Previous behavior:

In [3]: df.mean() Out[3]: Series([], dtype: float64)

In [4]: df[["A"]].mean() Out[4]: A 1.0 dtype: float64

New behavior:

In [3]: df.mean() Out[3]: A 1.0 dtype: float64

In [4]: df[["A"]].mean() Out[4]: A 1.0 dtype: float64

Moreover, DataFrame reductions with numeric_only=None will now be consistent with their Series counterparts. In particular, for reductions where the Series method raises TypeError, the DataFrame reduction will now consider that column non-numeric instead of casting to a NumPy array which may have different semantics (GH 36076,GH 28949, GH 21020).

In [29]: ser = pd.Series([0, 1], dtype="category", name="A")

In [30]: df = ser.to_frame()

Previous behavior:

In [5]: df.any() Out[5]: A True dtype: bool

New behavior:

In [5]: df.any() Out[5]: Series([], dtype: bool)

Increased minimum version for Python#

pandas 1.2.0 supports Python 3.7.1 and higher (GH 35214).

Increased minimum versions for dependencies#

Some minimum supported versions of dependencies were updated (GH 35214). If installed, we now require:

For optional libraries the general recommendation is to use the latest version. The following table lists the lowest version per library that is currently being tested throughout the development of pandas. Optional libraries below the lowest tested version may still work, but are not considered supported.

See Dependencies and Optional dependencies for more.

Other API changes#

Deprecations#

Calling NumPy ufuncs on non-aligned DataFrames

Calling NumPy ufuncs on non-aligned DataFrames changed behaviour in pandas 1.2.0 (to align the inputs before calling the ufunc), but this change is reverted in pandas 1.2.1. The behaviour to not align is now deprecated instead, see the the 1.2.1 release notes for more details.

Performance improvements#

Bug fixes#

Categorical#

Datetime-like#

Timedelta#

Timezones#

Numeric#

Conversion#

Strings#

Interval#

Indexing#

Missing#

MultiIndex#

I/O#

Period#

Plotting#

Styler#

Groupby/resample/rolling#

Reshaping#

ExtensionArray#

Other#

Contributors#

A total of 257 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.