DEPR: Some dropna behaviors in DataFrame.pivot_table · Issue #53521 · pandas-dev/pandas (original) (raw)

Currently dropna is used in four places within DataFrame.pivot_table:

1. It takes the cartesian product of all index/column levels when there are multiple levels; this was the original use
2. It is passed through to groupby
3. After the groupby aggregation, any rows that are all null are dropped
4. When computing the margins, rows in the original data where the keys and values are all null are dropped

1, 2, and 4 were all implemented for crosstab, which is essentially a call to pivot_table.

The API docs for crosstab document the dropna argument as:

Do not include columns whose entries are all NaN.

The only other documentation in the API and User Guide mentions using dropna=False to include rows/columns for categorical data with missing categorical values.

I think this is too much for a single Boolean argument to handle. I propose the following:

a. Add cartesian_product=[True|False] to pivot_table and crosstab
b. Add observed=[True|False] to crosstab for use with categoricals
c. Deprecate behavior (1) (with dropna), (3), and (4) above. The user may do each of these by dropping null values from the input data if they so desire.

We can implement (c) without affecting the behavior of crosstab by changing the data there to be a mixture of null/non-null values depending on the input and using the aggregation count instead of len.