Groupby behaves differently when using levels and list of column keys · Issue #9344 · pandas-dev/pandas (original) (raw)

When grouping by several levels of a MultiIndex, groupby evaltuates all possible combinations of the groupby keys. When grouping by column name, it only evaluates what exist in the DataFrame. Also, this behavior does not exist in 0.14.1, but does in all final releases from 0.15.0 on.

This may be a new feature, not a bug, but I couldn't find anything in the docs, open or closed issues, etc. (closest was Issue #8138). If this is the intended behavior, it would be nice to have in the docs.

import pandas as pd import numpy as np

df = pd.DataFrame(np.arange(12).reshape(-1, 3)) df.index = pd.MultiIndex.from_tuples([(1, 1), (1, 2), (3, 4), (5, 6)]) idx_names = ['x', 'y'] df.index.names = idx_names

Adds nan's for (x, y) combinations that aren't in the data

by_levels = df.groupby(level=idx_names).mean()

This does not add missing combinations of the groupby keys

by_columns = df.reset_index().groupby(idx_names).mean()

print by_levels print by_columns

This passes in 0.14.1, but not >=0.15.0 final

assert by_levels.equals(by_columns)

INSTALLED VERSIONS

commit: None
python: 2.7.7.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.15.2
nose: 1.3.4
Cython: 0.20.1
numpy: 1.9.1
scipy: 0.15.1
statsmodels: 0.7.0.dev-161a0f8
IPython: 2.3.0
sphinx: 1.2.2
patsy: 0.3.0
dateutil: 1.5
pytz: 2014.9
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.4.2
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.5.5
lxml: 3.3.5
bs4: 4.3.1
html5lib: None
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.4
pymysql: None
psycopg2: None