df.apply ignores reduce=True, if function returns list · Issue #18901 · pandas-dev/pandas (original) (raw)

def get_candidates_3(closing, sales, settings: dict=_default_) -> list:
        
    S = sales.loc[:closing['future_cap'], :]  # sales were sorted by date beforehand
    mask = (S['building_id'] == closing['building_id'])|(S['unit_bbl'] == closing['bbl'])
    ... 
    if mask.any():
        return S.loc[mask, 'id'].tolist()
    return []

sample.apply(get_candidates_3, reduce=True, axis=1,
             sales=sales_ts,settings=_default_)

( python 3.6, pandas 0.21.0 )

The behavior I want to achieve is to get a single column of lists.
However, despite reduce=True this apply tries to map returned list as a series (?), raising the following:

ValueError: Shape of passed values is (10, 8), indices imply (10, 23)
(here, 10 is the size of the sample, 23 is the number of columns in the sample, 8 is the number of elements in the list that was returned for the first column?

When I change get_candidates_3 to return set instead, I get a full new dataframe of the same shape (10,23), filled with exactly the same sets row-wise, instead.

I am pretty sure I was using the same pattern successfully with a previous version...