ENH: Dedicated method for creating conditional columns · Issue #39154 · pandas-dev/pandas (original) (raw)

Creating conditional columns is an operation you perform a lot when wrangling data. Still right now there is no dedicated and explicit way of creating conditional columns in pandas. This can be seen on StackOverflow which is my top hit when I google "pandas conditional column" (this will obviously vary per person).

On this StackOverflow link we see the result of not being explicit since there are at least 6 different methods mentioned and a lot of anti patterns, also methods we do not want users to use because of inefficiency (apply with if else, map with lambda x for example).

This ticket is made as a starting point for discussion if we should have a more explicit way of creating conditional columns in pandas. Also because we need to look ahead when switching to EA's because the most used methods (and top accepted answer on SO) are numpy methods and return numpy arrays.

What do we have now?:

For single conditional columns:

• np.where
• .loc with assignment
• Series.where
• Series.mask, see comment

For multi conditional columns:

• np.select
• complex list comprehensions
• .apply with if else (not an option imo)

I don't have a suggestion yet, but I know I really like the clean code you can create with np.select and also with SQL CASE WHEN.

Other languages:

SQL has CASE WHEN:

CASE WHEN condition1 THEN result1 WHEN condition2 THEN result2 WHEN conditionN THEN resultN ELSE result END;

R (dplyr) has case_when:

df %>% mutate(g = case_when(a == 2 | a == 5 | a == 7 | (a == 1 & b == 4) ~ 2, a == 0 | a == 1 | a == 4 | a == 3 | c == 4 ~ 3, TRUE ~ NA_real_))

Related tickets / questions:

• Pandas assign malfunctions with if/else conditional check #30357 (using apply with if else)
• ENH: Allow join based on conditional statement #34543 (conditional join)