get_dummies(df,sparse=True) does not return sparse DataFrame · Issue #10531 · pandas-dev/pandas (original) (raw)

Skip to content

Navigation Menu

• • GitHub Copilot Write better code with AI
  • GitHub Advanced Security Find and fix vulnerabilities
  • Actions Automate any workflow
  • Codespaces Instant dev environments
  • Issues Plan and track work
  • Code Review Manage code changes
  • Discussions Collaborate outside of code
  • Code Search Find more, search less
• Explore
  • Learning Pathways
  • Events & Webinars
  • Ebooks & Whitepapers
  • Customer Stories
  • Partners
  • Executive Insights
• • GitHub Sponsors Fund open source developers
  • The ReadME Project GitHub community articles
• • Enterprise platform AI-powered developer platform
• Pricing

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sign up

Appearance settings

Description

Just like it says in the subject. Here's an example:

In [216]: pd.version.version Out[216]: '0.16.2'

In [217]: df = pd.DataFrame(np.random.randint(10,size=(10000,5)),columns=list('abcde'))

In [218]: df.head() Out[218]: a b c d e 0 2 6 1 4 0 1 5 2 6 8 5 2 0 8 7 3 1 3 9 2 6 3 1 4 4 8 6 8 6

In [219]: ddf = pd.get_dummies(df,columns=df.columns,sparse=True)

In [220]: ddf.head() Out[220]: a_0 a_1 a_2 a_3 a_4 a_5 a_6 a_7 a_8 a_9 ... e_0 e_1 e_2 e_3
0 0 NaN 1 NaN 0 0 NaN NaN NaN 0 ... 1 0 NaN NaN
1 0 NaN 0 NaN 0 1 NaN NaN NaN 0 ... 0 0 NaN NaN
2 1 NaN 0 NaN 0 0 NaN NaN NaN 0 ... 0 1 NaN NaN
3 0 NaN 0 NaN 0 0 NaN NaN NaN 1 ... 0 1 NaN NaN
4 0 NaN 0 NaN 1 0 NaN NaN NaN 0 ... 0 0 NaN NaN

e_4 e_5 e_6 e_7 e_8 e_9
0 NaN 0 0 NaN NaN NaN
1 NaN 1 0 NaN NaN NaN
2 NaN 0 0 NaN NaN NaN
3 NaN 0 0 NaN NaN NaN
4 NaN 0 1 NaN NaN NaN

[5 rows x 50 columns]

In [221]: type(ddf) Out[221]: pandas.core.frame.DataFrame

In [222]: hasattr(ddf,'density') Out[222]: False

In [223]: ddf = ddf.to_sparse()

In [224]: type(ddf) Out[224]: pandas.sparse.frame.SparseDataFrame

In [225]: ddf.density Out[225]: 0.1

I notice the NaN encoding in the DataFrame returned by get_dummies when sparse=True but the datatype is not sparse. Is this expected behavior?