Pivot to SparseDataFrame: TypeError: ufunc 'isnan' not supported in sparse matrix conversion · Issue #11633 · pandas-dev/pandas (original) (raw)

I want to convert a DataFrame to SparseDataFrame before pivoting it (when it gets really sparse, see also this discussion ). I have a textual key, which I need to keep ("chr"):

df = pd.DataFrame( list(zip([3,2,4,1,5,3,2],
             ["chr1", "chr1", "chr1",  "chr1", "chr2", "chr2", "chr3"], 
            [100,100, 100, 200, 1,3,1],
            [True, True, True, False, True, False, True],
            [-1,0,1,3, 0,2,1])) ,
            columns = ["counts", "chr", "pos", "strand", "distance"])

df.iloc[:,1:].dtypes
Out[]: 
chr         object
pos          int64
strand        bool
distance     int64
dtype: object

pd.pivot_table(df, index= [ "chr", "pos"], columns= ["strand","distance"], values= "counts").fillna(0)

     strand   False    True       
distance     2  3    -1  0  1
chr  pos                     
chr1 100     0  0     3  2  4
     200     0  1     0  0  0
chr2 1       0  0     0  5  0
     3       3  0     0  0  0
chr3 1       0  0     0  0  2

But I need to do it on much larger matrices. So I tried to do following trick:

dfpiv = pd.pivot_table(pd.SparseDataFrame(df), index= [ "chr", "pos"], columns= ["strand","distance"], values= "counts")

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Are there any plans to include a functionality option into pivot function for automatic conversion into SparseDataFrame?