API: Gather and other inspiration from tidyr

[datnamer]

http://connor-johnson.com/2014/08/28/tidyr-and-pandas-gather-and-melt/

In the spirit of the excellent assign method, wondering if there is support for some tidyr style transformations?

[shoyer]

Hadley Wickham is brilliant at API design, so I'm always happy to use his work for inspiration. Concrete suggestions would be helpful.

At the very least, gather looks like a broadly useful method to add to pandas dataframes. It seems like an improved version of melt, which we already borrowed from Hadley. So +1 from me, though I'd wait for a few other core devs to chime in (@TomAugspurger?) before you start writing your PR.

[jreback]

This already exists (sort of). as wide_to_long. +1 on changing the name to gather, better API (this uses regexes to 'infer' the variable names).

In [26]: df
Out[26]: 
    a   b    names
0  67  56   Wilbur
1  80  90  Petunia
2  64  50  Gregory

In [27]: df2 = df.rename(columns={'a' : 'A1', 'b' : 'B2'})

In [28]: df2
Out[28]: 
   A1  B2    names
0  67  56   Wilbur
1  80  90  Petunia
2  64  50  Gregory

In [29]: pd.wide_to_long(df2,['A|B'],'names','heartrate')
Out[29]: 
                   A|B
names   heartrate     
Wilbur  1           67
Petunia 1           80
Gregory 1           64
Wilbur  2           56
Petunia 2           90
Gregory 2           50

[jorisvandenbossche]

I just wanted to give the same reference to wide_to_long
But I think that is exactly an example of how to not do it (in the sense of "I have a specific case, lets add a function for that"). To be clear, at least not for the scope of pandas (or at least top-level namespace of pandas).

In that sense, I am -1 on just adding 'yet another reshape-like' function, before thinking it a bit more through (but I fully agree that the current reshape functionality (melt and wide_to_long) can use some love to make it more flexible and user friendly):

• Is it possible to improve melt without breaking back compat (I don't know if I find the name 'gather' that much better)
• How do we keep the interface somewhat consistent between the 'opposite' functions (pivot, pivot_table, spread in tidyr)
• Maybe someone should rather write a 'pandas-tidy' package that implements these functions based on pandas first?
• Go through the different problem cases of reshaping, see how it can currently done in pandas, what is good, what is lacking, and how can we improve this?

The problem is a bit that pandas is becoming a monolithic package. Hadley Wickham has indeed very nice API's (and we can learn a lot from that to strive for in pandas). But, if he has a new idea, he just starts a new package. For example, the current melt function in pandas is based on his older reshape(2) package. Now you have tidyr, and we could get inspiration from there. But then maybe next, there is another package, en want can't keep adding functions .. (or, we can, but the question is if we want this).
But of course, this is a much more general discussion ..

[datnamer]

-1 on my own suggestion and further overloading the pandas namespace and +1 on using simple, composable building block abstractions and maybe starting a new package. Pydata is great for consistency, but R has more rapid diffuse iterative innovation...I wonder if we can help foster the latter.

[kay1793]

That's well put, but I'm forced to strongly disagree. While as you say R unarguably benefits from a rapid diffuse iterative innovation, If you really examine the issue closely you must realize that pydata tools tend to intentionally embrace a more focused decentralized convergent behaviour-driven amalgamation approach, one that is inherently aspect-oriented and inline with the best-of-breed theories of cloud-first which rain supreme over this exciting new age of "stuff".

[datnamer]

Sorry, I don't understand what that means.

[shoyer]

Indeed, a separate package is probably a better place to start. The only unfortunate bit about doing this outside of pandas is that users can't do method chaining with third party packages.

@kay1793 Please don't troll.