BUG: Groupby-Apply with modifying dataframe and sorting overwrites group values

[Rik-de-Kort]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd


def wtf(x):
    x['baz'] = ''
    return x.sort_values(by='boo')


df = pd.DataFrame({'boo': ['A', 'B'], 'grouper': ['foo', 'bar']})
df.groupby('grouper').apply(wtf)

Problem description

The output I get from above code is

          boo grouper baz
grouper                  
bar     1   B     bar    
foo     0   B     bar    

whereas I would expect

          boo grouper baz
grouper                  
bar     1   B     bar    
foo     0   A     foo    

Both steps are necessary. If we leave out entirely or apply df['baz'] = '' before executing the groupby-apply, the bug doesn't happen. If we leave out the sort_values, the bug doesn't happen. (Applying it outside of the groupby-apply is different semantically).
Order matters. If we first sort then modify, the bug doesn't happen.

Any pointers as to where to start debugging this?

Output of pd.show_versions()

INSTALLED VERSIONS

commit : f2ca0a2
python : 3.8.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.17763
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 4, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 1.1.1
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.2
setuptools : 49.6.0.post20200814
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[rhshadrach]

Modifying a DataFrame in the midst of a computation can produce unpredictable results. You get the expected output if you change the function to:

def wtf(x):
    x = x.copy()
    x['baz'] = ''
    return x.sort_values(by='boo')

[Rik-de-Kort]

Thanks for the quick response. Would definitely be worth generating a user warning for this.

[arw2019]

Would definitely be worth generating a user warning for this.

Is it workable to do this? We'd want something like what python does when somebody modifies keys in a dict during iteration:

In [8]: d = {k: v for k,v in enumerate("pandas")} 
   ...: for k, v in d.items(): 
   ...:     d[2*k] = v 
   ...:                                                                         
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-8-22f2b5d656c9> in <module>
      1 d = {k: v for k,v in enumerate("pandas")}
----> 2 for k, v in d.items():
      3     d[2*k] = v
      4 

RuntimeError: dictionary changed size during iteration

I have no idea how easy this is to do

[mroeschke]

We have this already documented here: https://pandas.pydata.org/docs/user_guide/gotchas.html?highlight=mutate#mutating-with-user-defined-function-udf-methods

I suppose we could also document these directly in the apply methods

[rhshadrach]

Thanks @mroeschke - I did add a note for other UDF methods (e.g. https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html), but missed Groupby.apply. Agreed it should be added here as well.