BUG: groupby apply KeyError, hash instability?

[HHest]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import pandas as pd
def aggfunc(df):
    return pd.Series([df['b'].mean()], index=['b_mean'])
df=pd.DataFrame({
    'a': ['aa', 'a2', 'a3'],
    'b': [1, 2, 3],
}, index=pd.Index([1., 2., 3.]))
df.groupby('a').apply(aggfunc)

Issue Description

``I get a KeyError with the provided code. The call stack is:

Traceback (most recent call last):
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\indexes\base.py", line 3361, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas\_libs\index.pyx", line 76, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 1533, in pandas._libs.hashtable.Float64HashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 1542, in pandas._libs.hashtable.Float64HashTable.get_item
KeyError: 0.0
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "<input>", line 8, in <module>
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\groupby\groupby.py", line 1275, in apply
    result = self._python_apply_general(f, self._selected_obj)
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\groupby\groupby.py", line 1309, in _python_apply_general
    keys, values, mutated = self.grouper.apply(f, data, self.axis)
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\groupby\ops.py", line 804, in apply
    result_values, mutated = splitter.fast_apply(f, sdata, group_keys)
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\groupby\ops.py", line 1349, in fast_apply
    return libreduction.apply_frame_axis0(sdata, f, names, starts, ends)
  File "pandas\_libs\reduction.pyx", line 369, in pandas._libs.reduction.apply_frame_axis0
  File "pandas\_libs\reduction.pyx", line 428, in pandas._libs.reduction.BlockSlider.__init__
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\frame.py", line 3430, in __getitem__
    indexer = convert_to_index_sliceable(self, key)
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\indexing.py", line 2329, in convert_to_index_sliceable
    return idx._convert_slice_indexer(key, kind="getitem")
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\indexes\numeric.py", line 242, in _convert_slice_indexer
    return self.slice_indexer(key.start, key.stop, key.step, kind=kind)
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\indexes\base.py", line 5685, in slice_indexer
    start_slice, end_slice = self.slice_locs(start, end, step=step)
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\indexes\base.py", line 5893, in slice_locs
    end_slice = self.get_slice_bound(end, "right")
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\indexes\base.py", line 5807, in get_slice_bound
    raise err
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\indexes\base.py", line 5801, in get_slice_bound
    slc = self.get_loc(label)
  File "C:\Users\nbttn1\python_venv\pandas_bug\lib\site-packages\pandas\core\indexes\base.py", line 3363, in get_loc
    raise KeyError(key) from err
KeyError: 0

Note that the KeyError doesn't appear when I send in these other very similar dataframes:

# okay when the 1st element of 'a' is written differently
df=pd.DataFrame({
    'a': ['a1', 'a2', 'a3'],
    'b': [1, 2, 3],
}, index=pd.Index([1., 2., 3.]))

# okay when index is not float
df=pd.DataFrame({
    'a': ['aa', 'a2', 'a3'],
    'b': [1, 2, 3],
}, index=pd.Index([1, 2, 3]))

# okay when there are only two rows
df=pd.DataFrame({
    'a': ['aa', 'a2'],
    'b': [1, 2],
}, index=pd.Index([1., 2.]))

Expected Behavior

There should not be any KeyError, and the output should look something like the following:

    b_mean
a         
a2     2.0
a3     3.0
aa     1.0

Installed Versions

INSTALLED VERSIONS

commit : 945c9ed
python : 3.7.9.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None
pandas : 1.3.4
numpy : 1.21.3
pytz : 2021.3
dateutil : 2.8.2
pip : 21.1.2
setuptools : 57.0.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[HHest]

This issue appeared in 1.0.0; the sample code works with the version prior (0.25.3).

[jreback]

pls try on master

[phofl]

Works on master

[HHest]

Thanks for checking @phofl. I confirm that the example code also works on my local build of master. Sorry for not checking before submitting.

1.4.0.dev0+1043.g66865fcd53
    b_mean
a         
a2     2.0
a3     3.0
aa     1.0

Given that the error is sensitive to the content of column 'a', for some reason unknown to me, I can't conclude that the example code captures the underlying issue, which master has resolved. But if you think the example is worth turning into a test, I'll volunteer for that. Just let me know where I should put the test.

[jG-codeR13]

I have got a way to do this
See,whenever you use apply function with groupby, you can't access the group key inside the function.

Instead you can loop through the grouped dataframe group by group, and append the processed dataframes together.

Create a new dataframe and append the processed groups to it. This is working

for key,t in df.groupby('a'):
temp = t
for i in range(temp['b'].shape[0]):
temp['b_mean']=temp['b'].mean()
df = pd.concat([df,temp])
df = df.iloc[3: ,:]

df=df.reset_index(drop=True)
df.index+=1
df =df.drop(['b'], axis = 1)
print(df)

Sorry ,just a small amount of tricky code added for the desired output but I hope you have understood the core concept

[brunoacpcosta]

Hey, I would like to contribute! Can I work on the tests for this issue?

[mdfarazzakir]

Hey guys,
I wanted to try from my side on can I do it and make a sort of generic code.

df=pd.DataFrame({
'a': ['aa', 'a2', 'a3'],
'b': [1, 2, 3],
}, index=pd.Index([1., 2., 3.]))

def groupByEval(df,groupByColumn,targetColumn):
dataList = []
indexList = [groupByColumn]
for i in set(df[groupByColumn]):
dataList.append(df[df[groupByColumn] == i][targetColumn].mean())
indexList.append(i)
df = pd.DataFrame({'{}_mean'.format(targetColumn):dataList},index=indexList[1:])
df.index.name = indexList[0]
return df

print(groupByEval(df,'a','b'))