Cleanup filter_rows.py after upgrading to pandas v1

[EinarJohnsen]

Cleanup code in filter_rows after upgrade to pandas 0.25 as it should support quoting column names with backticks to refer to names with spaces.

[milesgranger]

Depends on #386

[EinarJohnsen]

Waiting for replies on issue on Pandas, this will be on standby as we can not use DataFrame.query DataFrame.query without support for multiple spaces.

[epa095]

Link to the pandas issue?

[EinarJohnsen]

pandas-dev/pandas#28893
There is a PR on it, fixing other stuff and also our issue.
pandas-dev/pandas#28215

[hwalinga]

As the implementer of that PR I would like to remind you that you should not rely on differentiating column names by spaces. In fact, this will raise an ambiguity warning in the new PR, as the information of multiple spaces will get lost when the query is parsed as Python.

I.e.

df = pd.DataFrame({"A A": [1, 2], "A  A": [3, 4]})
df.query("`A A` > 1")

This will raise an error. As on the parser end, the only information that is received is ('`', 'A', 'A', '`', '>', '1'), even with two spaces between the two A's, thus one could be referring. to

"A A"

or

"A  A"

This will therefore raise an error.

Implementing such a feature would require a different approach, and in general I would highly advise against using multiple spaces to differentiate your column names.

For now, what will improve with the new PR, is that, if you only would have a "A A" in your column names, you can still use it in a query. The error will only be raised if there is an unresolvable ambiguity in your column names.

[hwalinga]

update: Inspired by your hope of multiple spaces being supported in the .query function, I implemented a different approach that catches does as well. So the PR will support that as well.

(You still cannot use the hashtag and unpaired quotes.)

[EinarJohnsen]

Thank you @hwalinga
I noticed you made a commit for that. The point of it was more to allow having multiple spaces, as it is not always the case where the columns have only one space. The example given was more of a "special" case, since we do not often use spaces to differentiate column names, but we need to support names possibly containing multiple spaces.
And as the support for names with was added, it seemed like a somewhat unnecessary limitation to only have support for one space 😄
Thank you for taking your time to inform us.

[EinarJohnsen]

In the next pandas release (0.26), this should be possible. I have built pandas from source with version 0.26.0.xx, and it works as we expect it to.There is a couple of tests that has to be changed slightly, but otherwise it looks good.