BUG: sqrt not implemented in df.eval

[acorbe]

In the following scenario:

import pandas as pd
import numpy as np

a = np.random.rand(1000)
df = pd.DataFrame({'a' : a })

This call doesn't work (while it is supposed to, I guess)

df.eval('sqrt(a)')

     NotImplementedError: 'Call' nodes are not implemented

while this one does

df.eval('(a)**(.5)')

I guess this is a bug.

[jreback]

Well, its NotImplemented, the AST parsing does not interpret the Call node ATM. I just don't think @cpcloud had time to implement it (their may have been another reason but I don't recall). You are welcome to take a stab.

This is applies to function calls in general in eval/query. Note that this IS implemented for the pytables parsing.

[acorbe]

Hi,
I can see whether I can implement it. I had no idea where I should start looking at though at though.

Do you have suggestions?

[jreback]

https://github.com/pydata/pandas/blob/master/pandas/computation/expr.py

hmm, this IS implemented. So might be buggy somewhere. in the debugger step thru this and see if you can see where/why this is rejected.

[cpcloud]

the parser disallows function calls by original design, mostly for time reasons but this could be opened up. there's a frozenset object somewhere (I think in computation/expr.py) that contains the valid numexpr functions. I don't think arbitrary function calls are in scope of eval, but we could certainly open it up for the numexpr functions.

[cpcloud]

@jreback that implementation is only used by the pytables engine

[jreback]

@cpcloud right....this was a while ago. yes I remember now, the arbitrary functions have to be done in python space, so was a bit complicated (e.g. have to validate which are numexpr capable and which are not).

[jreback]

@acorbe this is a bit non-trivial. but would love some eyes on it! this is actually a pretty interesting thing to understand, you get to learn AST parsing, and numexpr internals ( maybe +1 or -1) by your definition !

[cpcloud]

@acorbe just a bit of what's going on to (maybe?) whet your appetite:

An expression like:

a ** .5

which as you show parses to

(a) ** (.5)

eventually goes into numexpr (effectively) as

numexpr.evaluate('(df.a.values) ** (0.5)')

Since ** is a binary operator, you get it for "free" (as long as you implement a generic enough representation of binary operators).

With function calls, the parsing is more complicated (though most of the code necessary to do it is already there thanks to @jreback), but Python has the ast module which is fairly easy to use to parse Python code.

numexpr has a small set of functions that it implements in its special (extensible) evaluator.

There's a lot going in eval/query, which I'm happy to discuss with you, would love to have some more eyes on this.