Failing interpolation test

[cpcloud]

$ nosetests pandas/tests/test_generic.py:TestSeries.test_interp_quad
F
======================================================================
FAIL: test_interp_quad (pandas.tests.test_generic.TestSeries)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/phillip/Documents/code/py/pandas/pandas/tests/test_generic.py", line 339, in test_interp_quad
    assert_series_equal(result, expected)
  File "/home/phillip/Documents/code/py/pandas/pandas/util/testing.py", line 452, in assert_series_equal
    assert_attr_equal('dtype', left, right)
  File "/home/phillip/Documents/code/py/pandas/pandas/util/testing.py", line 369, in assert_attr_equal
    assert_equal(left_attr,right_attr,"attr is not equal [{0}]" .format(attr))
  File "/home/phillip/Documents/code/py/pandas/pandas/util/testing.py", line 354, in assert_equal
    assert a == b, "%s: %r != %r" % (msg.format(a,b), a, b)
AssertionError: attr is not equal [dtype]: dtype('int64') != dtype('float64')

----------------------------------------------------------------------
Ran 1 test in 0.041s

FAILED (failures=1)

[cpcloud]

cc @TomAugspurger

[TomAugspurger]

Does this make any sense? A float block with array of items with int dtype?

ipdb> self
SingleBlockManager
Items: Int64Index([1, 2, 3, 4], dtype=int64)
FloatBlock: 4 dtype: float64

I'm in core.internals.apply

[jreback]

items are the 'index'...so that is right

[jreback]

@cpcloud where do you see this failing? I can't repro on 64 or 32-bit

[jreback]

spoke too soon!

[jreback]

@TomAugspurger that test just needs to have the expected be int64 otherwise looks fine. as an FYI, maybe need
some tests that don't infer dtypes (e.g. set downcast=False to have no inferring of the results)

[TomAugspurger]

So are you saying to change expected to expected = Series([1, 4, 9, 16], index=[1, 2, 3, 4]) (int type), because that fails for me. I'm trying to figure out why the result [1., 4., 9., 16.] doesn't get downcast for me right now.

[jreback]

@TomAugspurger you may also want to put some more creative logic in there for inference. Since we know that we are going to only float/int coming in, you could always infer the ints so that you will get ints if possible and the floats will stay floats.

[jreback]

@TomAugspurger the result IS downcast (to int64), its the expected that is float64

[TomAugspurger]

@TomAugspurger the result IS downcast (to int64), its the expected that is float64

Not for me:

In [5]: result = sq.interpolate(method='quadratic')
In [6]: result
Out[6]: 
1     1
2     4
3     9
4    16
dtype: float64

---

Can you clear this up for me? I think this is where things aren't going the same way. b is the float block with the nan interpolated.

ipdb> !b
FloatBlock: 4 dtype: float64
ipdb> !b.values
array([  1.,   4.,   9.,  16.])
ipdb> !b.downcast(downcast)[0].values  # should be ints?
array([  1.,   4.,   9.,  16.])
ipdb> downcast
'infer'

That's in pandas/core/internals.py(337)_maybe_downcast()

I'll dig a bit deeper.

[TomAugspurger]

umm... in /pandas/core/common.py(1064)_possibly_downcast_to_dtype():

ipdb> result
array([  1.,   4.,   9.,  16.])
ipdb> result.astype(dtype)
array([ 1,  4,  8, 16])
ipdb> dtype
dtype('int64')
ipdb> 

but back at in the interpreter:

In [10]: a.astype(np.int64)
Out[10]: array([ 1,  4,  9, 16])

In [11]: a = np.array([1., 4., 9., 16.])

In [12]: a.astype(np.int64)
Out[12]: array([ 1,  4,  9, 16])

[jreback]

This is a precision issue

array([  1.,   4.,   9.,  16.])
(Pdb) p result[0]
1.0
(Pdb) p result[1]
4.0
(Pdb) p result[2]
9.0000000000000036
(Pdb) p result[3]
16.0

thus this array is NOT equal to array([1,4,9,16])

thus should not be downcasted (though you can make a case that it close 'enough') to be....

(Pdb) result == new_result
array([ True,  True, False,  True], dtype=bool)
(Pdb) result.round(8) == new_result
array([ True,  True,  True,  True], dtype=bool)

should we round when trying to downcast to int?

[jreback]

I think I should just do allclose with the default tolerances (1e-5,1e-8).....

[TomAugspurger]

Fair enough. And users can override that with s.interpolate(…, infer=False) right? Where would the necessary changes need to be made?

[TomAugspurger]

Or were you saying allclose for the test?