BUG: read_table with full line comment and delim_whitespace=True

[jmkuhn]

I expect the following to produce a DataFrame with 3 rows, 5 columns and no NaNs. Instead it produces 5 rows with the comment lines filled with NaNs. Pandas version 0.14.1.

In [2]: !cat test.txt
# comment
 0  1  2  3  4
 5  6  7  8  9
# comment
10 11 12 13 15

In [3]: df = pd.read_table("test.txt", skipinitialspace=True,
   ...: names=["A", "B", "C", "D", "E"], delim_whitespace=True, comment="#")

In [4]: df
Out[4]: 
    A   B   C   D   E
0 NaN NaN NaN NaN NaN
1   0   1   2   3   4
2   5   6   7   8   9
3 NaN NaN NaN NaN NaN
4  10  11  12  13  15

[jreback]

iirc this is only implemted in master (not 0.14.1) as it is an API change for a couple of reasons

pls check

[jmkuhn]

I just installed from master (0.14.1.dev) and the behavior of my example has not changed.

[jreback]

not merged yet
see #7470

[jorisvandenbossche]

@jreback It's a bit confusing, as we split the comment part of that PR #7470 into a new PR, which is included in 0.14.1: #7582

There seems to be a bug in comment in combination with delim_whitespace, if I use sep it does work:

In [13]: pd.read_csv(StringIO(s), skipinitialspace=True,
    ...:               names=["A", "B", "C", "D", "E"], delim_whitespace=True, comment="#")
Out[13]: 
    A   B   C   D   E
0 NaN NaN NaN NaN NaN
1   0   1   2   3   4
2   5   6   7   8   9
3 NaN NaN NaN NaN NaN
4  10  11  12  13  15

In [15]: pd.read_csv(StringIO(s), sep=' ', skipinitialspace=True,
    ...:               names=["A", "B", "C", "D", "E"], comment="#")
Out[15]: 
    A   B   C   D   E
0   0   1   2   3   4
1   5   6   7   8   9
2  10  11  12  13  15

[jreback]

the associated pr is not merged yet
I closed it because their is already an issue
u can leave this but make sure the pr then closes it

[jorisvandenbossche]

Which PR is not merged? #7582 is merged, and following the whatsnew entry in that PR, the above example should work

[jreback]

#7470 it was split out

[jreback]

cc @AmrAS1

[mdmueller]

Oh, I see. This isn't an issue with the PR splitup, it actually is a bug--the C tokenizer uses a different function for whitespace-delimited reading, and evidently I forgot to change line comment behavior there. This should be easily fixable, I'll start a new PR to address this.