Restructure percentile methods

[ricardoV94]

As exemplified in the Wikipedia page: https://en.wikipedia.org/wiki/Percentile#The_nearest-rank_method

[eric-wieser]

I think this already exists? Using the wikipedia example:

>>> np.percentile(15, 20, 35, 40, 50], [5, 30, 40, 50, 100], interpolation='lower')
array([15, 20, 20, 35, 50])

[ricardoV94]

It does not. Look at example 2 in the wikipedia page:

>>> np.percentile([3, 6, 7, 8, 8, 10, 13, 15, 16, 20], [25,50,75,100], interpolation='lower')
array([ 7,  8, 13, 20])

When it should be [7,8,15,20]

It similarly fails in the third example

[seberg]

Nearest sounds a lot like "nearest"? Though there is always another point about how exactly the boundaries work.
EDIT: That is, where exactly is 0 and 100 considered to be, at the datapoint or before the datapoint? (that is IIRC, anyway there are a lot of annoying complexities here)

[seberg]

don't want to read it, I think the difference might be the C parameter further down, so if someone who knows this wants to add this....

[seberg]

Frankly, I think adding the C parameter would likely really be good. But mostly better documentation would be nice, and someone who really knows this stuff is needed....

[ricardoV94]

I don't know if this has anything to do with the C-parameter, although I agree that the option of choosing it could be desirable.

I have found another thread that incidentally brought up this issue (Dec. 2016). It seems that the algorithm I am looking for (and which wikipedia calls nearest-rank) is mentioned in this commonly cited paper by Hyndman-Fan (H&F) as being the oldest and most studied definition of percentile (it was the one I learned in Stats course). It is a discontinuous function, so I think the parameter C does not apply here (I may be wrong).

Here is how it would look like against the other options provided by numpy that intuitively seem to compute a similar thing (i.e., 'lower', 'nearest'):

[seberg]

To me it looks exactly like the C parameter on first sight, the nearest curve is more stretched then the H&F curve, which is expected since numpy uses 1 and apparently H&F uses 0.

[seberg]

If you want proof. Repeat the whole thing with the same values repeated 1000 times, my guess is they will converge.
EDIT: Or maybe not, don't have the patience or time to really think about it. But I still think it is the C parameter wikipedia mentions, so please proof me wrong :)

[eric-wieser]

A graph like that would be a great addition to the percentile docs

edit: preferably one showing the open/closedness of the discontinuities

Note to readers: To keep this thread manageable, I've marked all discussions below about adding this graph to the docs as "resolved". The graph is now at the bottom of https://numpy.org/devdocs/reference/generated/numpy.percentile.html.

[ricardoV94]

@eric-wieser I don't mind making that graph. I will come back with something later today, should I post it in here?

[ricardoV94]

@seberg I will be honest here, I don't know how the interpolation is being calculated based on the C-parameter. What makes me think that it is not related is that the C-parameter is only discussed in the linear interpolation section (Wikipedia), and both the Wikipedia and Hyndmand & Fan paper discuss the algorithm I requested in separate sections from the interpolation ones.

I don't know if there is any interpolation parameters that always give the same results as the algorithm I am interested in.

Even if there are, should this be the way used to get to it? Changing a 'strange' parameter to get the most common definition of percentile does not seem the best way to implement it imho.

[seberg]

@ricardoV94, maybe, but you can't just change the defaults, no matter how bad they are. We could expose something like method="H&K" to override both parameters at once.

The C parameters is where you define 0% and 100% to be with respect to the datapoints (on the data point or not, etc.). As parameter C on wikipedia, it may well be only for interpolation, but the same issue causes the difference here I am sure. C is dubious of course, a proper name might be something like range='min-max' or range='extrapolated' or probably something completely different. As I said, redo the plots with many many datapoints (possibly with tiny noise), and I think you will see them converge, since the range definition becomes less obvious.

[ricardoV94]

@seberg I am fine with method="H&K" or maybe method="classic". Interpolation="none" could also make sense.