Review of new/fixed quantile and percentile methods in NumPy

Sebastian Berg <[email protected]> Tue, 21 Sep 2021 12:59:07 -0500
Newsgroups	gmane.comp.python.scientific.user
Message-ID	<[email protected]>
Hi all,

I thought the SciPy community is more likely to have an opinion on
quantile calculation/estimation methods than the NumPy one.

We would like to refactor the quantile function in NumPy:

     https://github.com/numpy/numpy/pull/19857/files

And comments about the API choice (or math) would be very welcome!

The PR basically introduces the "9 types" that R has (Which are from
the Hyndman and Fan paper [1].  The R code is also by Hyndman.  The
paper was a largely a review of methods implemented at the time.).

I pinged Hyndman, because I was wondering if there is a more minimal
set or better naming for the methods.  This did not reduce the number
but gave us the current methods and names (all of them, plus the ones
NumPy currently has), the Numbers map to the R/Hyndman "type":

1. inverted cdf
2. averaged inverted cdf
3. closest observation
4. interpolated inverted cdf
5. hazen  (these names come from wolfram)
6. weibull
7. inclusive (default! Name from `statistics.quantiles`: rename?!)
8. median unbiased
9. normal unbiased

And the ones NumPy (unfortunately) currently has:

* lower
* higher
* nearest
* midpoint

Note that Python's `statistics.quantiles` also has `exclusive` which is
none of those methods :(.


Any review of the math and the API would be great!  I am still
considering asking for smaller changes [2].  But in general, I think
this is a necessary step.
Help with documenting this better could also be great!

Cheers,

Sebastian


[1] Hyndman, R. J., & Fan, Y. (1996).
     Sample quantiles in statistical packages.
     The American Statistician, 50(4), 361-365

[2] Some thoughts of things that maybe we should consider:

* Rename `interpolation` to `method`
* Rename any of the methods?  Especially the default?
* Rename the current "lower", "nearest", etc. to
  "legacy-lower" or so to discourage their use.
* Expose or do not expose the `QuantileInterpolation` enum?
* If I knew a way to pick 4 of those 9 methods only, I would
  consider doing that :).
  Right now I can't, and as much as I dislike having 13 methods,
  it seems better than what we have now.  And making the parameters
  public does not feel helpful either...

_______________________________________________
SciPy-User mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/scipy-user.python.org/
Member address: [email protected]
signature.asc (application/pgp-signature, 833 B)