Re: Zero-inflated and/or upper-censored Poisson / negative binomial distributions

FYI

statsmodels has zero-inflated countmodels for regression and similar
variation (truncated, hurdle) models,
and some scipy.stats subclasses for the related distributions (to support
`get_distribution` for the estimated models).

censored models are currently only for 0-1 censoring used internally as
component for hurdle models.
truncated count models were also mainly written for use in hurdle models
but are more "finished" to work standalone.
(only lower or left truncation)
Current limitation in statsmodels, truncation is fixed across observations,
i.e. truncation point cannot vary by observation.

The negative binomial models follow a mean-dispersion parameterization
usually used in regression models (and GLM) and not the standard
parameterization as in scipy distributions.

https://www.statsmodels.org/dev/discretemod.html#module-statsmodels.discrete.discrete_model
https://www.statsmodels.org/dev/distributions.html#count-distributions

Josef

On Fri, Sep 8, 2023 at 5:43 PM Robert Kern <[email protected]> wrote:

> On Fri, Sep 8, 2023 at 3:48 PM <[email protected]> wrote:
>
>> Hi,
>>
>> I'm a newbie regarding scipy development; I've been using scipy for many
>> years, and I've implemented a zero-inflated and upper-censored variant of
>> the Poisson and negative-binomial distributions, which we are using quite
>> extensively. Those implementations inherit from rv_discrete.
>>
>> In words, that distribution describes a zero-inflated and censored
>> counting distribution of a random non-negative integer variable k,
>> characterized by (uncensored and non-inflated) mean rate mu, (in case of
>> negative binomial, by a dispersion parameter), an upper censoring threshold
>> and zero-inflation probability. Note that "truncated" (cut beyond threshold
>> & renormalized) is different from "censored" (map all values above
>> threshold to threshold). For censoring, all probability mass for events
>> with k > upper_censoring is concentrated at k=upper_censoring. We use this
>> quite a lot for modelling time series in retail in which finite stocks
>> (censoring) or occasional out-of-stock-situations (zero-inflation) occur.
>>
>> Since the detailed roadmap of scipy mentions "Implement additional widely
>> used continuous and discrete probability distributions, e.g. mixture
>> distributions.", I thought it might be interesting to add this to the
>> discrete distributions?
>>
>
> Yes, I would consider these worthwhile. I would suggest taking a look at
> all of the discrete distributions and evaluating whether and how this kind
> of modification might be worth applying to them too. Even if your only
> interest is in Poisson and negative-binomial now, we'd at least want to
> ensure that the parameters and their semantics can be applied the same way
> later to any applicable discrete distribution.
>
> --
> Robert Kern
> _______________________________________________
> SciPy-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/scipy-dev.python.org/
> Member address: [email protected]
>

_______________________________________________
SciPy-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/scipy-dev.python.org/
Member address: [email protected]