Re: PERT fit with curve_fit function

On Tue, Dec 20, 2022 at 3:19 AM Julian CHAMBRIER <[email protected]>
wrote:

> Hello Robert,
>
> Thank you for your answer,
>
> As documented, if you don't provide an initial guess `p0`, it will use
>> `np.ones()` to make the initial parameter vector. This happens to be a
>> degenerate input for you since `low==peak==high`, so it will likely not see
>> an improvement by changing the parameters in its initial steps, so it will
>> think that it has already converged or won't converge. You can try
>> providing a more appropriate `p0`, say `[x.min(), (6 * x.mean() - x.min() -
>> x.max()) / 4, x.max(), 4.0]`.
>>
>
> > Yes indeed it's a bit better but the results differ a lot, which I
> suspected.
>
>
> 2. If you must fit it, fit the Beta distribution using the existing
>> algorithms and convert the fitted parameters to the PERT parameterization.
>>
>
>
> >  Following your advice, I use the beta distribution where I modify the
> parameters to match a Pert distribution.
>
> Nevertheless, I have a particular need. Indeed, in many cases the beta
> inference is enough, but on samples that do not necessarily follow a Pert
> distribution, the beta works very badly (which is normal).
>
> I would have liked a fast alternative that would infer (roughly) a Pert on
> any sample, in case the Beta inference does not give me good results. You
> would have understood that my need is to represent any data sample with a
> Pert.
>

Remember, a Beta is a PERT and a PERT is a Beta. They are the same
distribution. The reason that _all_ of these algorithms are not working
well is because this distribution is not an appropriate model for your data.


> Granted, sometimes other distributions will be much more suitable, but
> that will be in a second time.
>

You are making all of these problems for yourself by ignoring the Folk
Theorem. Your model is the root of the problem, not the algorithms.

That's why I'm looking at other alternatives to the Beta.
>
> To summarize:
> 1) I infer with a Beta in case the inference goes well and the sample is
> representative of that distribution. I then change the parameters to a PERT
> distribution.
> 2) If the Beta inference is not conclusive, I must have an alternative
> that finds me the Pert distribution that best represents my data (even if
> the data do not follow this distribution). I would have liked an
> alternative that is rather fast in time, but that is more efficient than
> just retrieving a min, a max, ...
>

The problem is that the parameters found by the Beta fit _are_ the PERT
distribution that best represents your data (for a definition of "best
represents" being a typical "maximum likelihood" meaning). The
best-representing parameters just look weird to you, but they _are_ better
for your data than other Beta/PERT parameters. The reason that the
best-representing parameters look weird is because your data is not
well-represented by _any_ Beta/PERT parameter sets.

If you want other parameters, you will have to specify, mathematically,
what "best parameters" means for your purposes. It will deviate from "best
representing the data" in some way. It is a form of changing the
model, though perhaps not enough. For example, when minimizing the negative
log-likelihood, you could add regularization terms that penalize lambda
values going too high, for example (doing this well is an art; I have no
specific advice for doing this since this is really about satisfying your
internal preferences). This does mean that whatever you are attempting to
use the parameters _for_ downstream of this will be informed in part, if
not mostly, by the regularization rather than the data itself. Of course,
since the data is not well-represented by a Beta/PERT distribution, those
didn't have much meaning to begin with.

This is a lot of work! If you use an appropriate model for your
data, even just a somewhat more flexible distribution, you can avoid all of
it.

-- 
Robert Kern

_______________________________________________
SciPy-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/scipy-dev.python.org/
Member address: [email protected]