Re: fmin_powell returns incorrect parameters for simple least-squares problem

[image: trajectory.png]
This is the trajectory of each one plotted with the loss. It seems like
Powell has a hard time navigating narrow valleys. BFGS, on the other hand,
is locally fitting a parabola, which is a good fit for the loss function
here.

This is what I get if I instead feed noiseless data, further illustrating
the fact that Powell doesn't like ravines.
[image: traj_narrow.png]
The code is here:
https://gist.github.com/Dapid/1da960739b9e006f41a962607f6b1c54

I can't say why Scipy's fails and Octave doesn't, but hopefully this gives
someone else an idea.

/David.

(PS, the images are 100 kB, I hope it is fine to send them on the list).

On Wed, 1 Mar 2023 at 12:22, Matthew Brett <[email protected]> wrote:

> Hi,
>
> Could I ask for your collective advice?
>
> While writing examples for students I found an instance of
> `fmin_powell` claiming success while returning an incorrect minimum,
> on a very simple least-squares problem.
>
> I've put up the reproducer in this repository:
>
> https://github.com/matthew-brett/powell-fails
>
> The take-home is that, for a simple least-squares problem, and
> ordinary-looking data, for a particular starting value, `fmin_powell`
> stops on a not-minimum value and claims success, where other
> optimizers do find the minimum, as does the Octave implementation.
>
> Here is the output from the reproducer:
>
> LS inter: 2.114798570871842
> LS slope: 0.5088641205763367
> LS SSE error: 1.0674470960898728
>
> BGFS minimization:
> Optimization terminated successfully.
>          Current function value: 1.067447
>          Iterations: 3
>          Function evaluations: 15
>          Gradient evaluations: 5
> [2.11480099 0.50886336]
>
> Powell minimization:
> Optimization terminated successfully.
>          Current function value: 1.074363
>          Iterations: 6
>          Function evaluations: 152
> [2.22069108 0.47457665]
>
> As you can see, the Powell result has a higher sum of squared error,
> different parameters, and claims success.  I get this for Scipy 1.10.1
> on Mac M2, Intel, and Ubuntu 22.04 amd64.
>
> I notice that we have a "modified" Powell implementation.  Can I ask
> for hints as to where to go next in exploring this problem?  Could
> there be a flaw in our implementation?
>
> Cheers,
>
> Matthew
> _______________________________________________
> SciPy-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/scipy-dev.python.org/
> Member address: [email protected]
>

_______________________________________________
SciPy-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/scipy-dev.python.org/
Member address: [email protected]