Re: Question: Can LowLevelCallable functions be beneficial to scipy.optimize performance?

--===============1671367590695989219==
Content-Type: multipart/alternative; boundary="000000000000cce69d0611b330c0"

--000000000000cce69d0611b330c0
Content-Type: text/plain; charset="UTF-8"

Dear Oyibo,
first of all, I should say that I'm not convinced that the example
contained in the matecdev site is a good example. Whilst I haven't checked
it seems that using `interpolate.LSQUnivariateSpline` may achieve what that
poster was trying to accomplish, and would be much faster than the python
code presented.

Most of the answer to your question is pretty much contained in
https://github.com/scipy/scipy/issues/11783.

In terms of speed the most important thing is to ensure that the objective
function calculates quickly. This typically involves moving calculations
from Python to C/Cython, etc. Only once this has been investigated is it
worth looking into optimiser overhead. This overhead is often (but not
always) much smaller than the time required for objective function
calculation. Such overhead includes objective function call time, or
overhead from the optimiser itself.

There is a wide heterogeneity of code design in scipy.optimise. Some work:

A) in pure python (e.g. BFGS, nelder-mead)
B) in C/Fortran where you supply a callback function for the native code to
call (e.g. tnc, leastsq)
C) in C/Fortran but where iteration is controlled by python and the native
code expects a objective/gradient function updates each time it's called
(L-BFGS-B).

Using LowlevelCallable would not furnish any gains for case A. Rewriting of
those codes to cython would be a lot of work for an uncertain amount of
gain.
Using LowlevelCallable for case B would likely furnish the most gain.
Using LowlevelCallable for case C would probably involve rewriting so that
all iteration occurred from Cython/C, but the level of gain is still
uncertain.

There remain further issues. Let's take `tnc` as an example (case B). For
that minimiser it's necessary to supply both objective AND gradient
evaluations. The user must supply the first, but if the user cannot supply
the gradient then `minimize` is asked to estimate it via numerical
differentiation. Estimating the gradient with numerical differentiation
occurs in Python. This kind of setup is taken care of by using the
`optimize._differentiable_functions.ScalarFunction` object.
Thus, a LowlevelCallable that only furnishes objective evaluations is not
much use, because the numerical differentiation (in Python) is still
required to evaluate gradients, which requires at least N objective calls
(from Python to whatever the objective is contained in).
In order for this to be fully streamlined the numerical differentiation
code would also have to be written in cython/c.
Note that the user can either supply the objective function, separate
objective and gradient functions, or a combined objective and
gradient function. All those different cases have to be catered for.
Moreover, for each of those cases one would either be supplying a
`LowlevelCallable` or Python function.
Last, but not least, most minimisers often ask numpy to perform operations,
so porting minimisers to cython will still have python calls.

Thus, it's a large task just to amend a *single* minimiser. There are lots
of minimisers and one would want to do this in a uniform manner across them
all.

Given the width of such a project, one would want to conclusively know that
it would be worth it, and that's definitely not clear.

Andrew.

--000000000000cce69d0611b330c0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Dear Oyibo,</div><div>first of all, I should say that=
 I&#39;m not convinced that the=C2=A0example contained in the matecdev site=
 is a good example. Whilst I haven&#39;t checked it seems that using `inter=
polate.LSQUnivariateSpline` may achieve what that poster was trying to acco=
mplish, and would be much faster than the python code presented.</div><div>=
<br></div><div>Most of the answer to your question is pretty much contained=
 in <a href=3D"https://github.com/scipy/scipy/issues/11783">https://github.=
com/scipy/scipy/issues/11783</a>.</div><div><br></div><div>In terms of spee=
d the most important thing is to ensure that the objective function calcula=
tes quickly. This typically involves moving calculations from Python to C/C=
ython, etc. Only once this has been investigated is it worth looking into o=
ptimiser overhead. This overhead is often (but not always) much smaller tha=
n the time required for objective function calculation. Such overhead inclu=
des objective function call time, or overhead from the optimiser itself.</d=
iv><div><br></div><div>There is a wide heterogeneity of code design in scip=
y.optimise. Some work:</div><div><br></div><div>A) in pure python (e.g. BFG=
S, nelder-mead)</div><div>B) in C/Fortran where you supply a callback funct=
ion for the native code to call (e.g. tnc, leastsq)</div><div>C) in C/Fortr=
an but where iteration is controlled by python and the native code expects =
a objective/gradient function updates each time it&#39;s called (L-BFGS-B).=
</div><div><br></div><div>Using LowlevelCallable would not furnish any gain=
s for case A. Rewriting of those codes to cython would be a lot of work for=
 an uncertain amount of gain.</div><div>Using LowlevelCallable for case B w=
ould likely furnish the most gain.</div><div>Using LowlevelCallable for cas=
e C would probably involve rewriting so that all iteration occurred from Cy=
thon/C, but the level of gain is still uncertain.</div><div><br></div><div>=
There remain further issues. Let&#39;s take `tnc` as an example (case B). F=
or that minimiser it&#39;s necessary to supply both objective AND gradient =
evaluations. The user must supply the first, but if the user cannot supply =
the gradient then `minimize` is asked to estimate it via numerical differen=
tiation. Estimating the gradient with numerical differentiation occurs in P=
ython. This kind of setup is taken care of by using the `optimize._differen=
tiable_functions.ScalarFunction` object.</div><div>Thus, a LowlevelCallable=
 that only furnishes objective evaluations is not much use, because the num=
erical differentiation (in Python) is still required to evaluate gradients,=
=C2=A0which requires at least N objective calls (from Python to whatever th=
e objective is contained in).</div><div><div>In order for this to be fully =
streamlined the numerical differentiation code would also have to be writte=
n in cython/c.<br></div></div><div>Note that the user can either supply the=
 objective function, separate objective and gradient functions, or a combin=
ed objective and gradient=C2=A0function. All those different cases have to =
be catered for. Moreover, for each of those cases one would either be suppl=
ying a `LowlevelCallable` or Python function.<br></div><div>Last, but not l=
east, most minimisers often ask numpy to perform operations, so porting min=
imisers to cython will still have python calls.</div><div><br></div><div>Th=
us, it&#39;s a large task just to amend a *single* minimiser. There are lot=
s of minimisers and one would want to do this in a uniform manner across th=
em all.</div><div><br></div><div>Given the width of such a project, one wou=
ld want to conclusively know that it would be worth it, and that&#39;s defi=
nitely not clear.</div><div><br></div><div>Andrew.</div><div><div><br></div=
></div><div><br></div><div><br></div></div>

--000000000000cce69d0611b330c0--

--===============1671367590695989219==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
SciPy-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/scipy-dev.python.org/
Member address: [email protected]

--===============1671367590695989219==--