Re: Would you consider making ccallback part of the public scipy interface?

On Tue, Aug 29, 2023 at 11:48 AM Gabriel Fougeron <
[email protected]> wrote:

> Hi,
>
> This is my first message on the mailing list. The question was originally
> asked in the following github issue,
> https://github.com/scipy/scipy/issues/19148 .
>
> Here is the gist of it: I've been developing a symplectic ODE solver in
> Cython. Aiming for efficiency, the solver hugely benefits from scipy
> providing a public cython interface to blas/lapack. In other words, I'm
> very happy to use scipy.linalg.cython_blas.dgemm and such.
>
> For small systems, I clearly measure a bottleneck in the python machinery
> during the call to the user provided callback defining the ODE. I had left
> it at that for a few months till I looked at the input for
> scipy.integrate.quad and discovered the existence of scipy.LowLevelCallable.
>
> I tested it in my own code and it so happens that yes, it eliminates the
> bottleneck Iwas experiencing. Great.
> However, in order to access the pointer to the a c function in a
> scipy.LowLevelCallable, following
> https://github.com/scipy/scipy/blob/main/scipy/_lib/_ccallback_c.pyx , I
> need to import the follwoing:
>
>  from .ccallback cimport (ccallback_t, ccallback_prepare,
> ccallback_release, CCALLBACK_DEFAULTS, ccallback_signature_t)
>
> Those functions are not part of the public Cython interface in scipy as
> far as I can tell
> https://docs.scipy.org/doc/scipy/dev/contributor/public_cython_api.html ,
> so I had to copy paste the files
> https://github.com/scipy/scipy/blob/main/scipy/_lib/src/ccallback.h and
> https://github.com/scipy/scipy/blob/main/scipy/_lib/ccallback.pxd into my
> own project. This felt a bit hacky.
>
> Hence the question: would you consider making ccallback.pxd part of the
> public Cython API ?
>

Hi Gabriel, thanks for the suggestion! The reply is a bit late because the
question is a little hairy - both because it's a tricky bit of machinery,
and because there is no obvious place to expose this new Cython API. For
the `linalg` and `special` submodules we have `cython_linalg` and
`cython_special`, but we can't follow that pattern here because `_lib` is
private. If we'd follow the `cython_*` pattern, the logical place would be
`scipy.cython_scipy` next to LowlevelCallable. But I'm not sure I like
that:)

I think your request is very reasonable in principle though. Maybe you or
someone else have a better idea about where this should live. Also, I don't
think you shared your own code or a small self-contained example on the
GitHub issue - that could be instructive to see.

Cheers,
Ralf

_______________________________________________
SciPy-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/scipy-dev.python.org/
Member address: [email protected]