Re: Fast versions of scipy.io.mmread/write

On Fri, Apr 14, 2023 at 9:38 AM <[email protected]> wrote:

> Great!
>
> Ralf Gommers wrote:
> > On Wed, Apr 12, 2023 at 9:57 AM [email protected] wrote:
> > > Hi,
> > > I'm the author of fast_matrix_market, a package that reads and writes
> > > Matrix Market files:
> > > https://github.com/alugowski/fast_matrix_market/tree/main/python
> > > The Python binding is API compatible with scipy.io.mmio, but about 25x
> > > faster. A few other handy improvements include 64-bit indices (only if
> > > necessary), direct csc/csr writes with no coo intermediary,
> longdouble. It
> > > passes the SciPy mmio test suite.
> > > Would there be any interest in somehow integrating this library into
> SciPy?
> > > The speed difference really does make a big difference for large files.
> > > The package is written in C++17 and pybind11. Threading is with a
> simple
> > > thread pool based on c++11 threads, but can be changed.
> > > A significant part of the speed increase is parsing with
> std::from_chars
> > > instead of strto*, which alone contributes a massive improvement.
> > > Thanks for this package and proposal Adam! I had a quick look at your
> > package, and it looks good to me. This kind of upstreaming of code, when
> > there's a clear performance benefit and the code is maintained, seems
> nice
> > to me. Just to make sure: are you planning to continue maintaining this
> > code? Either within SciPy only, or also as a separate package and keeping
> > the two in sync?
>
> Yes, I'm seeing increasing interest in the package so I'm going to keep
> maintaining it.
> I do have a lot of use of the C++ bindings, so it makes sense to keep it
> as a separate package in addition to within SciPy.
>
> > The C++17 should be fine as far as I can tell - our
> > http://scipy.github.io/devdocs/dev/toolchain.html has an upgrade to
> C++17
> > marked for 2022 already, so as long as things build with what MSVC
> provides
> > (the usual limiting factor), that should be good.
> > Cheers,
> > Ralf
>
> I've gone through the MSVC pains already to build Windows wheels with
> cibuildwheel, so that shouldn't be an issue.
>

That all sounds good, thanks Adam. I think the next steps are then to
proceed with your proposal integration and open a PR. Here is what I
suggest:
- add your code in a new private directory
`scipy/io/_fast_matrix_market/src/` with a README explaining where the code
comes from next to the `src`
- integrate it in the SciPy build system in `scipy/io/meson.build
- you have to (unfortunately) also still deal with the setup.py-based build
in one way or the other. We're going to throw it out soon, so you can
either ignore it and keep the slow `_mmio.py` implementation for that, or
integrate the C++ code in `scipy/io/setup.py`. The latter is a little
nicer, but in case it's tricky to get to work then keeping the slow code is
also okay. The only reason we will still keep it around for the 1.11.0
release (probably) is conda-forge on Windows, so that's a limited number of
users who'd get the slow code.


Cheers,
Ralf

_______________________________________________
SciPy-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/scipy-dev.python.org/
Member address: [email protected]