Re: Fast versions of scipy.io.mmread/write

On Wed, Apr 12, 2023 at 9:57 AM <[email protected]> wrote:

> Hi,
>
> I'm the author of fast_matrix_market, a package that reads and writes
> Matrix Market files:
> https://github.com/alugowski/fast_matrix_market/tree/main/python
>
> The Python binding is API compatible with scipy.io.mmio, but about 25x
> faster. A few other handy improvements include 64-bit indices (only if
> necessary), direct csc/csr writes with no coo intermediary, longdouble. It
> passes the SciPy mmio test suite.
>
> Would there be any interest in somehow integrating this library into SciPy?
>
> The speed difference really does make a big difference for large files.
>
> The package is written in C++17 and pybind11. Threading is with a simple
> thread pool based on c++11 threads, but can be changed.
> A significant part of the speed increase is parsing with std::from_chars
> instead of strto*, which alone contributes a massive improvement.
>

Thanks for this package and proposal Adam! I had a quick look at your
package, and it looks good to me. This kind of upstreaming of code, when
there's a clear performance benefit and the code is maintained, seems nice
to me. Just to make sure: are you planning to continue maintaining this
code? Either within SciPy only, or also as a separate package and keeping
the two in sync?

The C++17 should be fine as far as I can tell - our
http://scipy.github.io/devdocs/dev/toolchain.html has an upgrade to C++17
marked for 2022 already, so as long as things build with what MSVC provides
(the usual limiting factor), that should be good.

Cheers,
Ralf

_______________________________________________
SciPy-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/scipy-dev.python.org/
Member address: [email protected]