Re: Fast versions of scipy.io.mmread/write
| Newsgroups | gmane.comp.python.scientific.devel |
|---|---|
| Message-ID | <[email protected]> |
Stefan van der Walt wrote: > On Tue, Apr 11, 2023, at 17:28, [email protected] wrote: > > The Python binding is API compatible with scipy.io.mmio, but about 25x > > faster. A few other handy improvements include 64-bit indices (only if > > necessary), direct csc/csr writes with no coo intermediary, longdouble. > > It passes the SciPy mmio test suite. > > Would there be any interest in somehow integrating this library into SciPy? > > That looks like a good upgrade without any obvious downsides; thanks for the proposal! I'm curious also to see how memory consumption compares during load. > Stéfan I take the same approach as the current SciPy mmio routines, allocate the final data structure then write values directly there. The main difference is that I effectively have a larger I/O buffer. I fread() large chunks (default 1MB works well, but configurable) then parse the chunks in parallel. I keep a queue of ready chunks to make sure the parser threads don't have to wait for I/O. -Adam _______________________________________________ SciPy-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: [email protected]