Re: ENH: allow scipy.sparse matrices to preserve `np.memmap` input data

Julien Jerphanion <[email protected]> Sun, 26 Mar 2023 20:46:30 +0000

Newsgroups	gmane.comp.python.scientific.devel
Message-ID	<YfrLroeKuw8i4Nu8nLxb_bz41gJ5JLFlhi_YRlDmbvh_0MPuIXXfkDuM0YlKmWj0cKahRpBhl5-iamkyMz17uLAptaCK_LHjMzkKu6zh6qo=@jjerphan.xyz>

Hi Aaron,

Thank you for reporting this use-case!

The current implementations of SciPy's sparse matrices cast arrays at creation,
but also after some operations as you mentioned.

Other issues and RFC reported similar usability problems and potential solutions
(with thread-offs not to break other use-cases). 


I think your request is in line with Scientific Python's Sparse Array Summit
which aims at identifying and addressing needs for the whole ecosystem
regarding Sparse data-structures:

https://discuss.scientific-python.org/t/sparse-array-summit/365

I also think participants to those discussions would benefit from your inputs.

What do other participants think?

Best,
Julien.

------- Original Message -------
On Sunday, March 26th, 2023 at 1:40 PM, Aaron Z. <[email protected]> wrote:


> Hi all - this is my first post here.
> 

> I've been using csr and coo matrices for large graph algorithms. I've found that scipy will beat frameworks like pregel on almost any task with large memory machines >200GB+ for storing large adjacency matrices. To run in environments with memory constraints, using numpy.memmap arrays for COO: (row, col, data) or CSR: (indices, indptr, data) has helped to reduce the memory footprint (obviously at the cost of some speed).
> 

> 

> However, getting `csr_matrix` and `coo_matrix` to accept `np.memmap` format involved quite a bit of hacky monkey patching, because currently the scipy constructors will convert everything to a base ndarray, effectively copying to memory from disk.
> 

> Would there be a good reason not to support this in a future version natively? The changes required would be just some usage of the `subok=True` kwarg for `np.array` which would preserve the `np.memmap` class and using `np.asanyarray` instead of `np.asarray` in other places.
> 

> For example, in the `csr_matrix.__init__`: https://github.com/scipy/scipy/blob/979102e73a8b9612ac3c97c438e40de26e46c9b7/scipy/sparse/_compressed.py#L69-L72
> `python self.indices = np.array(indices, copy=copy, dtype=idx_dtype, subok=True) self.indptr = np.array(indptr, copy=copy, dtype=idx_dtype, subok=True) self.data = np.array(data, copy=copy, dtype=dtype, subok=True)`
> keeps input data on disk.
> 

> When the user then performs an operation that creates a new matrix, for example matrix multiplication, it becomes more difficult to keep things to memmaps, because the user has to specify a filepath to keep the new matrix at. But I think a great first step would be to not cast the input data - this still leads to a decrease in memory usage, and the user can choose to memmap the output manually themselves.
> 

> I'd be happy to raise a PR for this if this has some support - looking forward to hearing thoughts!
> _______________________________________________
> SciPy-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/scipy-dev.python.org/
> Member address: [email protected]

_______________________________________________
SciPy-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/scipy-dev.python.org/
Member address: [email protected]

publickey - [email protected] - 0xDEDBF4EA.asc (application/pgp-keys, 3 KB)

signature.asc (application/pgp-signature, 855 B)