Re: Roadmap to Sparse Array and deprecation of Sparse Matrix

On Tue, Dec 5, 2023 at 8:22 AM Dan Schult <[email protected]> wrote:

> I'd like to get feedback on a plan to move forward with the conversion to
> Sparse Arrays and the deprecation of Sparse Matrices. Let me know if there
> is a better way to do this. I could e.g. make a PR to the roadmap part of
> the docs if that's preferred. But this probably gets a larger audience, so
> I'll start here. I've discussed this with people working on sparse arrays,
> (special thanks to @stefan and @perimosocordiae ).
>

Thanks for working on this Dan. This looks promising. And discussing it
here sounds right - a roadmap update can be made after indeed (there's also
other roadmap updates we need to do).


>
> Roadmap
> -------
>
> Current work:
> - Provide a feature-complete sparse array api (including 1d-array and
> construction functions).
>   - provide construction functions for sparse arrays <**very close**>
>   - extend sparse array api to 1d arrays.
>     - coo and dok formats <close but probs early v1.13dev>
>     - some 1d format that uses 2d csr sparsetools to do 1d things:
> indexing/min-max/arithmetic. This could be built within the current 2d
> coo/csr classes or as a new format. <v1.13dev>
>
> Next steps:
> - Help other libraries convert to sparse arrays from sparse matrices.
>     - networkx is already migrated
>     - sklearn is [in progress 27090](
> https://github.com/scikit-learn/scikit-learn/issues/27090)
>     - scikit-image [preparation in 6261](
> https://github.com/scikit-image/scikit-image/pull/6261)
>

This all sounds good to me.


> - wait for sparse array code to mature (1 cycle?)
>

One release cycle after the dust settles seems reasonable indeed. We've
seen a fair number of bug reports over the past couple of releases, but
once scikit-learn & co have support and usage is heavier, the feedback
cycle should be faster. So one release cycle with no blocking issues being
opened seems right.


> - create deprecation guide, transition guide, helpful scripts to flag code
> for further examination.
>

The transition guide and scripts could be added earlier, right? No need to
wait.


> - add deprecation warnings for sparse matrix.
>

This is the critical part, and I think requires feedback from at least
scikit-learn. They probably need a number of release cycles between having
sparse array support completed and sparse matrices being deprecated. Since
once matrices are deprecated, I imagine they should be removed from
scikit-learn's own code base at that point. Which means their range of
supported SciPy versions will get a bit narrow, and they tend to not like
that.

- shift focus to numpy and deprecation/removal of numpy matrix.
>

Agreed that numpy.matrix can be deprecated as soon as the sparse matrix
deprecation has landed (or released, depending on numpy/scipy release cycle
timing).


> - wait for requisite time after deprecation announced.
> - remove sparse matrix
> - start api shift of construction function names (`diags`, `block`, etc.)
>     - Note: as a whole, the construction functions will undergo two name
> shifts. Once to move from matrix creation to new functions for array
> creation (i.e. `eye` -> `eye_array`). Then a second move to change names to
> match the array_api name (i.e. `eye_array` to `eye`) after sparse matrices
> are removed.
>

It sounds a bit concerning that there's another large deprecation and code
move that downstream packages/users have to go through after sparse arrays
have been declared mature. So the `eye_array` & co names should probably
stay around for quite a while as aliases (even if hidden in the docs) to
the updated `eye` & co names, in order to not force another migration with
warnings and a fixed timeline on users.

This isn't much of an issue for your overall plan I'd say. Just avoid
deprecating the `_array` names, rather add a comment to only do it in a
major release or after 3-4 years.

Cheers,
Ralf


- add construction function names matching array_api names.
> - deprecate the transition construction function names.
>
> Timeframe
> ---------
> Steps with time impact:
> 1) feature-complete sparse array api
> 2) wait one release while helping other libraries convert to sparse array.
> 3) deprecation warnings for sparse matrix
> 4) wait requisite release cycles
> 5) remove sparse matrix code
> 6) work on function name shift deprecation process
>
> One possible timing could be:
> - scipy-1.13 feature-complete array api
> - scipy-1.14 add deprecation warnings to matrix api
> - scipy-1.16 remove matrix api
> - scipy-1.17 deprecate for name changes
> - scipy-1.19 complete name changes
> _______________________________________________
> SciPy-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/scipy-dev.python.org/
> Member address: [email protected]
>

_______________________________________________
SciPy-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/scipy-dev.python.org/
Member address: [email protected]