Re: Split docscrape out from numpydoc

--===============2673510023001313237==
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_8B51E2C8-0FBF-4446-A534-C2D3D0A3E338"


--Apple-Mail=_8B51E2C8-0FBF-4446-A534-C2D3D0A3E338
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hi Ross,

> On 21 Apr 2026, at 16:44, Ross Barnowski <[email protected]> wrote:
>=20
> > That sounds great! I=E2=80=99m still not sure if SciPy will want to =
add a runtime dependency on numpydoc, since we only have `numpy` at the =
minut
>=20
> IIUC you'll be adding a dependency either way, whether it's `numpydoc` =
or some vendored library that encapsulates just the =
getdocobject/NumpyDocString (I'd vote for `numpydoc-parser` as the name =
for this potential sub-library).


There is a pretty big difference between a dependency in the sense of =
https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#depe=
ndencies-optional-dependencies and a vendored dependency, at least for =
SciPy. With a vendored dependency, we have complete control over the =
(single) version, and importantly users can install a different version =
in their environment alongside SciPy without issue. With a proper =
dependency, we would be compelled to support a large range of versions =
of numpydoc to avoid making it impossible to install SciPy alongside =
other packages which overzealously pin the version of numpydoc. That has =
the potential to create extra maintenance churn for little benefit.

It would also be one extra point of variability to complicate the =
release process, which in turn is probably significant from a security =
perspective.

Anyway, `numpydoc-parser` sounds good to me! Let=E2=80=99s summarise =
that idea into the linked issue, possibly after any input from the =
numpydoc release team?

Cheers,
Lucas

> > Might it still make sense to split docscrape out into a separate =
repo after your work is done, or do you see those as two orthogonal =
efforts?
>=20
> No I don't think they're necessarily orthogonal. Nearly all of the =
complexity associated with numpydoc stems from the sphinx/validation =
interfaces. The bits actually dedicated to the docstring parsing are =
much more straightforward. Splitting this into a separate library =
slightly complicates the numpydoc maintenance/release process, but given =
how infrequently the parsing code actually changes I wouldn't expect the =
pinning of `numpydoc-parser` within the `numpydoc` sphinx extension to =
be a prohibitive burden. I'm not against it, but nor am I involved in =
the numpydoc release process - other numpydoc mainters' opinions are =
much more valuable here!
>=20
> =46rom a downstream library standpoint, the practical differences =
between a dependency-less numpydoc and a separate numpydoc library =
should be minimal; however, if that's a blocker for scipy then it's a =
stronger motivation for splitting things up IMV!
>=20
> ~Ross
>=20
> On Mon, Apr 20, 2026 at 12:52=E2=80=AFPM Lucas Colley =
<[email protected] <mailto:[email protected]>> wrote:
>> Hi Ross,
>>=20
>> That sounds great! I=E2=80=99m still not sure if SciPy will want to =
add a runtime dependency on numpydoc, since we only have `numpy` at the =
minute=E2=80=A6 but this is at least a step in the right direction!
>>=20
>> Might it still make sense to split docscrape out into a separate repo =
after your work is done, or do you see those as two orthogonal efforts?
>>=20
>> Cheers,
>> Lucas
>>=20
>>> On 20 Apr 2026, at 17:16, Ross Barnowski <[email protected] =
<mailto:[email protected]>> wrote:
>>>=20
>>> =EF=BB=BF
>>> Hi Lucas,
>>>=20
>>> Indeed I think this is a good idea - SciPy is not the only project =
that depends (or would like to have depended) on the NumpyDocString and =
docscrape functionality without pulling in Sphinx (or any other =
dependency).
>>>=20
>>> One solution is to refactor numpydoc to fix the dependency footprint =
- i.e. make sphinx a soft dependency. That work is mostly complete in =
[numpy/numpydoc#651](https://github.com/numpy/numpydoc/pull/651). There =
are a few final integration tests (see checkboxes in the top post in the =
PR) that I'd like to run to build confidence that the solution works for =
everyone. That should include replacing the vendored code in scipy.
>>>=20
>>> Does this solution work for scipy?=20
>>>=20
>>> On Mon, Apr 20, 2026 at 5:06=E2=80=AFAM Lucas Colley via =
NumPy-Discussion <[email protected] =
<mailto:[email protected]>> wrote:
>>>> Hi all,
>>>>=20
>>>> A while ago I proposed splitting docscrape out from the numpydoc =
repo: https://github.com/numpy/numpydoc/issues/619.
>>>>=20
>>>> Why? SciPy has a copy of the docscrape source which is used to =
generate some docstrings for the public API. Copying source code like =
this is never great given that the two copies can fall out of sync. =
Vendoring the entire numpydoc repo in SciPy, however, or adding numpydoc =
as a runtime dependency, seem off the table, making the situation worse =
rather than better.
>>>>=20
>>>> A better solution seems to be to have docscrape be a standalone =
project, which SciPy can vendor more easily, and which numpydoc can =
either depend on or vendor. =46rom a modularity perspective at least =
this seems ideal =E2=80=94 there are use-cases where you want docscrape, =
but not numpydoc, available at runtime. Joren suggested that I float =
this idea on the mailing list, given that he just submitted a patch to =
the copy in SciPy without realising that it was vendored code =F0=9F=98=84=
.
>>>>=20
>>>> What I don't know is whether this would negatively impact =
maintenance burden etc.? I assume there will be a bit of upfront cost in =
restructuring the repos, and a little more if we decide to distribute =
docscrape as a standalone project, but my hope would be that this =
wouldn't cause any problems long-term?
>>>>=20
>>>> An alternative solution would be to extract the docscrape source =
from the numpydoc repo in a vendoring script in SciPy. While okay, that =
still leaves us having to track commits by hand instead of using e.g. a =
git submodule, and isn't robust to upstream changes of directory =
structure.
>>>>=20
>>>> Feedback appreciated! Eric Larson responded on the PR asking =
whether SciPy can introduce a dependency on numpydoc, but that is all so =
far.
>>>>=20
>>>> Cheers,
>>>> Lucas
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list -- [email protected] =
<mailto:[email protected]>
>>>> To unsubscribe send an email to [email protected] =
<mailto:[email protected]>
>>>> https://mail.python.org/mailman3//lists/numpy-discussion.python.org
>>>> Member address: [email protected] <mailto:[email protected]>


--Apple-Mail=_8B51E2C8-0FBF-4446-A534-C2D3D0A3E338
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html aria-label=3D"message body"><head><meta http-equiv=3D"content-type" =
content=3D"text/html; charset=3Dutf-8"></head><body =
style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; =
line-break: after-white-space;"><div>Hi Ross,</div><div><br><blockquote =
type=3D"cite"><div>On 21 Apr 2026, at 16:44, Ross Barnowski =
&lt;[email protected]&gt; wrote:</div><br =
class=3D"Apple-interchange-newline"><div><div =
dir=3D"ltr"><div>&gt;&nbsp;<span class=3D"gmail-im">That sounds great! =
I=E2=80=99m still not sure if SciPy will want to add a runtime =
dependency on numpydoc, since we only have `numpy` at the =
minut</span></div><div><span =
class=3D"gmail-im"><br></span></div><div><span class=3D"gmail-im">IIUC =
you'll be adding a dependency either way, whether it's `numpydoc` or =
some vendored library that encapsulates just the =
getdocobject/NumpyDocString (I'd vote for `numpydoc-parser` as the name =
for this potential =
sub-library).</span></div></div></div></blockquote></div><div><br></div><d=
iv>There is a pretty big difference between a dependency in the sense =
of&nbsp;<a =
href=3D"https://packaging.python.org/en/latest/guides/writing-pyproject-to=
ml/#dependencies-optional-dependencies">https://packaging.python.org/en/la=
test/guides/writing-pyproject-toml/#dependencies-optional-dependencies</a>=
&nbsp;and a vendored dependency, at least for SciPy. With a vendored =
dependency, we have complete control over the (single) version, and =
importantly users can install a different version in their environment =
alongside SciPy without issue. With a proper dependency, we would be =
compelled to support a large range of versions of numpydoc to avoid =
making it impossible to install SciPy alongside other packages which =
overzealously pin the version of numpydoc. That has the potential to =
create extra maintenance churn for little =
benefit.</div><div><br></div><div>It would also be one extra point of =
variability to complicate the release process, which in turn is probably =
significant from a security =
perspective.</div><div><br></div><div>Anyway, `numpydoc-parser` sounds =
good to me! Let=E2=80=99s summarise that idea into the linked issue, =
possibly after any input from the numpydoc release =
team?</div><div><br></div><div>Cheers,</div><div>Lucas</div><div><br></div=
><div><blockquote type=3D"cite"><div><div dir=3D"ltr"><div><span =
class=3D"gmail-im">&gt;&nbsp;</span><span class=3D"gmail-im">Might it =
still make sense to split docscrape out into a
 separate repo after your work is done, or do you see those as two=20
orthogonal efforts?</span></div><div><span =
class=3D"gmail-im"><br></span></div><div><span class=3D"gmail-im">No I =
don't think they're necessarily orthogonal. Nearly all of the complexity =
associated with numpydoc stems from the sphinx/validation interfaces. =
The bits actually dedicated to the docstring parsing are much more =
straightforward. Splitting this into a separate library slightly =
complicates the numpydoc maintenance/release process, but given how =
infrequently the parsing code actually changes I wouldn't expect the =
pinning of `numpydoc-parser` within the `numpydoc` sphinx extension to =
be a prohibitive burden. I'm not against it, but nor am I involved in =
the numpydoc release process - other numpydoc mainters' opinions are =
much more valuable here!</span></div><div><span =
class=3D"gmail-im"><br></span></div><div>=46rom a downstream library =
standpoint, the practical differences between a dependency-less numpydoc =
and a separate numpydoc library should be minimal; however, if that's a =
blocker for scipy then it's a stronger motivation for splitting things =
up IMV!</div><div><br></div><div>~Ross</div></div><br><div =
class=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr" =
class=3D"gmail_attr">On Mon, Apr 20, 2026 at 12:52=E2=80=AFPM Lucas =
Colley &lt;<a =
href=3D"mailto:[email protected]">[email protected]</a>&gt; =
wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px =
0px 0px 0.8ex;border-left:1px solid =
rgb(204,204,204);padding-left:1ex"><div dir=3D"auto"><div =
dir=3D"ltr"></div><div dir=3D"ltr"><div dir=3D"ltr">Hi Ross,</div><div =
dir=3D"ltr"><br></div><div dir=3D"ltr">That sounds great! I=E2=80=99m =
still not sure if SciPy will want to add a runtime dependency on =
numpydoc, since we only have `numpy` at the minute=E2=80=A6 but this is =
at least a step in the right direction!</div><div =
dir=3D"ltr"><br></div><div dir=3D"ltr">Might it still make sense to =
split docscrape out into a separate repo after your work is done, or do =
you see those as two orthogonal efforts?</div><div =
dir=3D"ltr"><br></div><div dir=3D"ltr">Cheers,</div><div =
dir=3D"ltr">Lucas</div></div><div dir=3D"ltr"><br><blockquote =
type=3D"cite">On 20 Apr 2026, at 17:16, Ross Barnowski &lt;<a =
href=3D"mailto:[email protected]" =
target=3D"_blank">[email protected]</a>&gt; =
wrote:<br><br></blockquote></div><blockquote type=3D"cite"><div =
dir=3D"ltr">=EF=BB=BF<div dir=3D"ltr"><div>Hi =
Lucas,</div><div><br></div><div>Indeed I think this is a good idea - =
SciPy is not the only project that depends (or would like to have =
depended) on the NumpyDocString and docscrape functionality without =
pulling in Sphinx (or any other =
dependency).</div><div><br></div><div>One solution is to refactor =
numpydoc to fix the dependency footprint - i.e. make sphinx a soft =
dependency. That work is mostly complete in [numpy/numpydoc#651](<a =
href=3D"https://github.com/numpy/numpydoc/pull/651" =
target=3D"_blank">https://github.com/numpy/numpydoc/pull/651</a>). There =
are a few final integration tests (see checkboxes in the top post in the =
PR) that I'd like to run to build confidence that the solution works for =
everyone. That should include replacing the vendored code in =
scipy.</div><div><br></div><div>Does this solution work for =
scipy?&nbsp;</div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" =
class=3D"gmail_attr">On Mon, Apr 20, 2026 at 5:06=E2=80=AFAM Lucas =
Colley via NumPy-Discussion &lt;<a =
href=3D"mailto:[email protected]" =
target=3D"_blank">[email protected]</a>&gt; =
wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px =
0px 0px 0.8ex;border-left:1px solid =
rgb(204,204,204);padding-left:1ex">Hi all,<br>
<br>
A while ago I proposed splitting docscrape out from the numpydoc repo: =
<a href=3D"https://github.com/numpy/numpydoc/issues/619" =
rel=3D"noreferrer" =
target=3D"_blank">https://github.com/numpy/numpydoc/issues/619</a>.<br>
<br>
Why? SciPy has a copy of the docscrape source which is used to generate =
some docstrings for the public API. Copying source code like this is =
never great given that the two copies can fall out of sync. Vendoring =
the entire numpydoc repo in SciPy, however, or adding numpydoc as a =
runtime dependency, seem off the table, making the situation worse =
rather than better.<br>
<br>
A better solution seems to be to have docscrape be a standalone project, =
which SciPy can vendor more easily, and which numpydoc can either depend =
on or vendor. =46rom a modularity perspective at least this seems ideal =
=E2=80=94 there are use-cases where you want docscrape, but not =
numpydoc, available at runtime. Joren suggested that I float this idea =
on the mailing list, given that he just submitted a patch to the copy in =
SciPy without realising that it was vendored code =F0=9F=98=84.<br>
<br>
What I don't know is whether this would negatively impact maintenance =
burden etc.? I assume there will be a bit of upfront cost in =
restructuring the repos, and a little more if we decide to distribute =
docscrape as a standalone project, but my hope would be that this =
wouldn't cause any problems long-term?<br>
<br>
An alternative solution would be to extract the docscrape source from =
the numpydoc repo in a vendoring script in SciPy. While okay, that still =
leaves us having to track commits by hand instead of using e.g. a git =
submodule, and isn't robust to upstream changes of directory =
structure.<br>
<br>
Feedback appreciated! Eric Larson responded on the PR asking whether =
SciPy can introduce a dependency on numpydoc, but that is all so =
far.<br>
<br>
Cheers,<br>
Lucas<br>
_______________________________________________<br>
NumPy-Discussion mailing list -- <a =
href=3D"mailto:[email protected]" =
target=3D"_blank">[email protected]</a><br>
To unsubscribe send an email to <a =
href=3D"mailto:[email protected]" =
target=3D"_blank">[email protected]</a><br>
<a =
href=3D"https://mail.python.org/mailman3//lists/numpy-discussion.python.or=
g" rel=3D"noreferrer" =
target=3D"_blank">https://mail.python.org/mailman3//lists/numpy-discussion=
.python.org</a><br>
Member address: <a href=3D"mailto:[email protected]" =
target=3D"_blank">[email protected]</a><br>
</blockquote></div>
</div></blockquote></div></blockquote></div>
</div></blockquote></div><br></body></html>=

--Apple-Mail=_8B51E2C8-0FBF-4446-A534-C2D3D0A3E338--

--===============2673510023001313237==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
NumPy-Discussion mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3//lists/numpy-discussion.python.org
Member address: [email protected]

--===============2673510023001313237==--