Re: Split docscrape out from numpydoc

--===============5195350774449907397==
Content-Type: multipart/alternative; boundary="000000000000742170064ffa4c83"

--000000000000742170064ffa4c83
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

> That sounds great! I=E2=80=99m still not sure if SciPy will want to add a=
 runtime
dependency on numpydoc, since we only have `numpy` at the minut

IIUC you'll be adding a dependency either way, whether it's `numpydoc` or
some vendored library that encapsulates just the
getdocobject/NumpyDocString (I'd vote for `numpydoc-parser` as the name for
this potential sub-library).

> Might it still make sense to split docscrape out into a separate repo
after your work is done, or do you see those as two orthogonal efforts?

No I don't think they're necessarily orthogonal. Nearly all of the
complexity associated with numpydoc stems from the sphinx/validation
interfaces. The bits actually dedicated to the docstring parsing are much
more straightforward. Splitting this into a separate library slightly
complicates the numpydoc maintenance/release process, but given how
infrequently the parsing code actually changes I wouldn't expect the
pinning of `numpydoc-parser` within the `numpydoc` sphinx extension to be a
prohibitive burden. I'm not against it, but nor am I involved in the
numpydoc release process - other numpydoc mainters' opinions are much more
valuable here!

>From a downstream library standpoint, the practical differences between a
dependency-less numpydoc and a separate numpydoc library should be minimal;
however, if that's a blocker for scipy then it's a stronger motivation for
splitting things up IMV!

~Ross

On Mon, Apr 20, 2026 at 12:52=E2=80=AFPM Lucas Colley <lucas.colley8@gmail.=
com>
wrote:

> Hi Ross,
>
> That sounds great! I=E2=80=99m still not sure if SciPy will want to add a=
 runtime
> dependency on numpydoc, since we only have `numpy` at the minute=E2=80=A6=
 but this
> is at least a step in the right direction!
>
> Might it still make sense to split docscrape out into a separate repo
> after your work is done, or do you see those as two orthogonal efforts?
>
> Cheers,
> Lucas
>
> On 20 Apr 2026, at 17:16, Ross Barnowski <[email protected]> wrote:
>
> =EF=BB=BF
> Hi Lucas,
>
> Indeed I think this is a good idea - SciPy is not the only project that
> depends (or would like to have depended) on the NumpyDocString and
> docscrape functionality without pulling in Sphinx (or any other dependenc=
y).
>
> One solution is to refactor numpydoc to fix the dependency footprint -
> i.e. make sphinx a soft dependency. That work is mostly complete in
> [numpy/numpydoc#651](https://github.com/numpy/numpydoc/pull/651). There
> are a few final integration tests (see checkboxes in the top post in the
> PR) that I'd like to run to build confidence that the solution works for
> everyone. That should include replacing the vendored code in scipy.
>
> Does this solution work for scipy?
>
> On Mon, Apr 20, 2026 at 5:06=E2=80=AFAM Lucas Colley via NumPy-Discussion=
 <
> [email protected]> wrote:
>
>> Hi all,
>>
>> A while ago I proposed splitting docscrape out from the numpydoc repo:
>> https://github.com/numpy/numpydoc/issues/619.
>>
>> Why? SciPy has a copy of the docscrape source which is used to generate
>> some docstrings for the public API. Copying source code like this is nev=
er
>> great given that the two copies can fall out of sync. Vendoring the enti=
re
>> numpydoc repo in SciPy, however, or adding numpydoc as a runtime
>> dependency, seem off the table, making the situation worse rather than
>> better.
>>
>> A better solution seems to be to have docscrape be a standalone project,
>> which SciPy can vendor more easily, and which numpydoc can either depend=
 on
>> or vendor. From a modularity perspective at least this seems ideal =E2=
=80=94 there
>> are use-cases where you want docscrape, but not numpydoc, available at
>> runtime. Joren suggested that I float this idea on the mailing list, giv=
en
>> that he just submitted a patch to the copy in SciPy without realising th=
at
>> it was vendored code =F0=9F=98=84.
>>
>> What I don't know is whether this would negatively impact maintenance
>> burden etc.? I assume there will be a bit of upfront cost in restructuri=
ng
>> the repos, and a little more if we decide to distribute docscrape as a
>> standalone project, but my hope would be that this wouldn't cause any
>> problems long-term?
>>
>> An alternative solution would be to extract the docscrape source from th=
e
>> numpydoc repo in a vendoring script in SciPy. While okay, that still lea=
ves
>> us having to track commits by hand instead of using e.g. a git submodule=
,
>> and isn't robust to upstream changes of directory structure.
>>
>> Feedback appreciated! Eric Larson responded on the PR asking whether
>> SciPy can introduce a dependency on numpydoc, but that is all so far.
>>
>> Cheers,
>> Lucas
>> _______________________________________________
>> NumPy-Discussion mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>> https://mail.python.org/mailman3//lists/numpy-discussion.python.org
>> Member address: [email protected]
>>
>

--000000000000742170064ffa4c83
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>&gt;=C2=A0<span class=3D"gmail-im">That sounds great!=
 I=E2=80=99m still not sure if SciPy will=20
want to add a runtime dependency on numpydoc, since we only have `numpy`
 at the minut</span></div><div><span class=3D"gmail-im"><br></span></div><d=
iv><span class=3D"gmail-im">IIUC you&#39;ll be adding a dependency either w=
ay, whether it&#39;s `numpydoc` or some vendored library that encapsulates =
just the getdocobject/NumpyDocString (I&#39;d vote for `numpydoc-parser` as=
 the name for this potential sub-library).</span></div><div><span class=3D"=
gmail-im"><br></span></div><div><span class=3D"gmail-im">&gt;=C2=A0</span><=
span class=3D"gmail-im">Might it still make sense to split docscrape out in=
to a
 separate repo after your work is done, or do you see those as two=20
orthogonal efforts?</span></div><div><span class=3D"gmail-im"><br></span></=
div><div><span class=3D"gmail-im">No I don&#39;t think they&#39;re necessar=
ily orthogonal. Nearly all of the complexity associated with numpydoc stems=
 from the sphinx/validation interfaces. The bits actually dedicated to the =
docstring parsing are much more straightforward. Splitting this into a sepa=
rate library slightly complicates the numpydoc maintenance/release process,=
 but given how infrequently the parsing code actually changes I wouldn&#39;=
t expect the pinning of `numpydoc-parser` within the `numpydoc` sphinx exte=
nsion to be a prohibitive burden. I&#39;m not against it, but nor am I invo=
lved in the numpydoc release process - other numpydoc mainters&#39; opinion=
s are much more valuable here!</span></div><div><span class=3D"gmail-im"><b=
r></span></div><div>From a downstream library standpoint, the practical dif=
ferences between a dependency-less numpydoc and a separate numpydoc library=
 should be minimal; however, if that&#39;s a blocker for scipy then it&#39;=
s a stronger motivation for splitting things up IMV!</div><div><br></div><d=
iv>~Ross</div></div><br><div class=3D"gmail_quote gmail_quote_container"><d=
iv dir=3D"ltr" class=3D"gmail_attr">On Mon, Apr 20, 2026 at 12:52=E2=80=AFP=
M Lucas Colley &lt;<a href=3D"mailto:[email protected]">lucas.colley8=
@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=
=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding=
-left:1ex"><div dir=3D"auto"><div dir=3D"ltr"></div><div dir=3D"ltr"><div d=
ir=3D"ltr">Hi Ross,</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">That s=
ounds great! I=E2=80=99m still not sure if SciPy will want to add a runtime=
 dependency on numpydoc, since we only have `numpy` at the minute=E2=80=A6 =
but this is at least a step in the right direction!</div><div dir=3D"ltr"><=
br></div><div dir=3D"ltr">Might it still make sense to split docscrape out =
into a separate repo after your work is done, or do you see those as two or=
thogonal efforts?</div><div dir=3D"ltr"><br></div><div dir=3D"ltr">Cheers,<=
/div><div dir=3D"ltr">Lucas</div></div><div dir=3D"ltr"><br><blockquote typ=
e=3D"cite">On 20 Apr 2026, at 17:16, Ross Barnowski &lt;<a href=3D"mailto:r=
[email protected]" target=3D"_blank">[email protected]</a>&gt; wrote:<br=
><br></blockquote></div><blockquote type=3D"cite"><div dir=3D"ltr">=EF=BB=
=BF<div dir=3D"ltr"><div>Hi Lucas,</div><div><br></div><div>Indeed I think =
this is a good idea - SciPy is not the only project that depends (or would =
like to have depended) on the NumpyDocString and docscrape functionality wi=
thout pulling in Sphinx (or any other dependency).</div><div><br></div><div=
>One solution is to refactor numpydoc to fix the dependency footprint - i.e=
. make sphinx a soft dependency. That work is mostly complete in [numpy/num=
pydoc#651](<a href=3D"https://github.com/numpy/numpydoc/pull/651" target=3D=
"_blank">https://github.com/numpy/numpydoc/pull/651</a>). There are a few f=
inal integration tests (see checkboxes in the top post in the PR) that I&#3=
9;d like to run to build confidence that the solution works for everyone. T=
hat should include replacing the vendored code in scipy.</div><div><br></di=
v><div>Does this solution work for scipy?=C2=A0</div></div><br><div class=
=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Apr 20, 2026=
 at 5:06=E2=80=AFAM Lucas Colley via NumPy-Discussion &lt;<a href=3D"mailto=
:[email protected]" target=3D"_blank">[email protected]=
</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:=
0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">=
Hi all,<br>
<br>
A while ago I proposed splitting docscrape out from the numpydoc repo: <a h=
ref=3D"https://github.com/numpy/numpydoc/issues/619" rel=3D"noreferrer" tar=
get=3D"_blank">https://github.com/numpy/numpydoc/issues/619</a>.<br>
<br>
Why? SciPy has a copy of the docscrape source which is used to generate som=
e docstrings for the public API. Copying source code like this is never gre=
at given that the two copies can fall out of sync. Vendoring the entire num=
pydoc repo in SciPy, however, or adding numpydoc as a runtime dependency, s=
eem off the table, making the situation worse rather than better.<br>
<br>
A better solution seems to be to have docscrape be a standalone project, wh=
ich SciPy can vendor more easily, and which numpydoc can either depend on o=
r vendor. From a modularity perspective at least this seems ideal =E2=80=94=
 there are use-cases where you want docscrape, but not numpydoc, available =
at runtime. Joren suggested that I float this idea on the mailing list, giv=
en that he just submitted a patch to the copy in SciPy without realising th=
at it was vendored code =F0=9F=98=84.<br>
<br>
What I don&#39;t know is whether this would negatively impact maintenance b=
urden etc.? I assume there will be a bit of upfront cost in restructuring t=
he repos, and a little more if we decide to distribute docscrape as a stand=
alone project, but my hope would be that this wouldn&#39;t cause any proble=
ms long-term?<br>
<br>
An alternative solution would be to extract the docscrape source from the n=
umpydoc repo in a vendoring script in SciPy. While okay, that still leaves =
us having to track commits by hand instead of using e.g. a git submodule, a=
nd isn&#39;t robust to upstream changes of directory structure.<br>
<br>
Feedback appreciated! Eric Larson responded on the PR asking whether SciPy =
can introduce a dependency on numpydoc, but that is all so far.<br>
<br>
Cheers,<br>
Lucas<br>
_______________________________________________<br>
NumPy-Discussion mailing list -- <a href=3D"mailto:numpy-discussion@python.=
org" target=3D"_blank">[email protected]</a><br>
To unsubscribe send an email to <a href=3D"mailto:numpy-discussion-leave@py=
thon.org" target=3D"_blank">[email protected]</a><br>
<a href=3D"https://mail.python.org/mailman3//lists/numpy-discussion.python.=
org" rel=3D"noreferrer" target=3D"_blank">https://mail.python.org/mailman3/=
/lists/numpy-discussion.python.org</a><br>
Member address: <a href=3D"mailto:[email protected]" target=3D"_blank">r=
[email protected]</a><br>
</blockquote></div>
</div></blockquote></div></blockquote></div>

--000000000000742170064ffa4c83--

--===============5195350774449907397==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
NumPy-Discussion mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3//lists/numpy-discussion.python.org
Member address: [email protected]

--===============5195350774449907397==--