Re: RFC: decouple doctesting from refguide-check
Ralf Gommers <[email protected]> Fri, 23 Feb 2024 11:11:41 +0100
| Newsgroups | gmane.comp.python.scientific.devel |
|---|---|
| Message-ID | <CABL7CQiyhS_domW=KmtAkX4s0JzCJZuDXu4tZV++2rKVDw0RMQ@mail.gmail.com> |
--===============4250672282930928216== Content-Type: multipart/alternative; boundary="000000000000f3048b061209c97b" --000000000000f3048b061209c97b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Feb 22, 2024 at 2:20=E2=80=AFPM Evgeni Burovski <evgeny.burovskiy@g= mail.com> wrote: > Hi, > > tl;dr: here's an RFC for the plan to decouple the doctesting machinery > from refguide-check and use pytest as a runner. > The plan and several RFC points below, the original email is quoted > further down. > > Context: to keep examples in the documentation current, we use a modified > doctesting machinery in the refguide-check tool. The previous iteration w= as > to decouple the machinery into a separate tool with the API compatible wi= th > the standard library doctest module. Several people expressed a preferen= ce > to use pytest as the test runner. > Thanks to great work by Sheila Kahwai during her Quansight internship las= t > summer, we now have a pytest plugin layer. > > The tool currently lives under https://github.com/ev-br/scpdt, and here's > the SciPy PR https://github.com/scipy/scipy/pull/20127 which plumbs it > through the `dev.py` interface: > > $ python dev.py test -s linalg --doctest > > runs the doctests in the linalg module etc. > > The plan is > - move the tool repository to live under the scipy github organization; > - add it as a git submodule; > - merge the PR, remove now-duplicate parts of refguide-check; > - keep evolving the tool. > > I believe if the tool lives under the scipy organization, commit and writ= e > access will be shared with all the maintainer's team. Either way, I'm hap= py > to continue taking the lead maintaining it. > > Does this make sense, do you see anything I'm missing? > That sounds reasonable to me. Git submodule vs. separate package to install is a bit of a toss-up, either way works. Cheers, Ralf > Also there's a question about naming: of the tool itself and of the > `dev.py` command: > > 1. The tool needs renaming: 'scpdt' is awful, and was a quick name of the > throw-away experiment. The previous suggestion from Ralf was > "scipy-doctest": > Over at > https://github.com/scipy/scipy/pull/20127#pullrequestreview-1893722684, > Pamphile coined "smoke-docs" to parallel "smoke tests". > > 2. the `dev.py` interface: the current one is `$dev.py test --doctests`, > and it runs doctests _only_. Note the difference to `pytest --doctest`: > this runs both unit tests and doctests. > > So maybe `dev.py test --doctest-only`? Or indeed Pamphile's `dev.py test > --smoke-docs`? > > What we certainly do not want is to mix doctests and unit tests. These ar= e > two different things: testing is through unit tests, and doctests are onl= y > an implementation detail of how we keep documentation current. > > Thoughts? > > > > > Evgeni Burovski wrote: > > Hi, > > > > Code examples in the SciPy and NumPy documentation are doctested, with > > a modified doctesting machinery which understands floating point, > > numpy formatting and some details of our documentation and API. > > > > Our modified doctesting machinery is buried in the refguide-check > > utility, which also does several unrelated things, all of which are > > tightly coupled to each other and to the libraries themselves. It has > > been sometimes described as hard to understand or work with or extend > > due to this tight coupling and a lack of a dedicated test suite. > > The numpy and scipy versions of the utility are both vendored, and the > > numpy version has diverged from the scipy version somewhat. > > > > Following a discussion in https://github.com/numpy/numpy/issues/21070 > > I did a small experiment to decouple the doctesting into a separate > > package, so that it's easier to consolidate the two versions. Plus, a > > separate repo is generally easier to maintain, configure, and possibly > > extend or adopt to other projects. The work-in-progress result is > > here: https://github.com/ev-br/scpdt > > > > It currently can run the full doctesting of the scipy API > > documentation (docstrings of objects) --- see > > https://github.com/ev-br/scpdt/pull/33 (the log of a test run with > > warnings turned to errors is in the GH actions: > > https://github.com/ev-br/scpdt/runs/6743881766?check_suite_focus=3Dtrue= ). > > Note that it shows, among other things, a couple of deprecation > > warnings our docs have accumulated :-). > > > > The API of the tool closely follows that of the standard library > > doctest module and provides (nearly) drop-in replacements for doctest > > checking, parsing, finding and running. Various configuration options > > for our modifications are collected into a single bag object which is > > internally passed around. This way, it's user-configurable all the way > > from a plain standard doctest module behavior to what refguide-check > > does now. > > > > There are a couple of wrinkles to iron out; overall it does what > > refguide-check does already. One missing bit is doctesting rst or > > other text files, but it's coming soon. > > > > The current plan is to: > > - verify that the standalone version does not miss things checked by > > the refguide-check > > - plumb it through the SciPy dev interface and rip out the > > refguide-check bundled doctesting utilities. > > - Sync changes that NumPy version of refguide-check accumulated over ti= me > > - Make sure it correctly tests the NumPy docs, too. > > - Better document the internals, there is currently only a readme file. > > If someone's interested to join me working on these, great, the more > > the merrier :-). > > > > I think it could make sense to move the tool's repository to the scipy > > github org (or maybe even numpy org?). I'm offering to maintain it > > regardless of the location. Thoughts? > > > > Cheers, > > > > Evgeni > _______________________________________________ > SciPy-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/scipy-dev.python.org/ > Member address: [email protected] > --000000000000f3048b061209c97b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Thu, Feb 22, 2024 at 2:20=E2=80=AF= PM Evgeni Burovski <<a href=3D"mailto:[email protected]">evgeny= [email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_quo= te" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204= );padding-left:1ex">Hi,<br> <br> tl;dr: here's an RFC for the plan to decouple the doctesting machinery = from refguide-check and use pytest as a runner.<br> The plan and several RFC points below, the original email is quoted further= down.<br> <br> Context: to keep examples in the documentation current, we use a modified d= octesting machinery in the refguide-check tool. The previous iteration was = to decouple the machinery into a separate tool with the API compatible with= the=C2=A0 standard library doctest module. Several people expressed a pref= erence to use pytest as the test runner. <br> Thanks to great work by Sheila Kahwai during her Quansight internship last = summer, we now have a pytest plugin layer. <br> <br> The tool currently lives under <a href=3D"https://github.com/ev-br/scpdt" r= el=3D"noreferrer" target=3D"_blank">https://github.com/ev-br/scpdt</a>, and= here's the SciPy PR <a href=3D"https://github.com/scipy/scipy/pull/201= 27" rel=3D"noreferrer" target=3D"_blank">https://github.com/scipy/scipy/pul= l/20127</a> which plumbs it through the `dev.py` interface:<br> <br> $ python dev.py test -s linalg --doctest<br> <br> runs the doctests in the linalg module etc.<br> <br> The plan is<br> - move the tool repository to live under the scipy github organization;<br> - add it as a git submodule;<br> - merge the PR, remove now-duplicate parts of refguide-check;<br> - keep evolving the tool.<br> <br> I believe if the tool lives under the scipy organization, commit and write = access will be shared with all the maintainer's team. Either way, I'= ;m happy to continue taking the lead maintaining it.<br> <br> Does this make sense,=C2=A0 do you see anything I'm missing?<br></block= quote><div><br></div><div>That sounds reasonable to me. Git submodule vs. s= eparate package to install is a bit of a toss-up, either way works.</div><d= iv><br></div><div>Cheers,<br></div><div>Ralf</div><div><br></div><div><br><= /div><div>=C2=A0<br></div><blockquote class=3D"gmail_quote" style=3D"margin= :0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"= > Also there's a question about naming: of the tool itself and of the `de= v.py` command:<br> <br> 1. The tool needs renaming: 'scpdt' is awful, and was a quick name = of the throw-away experiment. The previous suggestion from Ralf was "s= cipy-doctest":<br> Over at <a href=3D"https://github.com/scipy/scipy/pull/20127#pullrequestrev= iew-1893722684" rel=3D"noreferrer" target=3D"_blank">https://github.com/sci= py/scipy/pull/20127#pullrequestreview-1893722684</a>, Pamphile coined "= ;smoke-docs" to parallel "smoke tests". <br> <br> 2. the `dev.py` interface: the current one is `$dev.py test --doctests`, an= d it runs doctests _only_. Note the difference to `pytest --doctest`: this = runs both unit tests and doctests. <br> <br> So maybe `dev.py test --doctest-only`? Or indeed Pamphile's `dev.py tes= t --smoke-docs`?<br> <br> What we certainly do not want is to mix doctests and unit tests. These are = two different things: testing is through unit tests, and doctests are only = an implementation detail of how we keep documentation current.<br> <br> Thoughts?<br> <br> <br> <br> <br> Evgeni Burovski wrote:<br> > Hi,<br> > <br> > Code examples in the SciPy and NumPy documentation are doctested, with= <br> > a modified doctesting machinery which understands floating point,<br> > numpy formatting and some details of our documentation and API.<br> > <br> > Our modified doctesting machinery is buried in the refguide-check<br> > utility, which also does several unrelated things, all of which are<br= > > tightly coupled to each other and to the libraries themselves. It has<= br> > been sometimes described as hard to understand or work with or extend<= br> > due to this tight coupling and a lack of a dedicated test suite.<br> > The numpy and scipy versions of the utility are both vendored, and the= <br> > numpy version has diverged from the scipy version somewhat.<br> > <br> > Following a discussion in <a href=3D"https://github.com/numpy/numpy/is= sues/21070" rel=3D"noreferrer" target=3D"_blank">https://github.com/numpy/n= umpy/issues/21070</a><br> > I did a small experiment to decouple the doctesting into a separate<br= > > package, so that it's easier to consolidate the two versions. Plus= , a<br> > separate repo is generally easier to maintain, configure, and possibly= <br> > extend or adopt to other projects. The work-in-progress result is<br> > here: <a href=3D"https://github.com/ev-br/scpdt" rel=3D"noreferrer" ta= rget=3D"_blank">https://github.com/ev-br/scpdt</a><br> > <br> > It currently can run the full doctesting of the scipy API<br> > documentation (docstrings of objects) --- see<br> > <a href=3D"https://github.com/ev-br/scpdt/pull/33" rel=3D"noreferrer" = target=3D"_blank">https://github.com/ev-br/scpdt/pull/33</a> (the log of a = test run with<br> > warnings turned to errors is in the GH actions:<br> > <a href=3D"https://github.com/ev-br/scpdt/runs/6743881766?check_suite_= focus=3Dtrue" rel=3D"noreferrer" target=3D"_blank">https://github.com/ev-br= /scpdt/runs/6743881766?check_suite_focus=3Dtrue</a>).<br> > Note that it shows, among other things, a couple of deprecation<br> > warnings our docs have accumulated :-).<br> > <br> > The API of the tool closely follows that of the standard library<br> > doctest module and provides (nearly) drop-in replacements for doctest<= br> > checking, parsing, finding and running. Various configuration options<= br> > for our modifications are collected into a single bag object which is<= br> > internally passed around. This way, it's user-configurable all the= way<br> > from a plain standard doctest module behavior to what refguide-check<b= r> > does now.<br> > <br> > There are a couple of wrinkles to iron out; overall it does what<br> > refguide-check does already. One missing bit is doctesting rst or<br> > other text files, but it's coming soon.<br> > <br> > The current plan is to:<br> > - verify that the standalone version does not miss things checked by<b= r> > the refguide-check<br> > - plumb it through the SciPy dev interface and rip out the<br> > refguide-check bundled doctesting utilities.<br> > - Sync changes that NumPy version of refguide-check accumulated over t= ime<br> > - Make sure it correctly tests the NumPy docs, too.<br> > - Better document the internals, there is currently only a readme file= .<br> > If someone's interested to join me working on these, great, the mo= re<br> > the merrier :-).<br> > <br> > I think it could make sense to move the tool's repository to the s= cipy<br> > github org (or maybe even numpy org?). I'm offering to maintain it= <br> > regardless of the location. Thoughts?<br> > <br> > Cheers,<br> > <br> > Evgeni<br> _______________________________________________<br> SciPy-Dev mailing list -- <a href=3D"mailto:[email protected]" target=3D= "_blank">[email protected]</a><br> To unsubscribe send an email to <a href=3D"mailto:[email protected]= g" target=3D"_blank">[email protected]</a><br> <a href=3D"https://mail.python.org/mailman3/lists/scipy-dev.python.org/" re= l=3D"noreferrer" target=3D"_blank">https://mail.python.org/mailman3/lists/s= cipy-dev.python.org/</a><br> Member address: <a href=3D"mailto:[email protected]" target=3D"_b= lank">[email protected]</a><br> </blockquote></div></div> --000000000000f3048b061209c97b-- --===============4250672282930928216== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ SciPy-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/scipy-dev.python.org/ Member address: [email protected] --===============4250672282930928216==--