SBCL dev regression testing with client code

Hello

Detecting SBCL regressions during the development cycle is much better
than only post freeze or post release. To this end, I explored
(prompted by Xof) how we can make testing user code with dev SBCL
easier. The following prototype solution is for GitHub.

The SBCL side
-------------

I added a rolling "latest-dev" release to SBCL, which gets updated on
push (intended for master only). This release has various binary
distribution tarballs (produced with binary-distribution.sh). These
changes are on the publish-latest-dev branch in my SBCL fork:

https://github.com/melisgl/sbcl/commits/publish-latest-dev/

This is what the latest-dev release that it creates looks like:

https://github.com/melisgl/sbcl/releases


The GitHub client side with Roswell
-----------------------------------

Then, a library can test on the latest and greatest SBCL by
downloading and installing a binary. In the following example, I
integrated this into the testing workflow of PAX:

https://github.com/melisgl/mgl-pax/commits/test-on-latest-sbcl/

PAX has a Roswell based setup because non-SBCL implementations need to
be tested too. I added a roswell-setup action that can download an
SBCL binary distribution tarball and make it available under Roswell.

https://github.com/melisgl/mgl-pax/blob/fda92dabf9def3d54f914ec6326e51b2c85b9d74/.github/actions/roswell-setup/action.yml

PAX's test.yml file is fairly independent from the complications:

https://github.com/melisgl/mgl-pax/blob/fda92dabf9def3d54f914ec6326e51b2c85b9d74/.github/workflows/test.yml

Here is a run:

https://github.com/melisgl/mgl-pax/actions/runs/21980661542

Note that the job gets annotated with the SBCL version because latest-
dev is a moving target. Here, that's "2.6.1.115.HEAD.13-b225129eb",
which includes the git version.

Also note that PAX tests are run on push but also daily to catch SBCL
regressions.


TODO
----

- It would likely be better to update the latest-dev release if tests
  on _all_ stable configurations/platforms pass.

- Note that the roswell-setup action probably belongs in SBCL. It's in
  PAX for ease of prototyping.

- There is no reporting of any kind. One only sees whether the
  individual test-stable and test-dev jobs succeeded. When test-dev
  fails but test-stable does not, that could indicate possible SBCL
  regressions.

- There is no automatic notification of sbcl-devel. That could be a
  can of worms. Maybe it's enough to have a failure in the daily test
  of client code alert the library author, who checks what's wrong
  and reports the issue if it's with SBCL.

- The current logic is based on the entire job suite failing or
  succeeding. Finer-grained regression testing (on the level of
  individual tests) is not considered yet.

- Maybe it would make sense to have a latest-stable rolling release as
  well. Possibly with the same set of binaries as latest-dev to ease
  comparisons.

- Still on GitHub but without Roswell, the situation would be similar
  and also simpler: one only needs to download the binary and install
  it. This could be another GitHub action like roswell-setup.

- In case a company has their own continual testing system, there is
  not a whole lot that this prototype provides yet, apart from the
  availability of a prebuilt library.


Before I take this further, let me know your thoughts about what has
been and what will need to be done.

Cheers,
Gábor

_______________________________________________
Sbcl-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/sbcl-devel