Re: SBCL dev regression testing with client code

On Fri, Feb 13, 2026 at 3:19 PM Gábor Melis <[email protected]> wrote:
>
> Hello
>
> Detecting SBCL regressions during the development cycle is much better
> than only post freeze or post release.

I regularly build the whole quicklisp, but without running tests,
because asdf:test-system, in its wisdom, doesn't return anything
useful.
It still catches some compiler errors and warnings.


> The SBCL side
> -------------
>
> I added a rolling "latest-dev" release to SBCL, which gets updated on
> push (intended for master only). This release has various binary
> distribution tarballs (produced with binary-distribution.sh). These
> changes are on the publish-latest-dev branch in my SBCL fork:
>
> https://github.com/melisgl/sbcl/commits/publish-latest-dev/
>
> This is what the latest-dev release that it creates looks like:
>
> https://github.com/melisgl/sbcl/releases
>
>
> The GitHub client side with Roswell
> -----------------------------------
>
> Then, a library can test on the latest and greatest SBCL by
> downloading and installing a binary. In the following example, I
> integrated this into the testing workflow of PAX:
>
> https://github.com/melisgl/mgl-pax/commits/test-on-latest-sbcl/
>
> PAX has a Roswell based setup because non-SBCL implementations need to
> be tested too. I added a roswell-setup action that can download an
> SBCL binary distribution tarball and make it available under Roswell.
>
> https://github.com/melisgl/mgl-pax/blob/fda92dabf9def3d54f914ec6326e51b2c85b9d74/.github/actions/roswell-setup/action.yml
>
> PAX's test.yml file is fairly independent from the complications:
>
> https://github.com/melisgl/mgl-pax/blob/fda92dabf9def3d54f914ec6326e51b2c85b9d74/.github/workflows/test.yml
>
> Here is a run:
>
> https://github.com/melisgl/mgl-pax/actions/runs/21980661542
>
> Note that the job gets annotated with the SBCL version because latest-
> dev is a moving target. Here, that's "2.6.1.115.HEAD.13-b225129eb",
> which includes the git version.
>
> Also note that PAX tests are run on push but also daily to catch SBCL
> regressions.
>
>
> TODO
> ----
>
> - It would likely be better to update the latest-dev release if tests
>   on _all_ stable configurations/platforms pass.
>
> - Note that the roswell-setup action probably belongs in SBCL. It's in
>   PAX for ease of prototyping.
>
> - There is no reporting of any kind. One only sees whether the
>   individual test-stable and test-dev jobs succeeded. When test-dev
>   fails but test-stable does not, that could indicate possible SBCL
>   regressions.
>
> - There is no automatic notification of sbcl-devel. That could be a
>   can of worms. Maybe it's enough to have a failure in the daily test
>   of client code alert the library author, who checks what's wrong
>   and reports the issue if it's with SBCL.
>
> - The current logic is based on the entire job suite failing or
>   succeeding. Finer-grained regression testing (on the level of
>   individual tests) is not considered yet.
>
> - Maybe it would make sense to have a latest-stable rolling release as
>   well. Possibly with the same set of binaries as latest-dev to ease
>   comparisons.
>
> - Still on GitHub but without Roswell, the situation would be similar
>   and also simpler: one only needs to download the binary and install
>   it. This could be another GitHub action like roswell-setup.
>
> - In case a company has their own continual testing system, there is
>   not a whole lot that this prototype provides yet, apart from the
>   availability of a prebuilt library.
>
>
> Before I take this further, let me know your thoughts about what has
> been and what will need to be done.
>
> Cheers,
> Gábor
>
> _______________________________________________
> Sbcl-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/sbcl-devel


_______________________________________________
Sbcl-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/sbcl-devel