Re: regarding build speed on riscv64

Hi Emilio,

On 2026-07-20 23:32, Aurelien Jarno wrote:
> Hi Emilio,
> 
> On 2026-06-25 15:21, Emilio Pozuelo Monfort wrote:
> > On 24/06/2026 19:34, Aurelien Jarno wrote:
> > > Hi Emilio,
> > > 
> > > On 2026-06-24 14:06, Emilio Pozuelo Monfort wrote:
> > > > Hi Aurelien,
> > > > 
> > > > On 27/05/2026 21:31, Aurelien Jarno wrote:
> > > > > Hi Paul and release team,
> > > > > 
> > > > > On 2026-05-26 21:30, Paul Gevers wrote:
> > > > > > Hi riscv64 porters,
> > > > > > 
> > > > > > In bug 1092153 [1] we discussed the slowness of the riscv64 hardware
> > > > > > available for Debian. At the time we accepted the situation as a
> > > > > > bootstrapping problem. Recently we've witnessed some delays because of
> > > > > > riscv64 build speed (e.g. migration delay for linux kernel CVE fixes), and
> > > > > > since riscv64 got added to ci.debian.net it needed shepherding to stay up
> > > > > > and a lot of tests actually fail because of the slowness.
> > > > > > 
> > > > > > Hence, I'd like to request an update on the situation. When do you think the
> > > > > > situation can be improved realistically? As an example, I've heard that for
> > > > > > some of the faster hardware, the kernel patches still aren't accepted
> > > > > > upstream. Also, on what timescale can the slow hardware (both buildds and
> > > > > > the ci.d.n hardware) be replaced with faster hardware that runs reliably?
> > > > > 
> > > > > I am answering here about the build daemons part as the debci part has
> > > > > already been answered by Bo Yu.
> > > > > 
> > > > > First of all the requirements for the build daemons is a bit more
> > > > > challenging than debci, as we want to run them on our own hardware,
> > > > > using the Debian (bpo) kernel, in a rack-mountable format. As pointed
> > > > > out in bug #1092153, we had identified two possible hardware platforms.
> > > > > Unfortunately neither turn out to be suitable for build daemons.
> > > > > Complete kernel support for the HiFive Premier P550 is still lacking,
> > > > > and we experienced some stability issues with the Milk-V Pioneer.
> > > > > 
> > > > > We therefore looked for alternatives and found the Milk-V Jupiter board
> > > > > with a SpacemiT K1 CPU to be a good candidate, with a lot of upstreaming
> > > > > work happening. The Debian 7.0 backports kernel is now suitable to run a
> > > > > build daemon. Testing shows a performance improvement by a factor of 1.7
> > > > > to 2.2, depending on the level of parallelism. For example, the Linux
> > > > > kernel can be built in about 9 hours and rustc in about 14 hours. We
> > > > > also expect an additional 10% gain once cpufreq support is available,
> > > > > as the CPU is currently not running at its nominal speed. In addition
> > > > > support for vector instructions may further improve performance for some
> > > > > specific packages.
> > > > > 
> > > > > The other option that appeared recently is the SpacemiT K3. The
> > > > > upstreaming process is progressing quite fast, also benefiting from the
> > > > > existing K1 code. The Debian backports kernel is already able to boot on
> > > > > such a machine, but with a quite limited set of devices. Kernel 7.2 or
> > > > > 7.3 (the 7.2 merge window is not yet fully closed) should provide
> > > > > sufficient support for use as a build daemons, so likely around fall.
> > > > > Online benchmarks indicate we should expect to halve the build time
> > > > > compared to the K1, although we have not yet been able to validate this
> > > > > ourselves due to lack of hardware.
> > > > > 
> > > > > Regarding the timescale, we started the process of upgrading the build
> > > > > daemons with the Spacemit K1, beginning with MANDA. The first two boards
> > > > > arrived there last week and are waiting to be installed. The plan is to
> > > > > upgrade the two other boards once we are sure everything is working fine
> > > > > with the two first boards.
> > > > 
> > > > Any progress on those first boards? Have they been installed, and has
> > > > testing on them begun?
> > > 
> > > Unfortunately they haven't been installed yet. There is an option to
> > > send someone on site instead, but that cannot happen before around mid
> > > July.
> > 
> > Ack, thanks for the update.
> 
> rv-manda-02 and rv-manda-04 are now running on the new boards. A big 
> thanks to the MANDA local contact who did the work this evening.

Unfortunately, while they worked well for the first 3 weeks, we are now 
facing an issue [1], since glibc 2.43 is in sid. Right now rv-manda-02 
and rv-manda-04 run with the vector extension disabled to workaround the 
issue, and they seems to work fine that way. Nevertheless, until we 
better understand the issue, it doesn't seem a good idea to deploy more 
K1 boards.

Disabling the vector extension also bring us time to fix packages with 
runtime vector extension support, but with some issues that break the 
testsuites.

In parallel, we continue to follow upstream kernel support for the K3, 
with the hope we can get everything in 7.4. Currently the PCI controller 
[2] and the thermal management chip [3] are missing, and a few smaller 
fixes that will likely go through the stable trees anyway. The bad news 
is that it 7.4 is currently scheduled for January [4], so 7.3 will be 
the LTS version that will go into Forky. This means we'll need to work 
with the kernel team to backport the corresponding patches into 7.3.

Regards
Aurelien


[1] https://lore.kernel.org/spacemit/[email protected]/T/#u
[2] https://lore.kernel.org/spacemit/[email protected]/
[3] https://lore.kernel.org/spacemit/[email protected]/T/#t
[4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=809177dbf32e