Re: KernelCI Weekly Newsletter (Week 50)

Hello,

Thanks Denys for the comprehensive weekly news.

On 11/12/2024 10:53 pm, Denys Fedoryshchenko wrote:
> - **New Pub/Sub Subsystem Design**
>    Work is underway on a new Pub/Sub subsystem to address fundamental
> issues with the current implementation, including non-guaranteed
> message delivery and the lack of load-balancing features.
>    - The new subsystem will use Redis Streams as its foundation.
>    - It will be backward-compatible, requiring no API changes, enabling
> seamless migration for clients.

This looks similar to the Task concept which was introduced in Renelick 
after the initial fork, see some WIP documentation here:

https://gtucker.gitlab.io/renelick/tasks/

Some polishing is still needed but the main concept has now been 
stabilised.  Basically they rely on message queues which guarantee each 
message to be delivered to exactly one consumer.  Any API client can 
schedule a task, typically this is done by an orchestrator service, then 
the task object will need to be accepted by a single compatible 
scheduler which will be responsible for executing it.  If you have 
multiple instances of the same scheduler, it balances the load between 
them by design.  It also makes it possible to implement retries and 
other features related to task management.

By contrast, Pub/Sub events aren't guaranteed to be delivered since 
they're not destined to any particular recipient.  They enable a 
decoupled, modular architecture where you can add and remove subscribers 
without changing anything in the existing system.

Let me know if you want to discuss it further, also I'll be covering 
this in an upcoming blog post.

Guillaume