[jira] [Commented] (JAMES-4159) EventBus: improve slow listener handling

    [ https://issues.apache.org/jira/browse/JAMES-4159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18050371#comment-18050371 ] 

Benoit Tellier commented on JAMES-4159:
---------------------------------------

Hello [~matthieu]

> If we know some listeners have different behaviors, why not making delivery system knows?
> For example, we could have a way to group listeners explicitly. We could group fast listeners but not slow ones.

I do believe the inner working of the EventBus is rather complex. I fear that by adding yet-another dimension in the way listeners behaves that we add more complexity than what is really needed. I am thus very reluctant to further rework the "EventBus" internals.

> Another question: the ticket is about EventBus and then it only focuses on RabbitMQ. Is it a RabbitMQ related problem?

Very true. I bet I did follow the path of least-implementation complexity and missed the big picture.

---

Thanks a lot for those questions. If we end up imtroduction complex constructs it is very likely that we (I) are trying to solve bad problems.

Another possible way to puzzle things together is to actually "make our listener fast". 
We could:
 - Decide quickly if our current email is worth further analysis
 - Then put that task onto another EventBus (which is configured with  a higher timeout and dedicated to that sole task)

This could easily be implemented in Twake Mail AI extension without impacting the project as a all.

We could still keep the concept of "QOS" and "execution time" for the event bus as it is likely to be beneficial.

> EventBus: improve slow listener handling
> ----------------------------------------
>
>                 Key: JAMES-4159
>                 URL: https://issues.apache.org/jira/browse/JAMES-4159
>             Project: James Server
>          Issue Type: Improvement
>          Components: eventbus
>            Reporter: Benoit Tellier
>            Priority: Major
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> h3. Context
> At LINAGORA we developped some custom listeners responsible of doing AI related tasks, either querying or feeding AI models. Mailbox listeners are a good fit because they are not blocking / delaying mail reception.
> The EventBus works the following way: it publishes a message onto RabbitMQ, and initially is consummed once by a James node that would execute all listeners. Using QOS we are consuming up to 10 messages simultaneously per node. Failed messages would then be re-published to individual per-listener retry queue.
> This "bundling" is done so that we limit the chatter of deserialization + wiring of the event bus, where each listener would have needed those operations, with separated execution. The amount of CPU time dedicated to those operations was significant ( > 40%).
> However this bundling means we are not efficient handling "slow" listeners which at scale can sinificantly distrurb and delay event treatment.
> h3. Proposal
> Be able to configure a timeout on the initial eventbus run, for each listener.
> That way, slow listeners would be individually retried.
> In `listeners.xml`:
> {code:xml}
> <listeners>
>   <executeGroupListeners>true</executeGroupListeners>
>   <!-- New block -->
>   <qos>20</qos>
>   <initialExecutionTimeout>30s</initialExecutionTimeout>
>   <executionTimeout>120s</executionTimeout>
>   <!-- -->
>   <listener>
>     <class>org.apache.james.mailbox.cassandra.MailboxOperationLoggingListener</class>
>   </listener>
> </listeners>
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)