[jira] [Commented] (JAMES-4192) ActiveMQ: delayed mails block the queue while dequeuing

    [ https://issues.apache.org/jira/browse/JAMES-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18071758#comment-18071758 ] 

Benoit Tellier commented on JAMES-4192:
---------------------------------------

Thanks for the great write up [~ouvtam]... I realize I am mostly naive on ActiveMQ topics so thanks for digging this deeper.

From the elements presented, given the importance of implemented correctly delays, I think choosing an embedded broker that naturally support this is a sound choice!

Way better then cheating with a double queue and in app scheduler stuff... 

Yes delivering a Artemis based Mail Queue will be a handover but I believe we have a decent enough test suite to catch most of the functional brakdown and ensure conformity. Tools like smtpsource can easily emulate traffic of various size and allow to bench things a bit.

Artemis sounds like the less bad option to me.

My recommandation is to start working on it in a new server/queue/activemq-artemis folder and allow conditionally it in the guice assemble, opt in first, then default choice, then removing historical rabbitmq code. 

> ActiveMQ: delayed mails block the queue while dequeuing
> -------------------------------------------------------
>
>                 Key: JAMES-4192
>                 URL: https://issues.apache.org/jira/browse/JAMES-4192
>             Project: James Server
>          Issue Type: Bug
>          Components: Queue
>    Affects Versions: 3.7.0, 3.8.0, 3.7.1, 3.7.3, 3.7.4, 3.8.1, 3.7.5, 3.8.2, 3.8.3
>            Reporter: ouvtam
>            Priority: Major
>
> We operate James and observed issues when encountering a lot of temporary failures on remote delivery. It seemed James could not dequeue the outgoing queue.
> *Bug*
> Delayed mails were enqueued into the regular ActiveMQ queue with {{{}NEXT_DELIVERY_TIME{}}}.
> Because of ActiveMQ queue paging ({{{}maxPageSize defaults to 200{}}}), delayed messages at the head of the queue could block ready messages behind them, preventing non-delayed mails from being dequeued.
> *Steps to reproduce*
>  * Configure RemoteDelivery with <maxRetries>1</maxRetries> and a high delay time (e.g. <delayTime>60 minutes</delayTime>)
>  * Send at least 200 mails to a remote server that causes temporary exceptions (e.g. 421 greylist). In our case we configured a <gateway> that simulated such behaviour (i.e. always sending a temporary exception 421)
>  * Sending the 201 mail will never be dequeued (i.e. returns always null), because the 200 prior mails are sitting in the queue (ActiveMQ's maxPageSize=200). After 60 minutes of delay time the queue will processed eventually.
> *Fix*
> Delayed mails are no longer kept in the regular JMS queue before they are due.
> They are stored in the ActiveMQ Scheduler and only appear in the regular queue once their delay expires.
> *Result*
> Non-delayed mails are no longer blocked by delayed mails.
> *Implementation Details*
> I will provide a PR that takes advantage of the ActiveMQ's Scheduler Support. We already tested this in our local setup.
> Mails with the delayed JMS property set (i.e. org.apache.activemq.ScheduledMessage.AMQ_SCHEDULED_DELAY) will be intercepted by ActiveMQ's Scheduler and not added to the queue (e.g. outgoing). Instead, the scheduler stores the messages in the scheduler store. After the delay has passed the scheduler sends the messages into the queue for the consumer.
> *Alternative*
> Another approach would be to increase maxPageSize (e.g. using Per Destination Policies). This would increase the memory footprint of ActiveMQ, because it would hold the amount of maxPageSize messages in memory. For delayed messages this would not be very efficient, because with high delays they just sit in memory. In addition, the problem could occur nonetheless (e.g. lot of deliveries)
> *Questions*
> Using ** the scheduler introduces an important question:
>  * For instance, if 100 mails are delayed, the ActiveMQCacheableMailQueue#getSize would return 0 if no mails are in the queue. The reason is the delayed mails sit in the scheduler store. What is the contract for ManageableMailQueue#getSize? Should it count the total mails with the delayed messages? Or only the ones that have no delay? Since we make use of WebAdmin this would be changing behaviour.
>  ** Unfortunately, we cannot count the size of the scheduled messages using the same approach as in normal queues (i.e. using Statistics Plugin).
>  *** Either we use a special JMS browse command to browse the messages in the scheduler store while filtering the scheduled messages by queue (i.e. only scheduled messages for the queue "outgoing"). 
>  *** Or we activate JMX on the Broker to access the ScheduledMessages directly and iterate through all jobs, unmarshal the payload, and filter the scheduler jobs by queue (i.e. only scheduled messages for the queue "outgoing").
>  *** Using WebAdmin it would be great to pass params such as /mailQueues/outgoing?delayed 
>  ** An alternative would be to create two manageable queues by ActiveMQMailQueueFactory#createCacheableMailQueue (e.g. "outgoing" and "outgoing-delayed") that are manageable through "/mailQueues/outgoing" and "/mailQueues/outgoing-delayed"
>  
> Happy for feedback, so I can create the PR in the next few days.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)