Re: outgoing queue limit in bearbox

I already have an issue open at https://redmine.kannel.org/issues/674..
Any comments on the diff?

thanks,
Nick

On Thu, Dec 13, 2012 at 3:09 PM, spameden <[email protected]> wrote:

> I think this issue should be escalated to the bug tracker because it's
> quite serious.
>
> Could you post all details here -
> http://redmine.kannel.org/projects/kannel
>
> Queue on one of the binds shouldnt affect queue of other ones at all.
>
> We experienced something similar when there were problems with one of our
> binds.
>
>
>
> 2012/12/14 Nick Mahilani <[email protected]>
>
>> I had tried setting the sms-outgoing-queue-limit = 0 but I still saw
>> messages getting enqueued to the outgoing_sms queue. Also, in my load tests
>> with SMPP simulator binds connected to kannel, I saw a lot of full_found
>> queue full error with this setting set to 0.. Having a small buffer ~10
>> helped avoid most of the errors, but setting the limit to 10 caused retries
>> for all binds going to the global outgoing_sms queue and hit the
>> sum(#queues) limit condition in smsc2_rout(), which is what I wanted to
>> avoid.. Since one saturated bind can affect traffic on other binds..
>>
>> On Thu, Dec 13, 2012 at 12:00 PM, Rene Kluwen <[email protected]>wrote:
>>
>>> What if you just set the queue-limit to 0. Won’t that have the same
>>> effect?****
>>>
>>> ** **
>>>
>>> == Rene****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>> *From:* [email protected] [mailto:
>>> [email protected]] *On Behalf Of *Nick Mahilani
>>>
>>> *Sent:* donderdag 13 december 2012 19:04
>>> *To:* Amin Mukhaimer
>>> *Cc:* [email protected]
>>> *Subject:* Re: outgoing queue limit in bearbox****
>>>
>>>  ** **
>>>
>>> Thanks Amin !****
>>>
>>> While this approach works it is not very scalable if we have more
>>> operators in the future. As an alternative, I have defined a configurable
>>> in kannel config to bypass retries using the outgoing_sms queue. When
>>> turned on, this option disables queuing any messages to the 'outgoing_sms'
>>> queue and sends back a DLR NACK for any messages that fail for any
>>> reason(QFULL or submit_sm_resp with error from SMSC). This essentially
>>> moves the retry logic from kannel to the application using kannel and gives
>>> the application more control of the retry logic. This also avoids one
>>> saturated bind from affecting other binds.****
>>>
>>> I wanted to get some feedback and if you see any potential problems with
>>> this approach. I have attached a diff for this change. Please review.***
>>> *
>>>
>>> ** **
>>>
>>> -Nick****
>>>
>>> ** **
>>>
>>> Hi,
>>> I am using kannel with multiple SMSCs in my org and we often see that
>>> high load on one of the binds impacts the traffic on the other binds. After
>>> looking at the code, I see that the outgoing_sms queue is used as a global
>>> retry queue and if high load is directed to one of the binds, this fills up
>>> the outgoing_sms queue and causes 503 for the other sms traffic going
>>> through other binds..
>>> As a potential fix, I have defined a configurable in kannel config to
>>> bypass retries using the outgoing_sms queue. When turned on, this option
>>> disables queuing any messages to the 'outgoing_sms' queue and sends back a
>>> DLR NACK for any messages that fail due to QFULL conditions or other cases
>>> that would otherwise enqueue to the 'outgoing_sms' queue. This essentially
>>> moves the retry logic from kannel to the application using kannel..****
>>>
>>> I wanted to get some feedback on if you see any potential problems with
>>> this approach.
>>> I can forward a diff if someone could review the diff and provide
>>> feedback.****
>>>
>>> thanks,
>>> Nick****
>>>
>>> ** **
>>>
>>> On Wed, Nov 21, 2012 at 11:01 PM, Amin Mukhaimer <
>>> [email protected]> wrote:****
>>>
>>> Yes of course, just make different config files for each and run an
>>> instance for each, like****
>>>
>>>  ****
>>>
>>> bearerbox  operator1_config****
>>>
>>> smsbox        operator1_config****
>>>
>>>  ****
>>>
>>> bearerbox operator2_config****
>>>
>>> smsbox      operator2_config****
>>>
>>>  ****
>>>
>>> make sure they have different ports/spool directory/database tables/ log
>>> files… and good luck****
>>>
>>>  ****
>>>
>>> Amin****
>>>
>>>  ****
>>>
>>> *From:* Nick Mahilani [mailto:[email protected]]
>>> *Sent:* Wednesday, November 21, 2012 7:00 PM
>>> *To:* [email protected]
>>> *Cc:* [email protected]
>>> *Subject:* RE: outgoing queue limit in bearbox****
>>>
>>>  ****
>>>
>>> Thanks Amin for your suggestion...****
>>>
>>> Can we run multiple kannel instances on the same box?****
>>>
>>>
>>> Amin Mukhaimer <[email protected]> wrote:****
>>>
>>> I had a somewhat similar problem is that it tends to get slow when
>>> sending high amounts of SMSs to specific SMSC, and SMSs sent to other SMSCs
>>> are delayed for a while, so I decided to run multiple Kannels, one for each
>>> operator, hopping that would increase overall performance.****
>>>
>>>  ****
>>>
>>> I don’t know if that helps, but good luck.****
>>>
>>>  ****
>>>
>>> *From:* [email protected] [mailto:[email protected]<[email protected]>]
>>> *On Behalf Of *Nick Mahilani
>>> *Sent:* Wednesday, November 21, 2012 1:16 AM
>>> *To:* [email protected]
>>> *Subject:* outgoing queue limit in bearbox****
>>>
>>>  ****
>>>
>>> Hi,****
>>>
>>> I have an application which is using Kannel to send outgoing sms via
>>> multiple SMSC's. However, there is an insane volume of messages that needs
>>> to go through one of the SMSC which is impacting the other binds due to the
>>> global outgoing queue limit check in the code.****
>>>
>>>  ****
>>>
>>>  if (max_outgoing_sms_qlength <http://doxygen.kannel.org/d5/d27/bb__smscconn_8c.html#a8> > 0 && !resend <http://doxygen.kannel.org/d6/d9f/wtp__init_8h.html#a33a30> &&****
>>>
>>> 01091          queue_length > gwlist_len <http://doxygen.kannel.org/da/d23/list_8h.html#a6>(smsc_list) * max_outgoing_sms_qlength) {****
>>>
>>> 01092         gw_rwlock_unlock <http://doxygen.kannel.org/d2/dbd/gw-rwlock_8h.html#a4>(&smsc_list_lock);****
>>>
>>> 01093         debug <http://doxygen.kannel.org/d7/d7f/log_8h.html#a14>("bb.sms", 0, "sum(#queues) limit");****
>>>
>>> 01094         return SMSCCONN_FAILED_QFULL;****
>>>
>>> 01095     }****
>>>
>>>  I am very new to the kannel codebase so wanted to get some input on
>>> how complicated is the fix to isolate the queue limits to each bind/SMSC. I
>>> would like the other binds to not get affected by high volume of traffic on
>>> one of the binds. So if one smsc queue is backed up, it does not impact the
>>> messages sent to other SMSC's.****
>>>
>>> Also, what is recommended outgoing queue limit value in terms of
>>> messages per second being sent through Kannel?****
>>>
>>>  ****
>>>
>>> thanks,****
>>>
>>> Nick****
>>>
>>> ** **
>>>
>>
>>
>