Re: Diagnosing problem with submission of MTs from Kannel's store file...

Sorry, just noticed that the problem I originally described was slightly 
different - I have yet to confirm whether that original problem is still 
present with Kannel CVS, but this latest 'throughput' issue definitely is.

Giulio Harding wrote:
> Ok, compile Kannel from CVS (vs-20060509), and deployed it in production 
> after some basic testing - all seems well so far.
> 
> Details:
> 
> Kannel bearerbox version `cvs-20060509'. Build `May 30 2006 17:37:39', 
> compiler `3.2 20020903 (Red Hat Linux 8.0 3.2-7)'. System Linux, release 
> 2.4.20-19.8smp, version #1 SMP Tue Jul 15 15:01:43 EDT 2003, machine 
> i686. Hostname vesuvius, IP 10.100.123.135. Libxml version 2.6.9. Using 
> OpenSSL 0.9.6b [engine] 9 Jul 2001. Using native malloc.
> 
> However, the 'throughput' problem still exists - here is the 
> configuration for one of our binds:
> 
> # Vodafone SMSC. Free MT services
> # This SMPP account is for all new free MT services provisioned through 
> MPAPS
> group = smsc
> smsc = smpp
> smsc-id = vodafonefrmt
> host = 202.81.65.20
> port = 3403
> transceiver-mode = true
> smsc-username = xxx
> smsc-password = yyy
> system-type = VMA
> address-range = 0
> source-addr-ton = 3
> source-addr-npi = 9
> dest-addr-ton = 1
> dest-addr-npi = 1
> alt-charset="ASCII"
> enquire-link-interval = 30
> allowed-smsc-id = "vodafonefrmt"
> throughput = 13
> 
> Upon starting Kannel, I see this in the bearer.log:
> 
> 2006-05-31 23:41:42 [18565] [0] INFO: Set throughput to 13.000 for smsc 
> id <vodafonefrmt>
> 
> If I send a batch of 100 MTs through Kannel, to that SMSC, they are all 
> sent at once, which triggers the following response from the SMSC:
> 
> 2006-05-31 23:56:49 [18590] [10] ERROR: SMPP[vodafonefrmt]: SMSC 
> returned error code 0x00000058 (Throttling error) in response to submit_sm.
> 2006-05-31 23:56:49 [18590] [10] WARNING: SMPP: PDU NULL terminated 
> string has no NULL.
> 2006-05-31 23:56:49 [18590] [10] ERROR: SMPP[vodafonefrmt]: SMSC 
> returned error code 0x00000058 (Throttling error) in response to submit_sm.
> 2006-05-31 23:56:49 [18590] [10] WARNING: SMPP: PDU NULL terminated 
> string has no NULL.
> 2006-05-31 23:56:49 [18590] [10] ERROR: SMPP[vodafonefrmt]: SMSC 
> returned error code 0x00000058 (Throttling error) in response to submit_sm.
> 2006-05-31 23:56:49 [18590] [10] WARNING: SMPP: PDU NULL terminated 
> string has no NULL.
> 2006-05-31 23:56:49 [18590] [10] ERROR: SMPP[vodafonefrmt]: SMSC 
> returned error code 0x00000058 (Throttling error) in response to submit_sm.
> 2006-05-31 23:56:49 [18590] [10] WARNING: SMPP: PDU NULL terminated 
> string has no NULL.
> 2006-05-31 23:56:49 [18590] [10] ERROR: SMPP[vodafonefrmt]: SMSC 
> returned error code 0x00000058 (Throttling error) in response to submit_sm.
> 2006-05-31 23:56:49 [18590] [10] WARNING: SMPP: PDU NULL terminated 
> string has no NULL.
> 2006-05-31 23:56:49 [18590] [10] ERROR: SMPP[vodafonefrmt]: SMSC 
> returned error code 0x00000058 (Throttling error) in response to submit_sm.
> 2006-05-31 23:56:49 [18590] [10] WARNING: SMPP: PDU NULL terminated 
> string has no NULL.
> 2006-05-31 23:56:49 [18590] [10] ERROR: SMPP[vodafonefrmt]: SMSC 
> returned error code 0x00000058 (Throttling error) in response to submit_sm.
> 2006-05-31 23:56:49 [18590] [10] WARNING: SMPP: PDU NULL terminated 
> string has no NULL.
> 2006-05-31 23:56:49 [18590] [10] ERROR: SMPP[vodafonefrmt]: SMSC 
> returned error code 0x00000058 (Throttling error) in response to submit_sm.
> 2006-05-31 23:56:49 [18590] [10] WARNING: SMPP: PDU NULL terminated 
> string has no NULL.
> 2006-05-31 23:56:49 [18590] [10] ERROR: SMPP[vodafonefrmt]: SMSC 
> returned error code 0x00000058 (Throttling error) in response to submit_sm.
> 2006-05-31 23:56:49 [18590] [10] WARNING: SMPP: PDU NULL terminated 
> string has no NULL.
> 2006-05-31 23:56:49 [18590] [10] ERROR: SMPP[vodafonefrmt]: SMSC 
> returned error code 0x00000058 (Throttling error) in response to submit_sm.
> 2006-05-31 23:57:13 [18590] [10] WARNING: SMPP: PDU NULL terminated 
> string has no NULL.
> 2006-05-31 23:57:13 [18590] [10] ERROR: SMPP[vodafonefrmt]: SMSC 
> returned error code 0x00000058 (Throttling error) in response to submit_sm.
> 2006-05-31 23:57:13 [18590] [10] WARNING: SMPP: PDU NULL terminated 
> string has no NULL.
> 2006-05-31 23:57:14 [18590] [10] ERROR: SMPP[vodafonefrmt]: SMSC 
> returned error code 0x00000058 (Throttling error) in response to submit_sm.
> ...
> 
> etc. for the next 5 minutes or so. Each minute, Kannel tries to send the 
> all the queued MTs at once, and a small number get through before the 
> throttling condition is triggered, and the SMSC stops accepting MTs. 
> After about 5 minutes, all hundred MTs did eventually get through, but 
> this is hardly ideal - in cases of high load, it could take many many 
> hours for all the messages to trickle through.
> 
> So:
> 
> - the throughput parameter is not honoured for MTs submitted to Kannel 
> via HTTP
> - the throughput parameter is not honoured for MTs delivered from 
> Kannel's queue/store
> 
> Have I interpreted the situation correctly? Has anyone else encountered 
> this problem? Let me know if I should provide any other info...
> 
> One thing - Vincent's email, with subject '[REQUEST] EMI/UCP + 
> throttling: Need beta-testers' - this sounds like it might be 
> applicable, but I'm not 100% sure - what does EMI/UCP mean?
> 
> Thanks,
> 
> Alexander Malysh wrote:
>> Hi,
>>
>> I would advice you to try CVS head version. It contains numerous bug 
>> fixes all over the place. Please try it and report when the problem 
>> persist.
>>
>> Thanks,
>> Alex
>>
>>
>> Giulio Harding schrieb:
>>> Hi, this is my first post to the Kannel Devel mailing list (and I've 
>>> only been subscribed for maybe a month or so), so please excuse any 
>>> cluelessness on my part...
>>>
>>> We're becoming quite reliant on Kannel because of significant 
>>> problems with our old SMS gateway, and a new SMS gateway we're trying 
>>> to migrate to.
>>>
>>> As such, I have a fairly urgent need to diagnose and resolve an issue 
>>> that's cropped up a few times, with regards to Kannel's handling of 
>>> MTs that are queued to its message store file.
>>>
>>> The problem is this:
>>>
>>> An SMPP link to a carrier goes down, for one reason or another, and 
>>> MTs submitted to Kannel are queued in its store file. When the link 
>>> comes back up (responds to bind requests again), *one* MT is 
>>> submitted, but the rest of the queued MTs are *not* submitted, 
>>> despite the bearer.log indicating (for sets of 10 MTs at a time) 
>>> "WARNING: SMPP[optuschmt]: Not ACKED message found, will retransmit. 
>>> SENT<90>sec. ago, SEQ<110>, DST<61402276846>" etc. (I've checked this 
>>> with Etheral, monitoring traffic to the SMSC's IP address and port - 
>>> only one SMPP submit_sm exchange occurs)
>>>
>>> Also, all further MTs submitted to Kannel for that SMPP link will 
>>> queue, even though the link is up (and responding to bind enquiries).
>>>
>>> If I restart Kannel, again, *one* MT will be submitted, and that's 
>>> it, and MTs will continue to queue.
>>>
>>> The only way to resolve this, so far, is to stop Kannel, move its 
>>> store file (and backup) aside, and restart Kannel so it has an empty 
>>> store. Subsequent MTs are submitted fine, and don't queue.
>>>
>>> I then have to parse the original store file and manually resend 
>>> those MTs that were not submitted before.
>>>
>>> This is a huge problem when we are experiencing heavy traffic, and it 
>>> can sometimes take hours to resolve.
>>>
>>> This is with Kannel 1.4.0 running on Redhat 8.0 (let me know if I 
>>> should provide extra info, e.g. SMSC connection configuration, log 
>>> excerpts etc.)
>>>
>>> I am currently trying to replicate this problem on a test setup, but 
>>> I wanted to check with people on the mailing list for any advice. Has 
>>> anyone else encountered this problem? What are the chances of getting 
>>> some help to diagnose & fix it? I (and some colleagues) have passable 
>>> C skills, but any tips on where I should start to diagnose the 
>>> problem would be greatly appreciated.
>>>
>>> Thanks,
>>>
>>
>>
>>
> 
> 


-- 
Giulio Harding
Systems Administrator

m.Net Corporation
Level 13, 99 Gawler Place
Adelaide SA 5000, Australia

Tel: +61 8 8210 2041
Fax: +61 8 8211 9620
Mobile: 0432 876 733
MSN: [email protected]

http://www.mnetcorporation.com