Re: PANIC bearerbox cvs-20090902
"Nikos Balkanas" <[email protected]>
| Newsgroups | gmane.comp.mobile.kannel.devel |
|---|---|
| Message-ID | <000901ca3318$d57dbee0$02b2a8c0@tardis> |
Hi Alej, Of course. However, fs scales also very well either over NFS or SAN. Just to make things clearer for other people listening in: Kernel code is very efficient and fast but is inflexible and limited (it will just do what it is designed to). User space code (aka application) is slower, but offers maximum flexibility. Basically you tailor it to your taste. Bottom line: Use kernel code to do your job, wherever you can. If you need more, go to an application. @Dante: Don't use a DB as a substitute for a plain fs. I apologize for spamming this list, I am always interested in this stuff. BR, Nikos ----- Original Message ----- From: Alejandro Guerrieri To: Nikos Balkanas Cc: Dante Moreno ; [email protected] Sent: Friday, September 11, 2009 7:46 PM Subject: Re: PANIC bearerbox cvs-20090902 A DB could be run on another server(s) as well, scaling much easier than a local filesystem. At least in my experience, there's not a one-fits-all solution on IT problems. What works for some may not for others... Regards, -- Alejandro Guerrieri [email protected] On 11/09/2009, at 18:42, Nikos Balkanas wrote: Sorry, Forgot to mention is that a DB works with sockets, where an fs works with optimized system-calls. No comparison. BR, Nikos ----- Original Message ----- From: Nikos Balkanas To: Dante Moreno Cc: Alejandro Guerrieri ; [email protected] Sent: Friday, September 11, 2009 7:40 PM Subject: Re: PANIC bearerbox cvs-20090902 Hi, In my oppinion, a well-tuned fs is much faster than a DB (especially with transaction logs). That's because it is in the kernel and it is optimized to do just one thing: move around files. DBs are optimized to do searches with keywords inside text. Different application and in user-space. And eventually they have to write everything to disk (transaction logs). But to the problem at hand. You have a panic situation here, not performance. Could you switch to spool, to see if it is any better? Bear in mind that spool is much safer, it is real time. You may loose just the last sms. With a file storage you might loose up to 10" or more worth of sms. BR, Nikos ----- Original Message ----- From: Dante Moreno To: Nikos Balkanas Cc: Alejandro Guerrieri ; [email protected] Sent: Friday, September 11, 2009 6:57 PM Subject: Re: PANIC bearerbox cvs-20090902 I remember that Alejandro Guerreri was working on a DB-store solution a while ago. Would spool still be a better approach? Regards, Dante 2009/9/11 Nikos Balkanas <[email protected]> Yes, but still haven't tried Reiser! By all means stay away from zfs (solaris). Despite all the hype, it is much slower than plain ufs. Still spool is much more efficient than file. Imagine that file store has to lock the sms-list while it is writing the store-file from scratch. And with long Qs this could take some time. BR, Nikos ----- Original Message ----- From: Alejandro Guerrieri To: Dante Moreno Cc: Nikos Balkanas ; [email protected] Sent: Friday, September 11, 2009 4:43 PM Subject: Re: PANIC bearerbox cvs-20090902 Yes, I've benchmarked ext3, ext2 and xfs and ext2 is by far the best performing filesystem for spool store. At least on my experience, ext3 gets very sluggish on stores over 50K. Regarding xfs, despite being quite faster than ext3 in loading the store, increased the load on my system under heavy traffic. IMHO, ext2 is the way to go. However, if you're sustaining heavy traffic the spool store stresses the filesystem a lot so I'd recommend you to use a dedicated ext2 partition for the store: you'd still use the more reliable ext3 for the OS while getting the speed of ext2 where's needed. Furthermore, if the ext2 partition crashes you'd be able to unmount it and repair it without rebooting the box. Regards, -- Alejandro Guerrieri [email protected] On 11/09/2009, at 15:34, Dante Moreno wrote: Hi Nikos, I can't use spool store-type right now since kannel runs on ext3 filesystem. I remember reading that there were performance issues if you have a large queue+spool+ext3. If I have no other choice, I can partition the system and create an ext2 or xfs partition just for the queue. However I want to do that as a last resource solution(and hope that the problem really doesn't happen again). On the other hand, there are a couple of good free smpp smsc simulators. SMPPSim(free and open source) or Logica's simulator for example. Regards, Dante 2009/9/11 Nikos Balkanas <[email protected]> Hi, I cannot find anything wrong with the code at that point. However, it looks like memory corruption. Could you please use spool instead of file? It is safer, more efficient and faster than file. In addition it uses different memory structures than file and you should get away with it. I will update and run valgrind on it over the weekend. Unfortunately, I don't have smsc connections, but I hope I can catch the problem with fake smsc. If not, someone else from the list will have to look at it. BR, Nikos ----- Original Message ----- From: Nikos Balkanas To: Dante Moreno Cc: [email protected] Sent: Thursday, September 10, 2009 10:44 PM Subject: Re: PANIC bearerbox cvs-20090902 Hi, How can you say they are the same? Even the the problem is different this time. Anyway I 'll have to look at it. BR, Nikos ----- Original Message ----- From: Dante Moreno To: Nikos Balkanas Cc: [email protected] Sent: Thursday, September 10, 2009 8:12 PM Subject: Re: PANIC bearerbox cvs-20090902 Hi Nikos, Thanks for answering. First of all, i'm using the file store-type option and there is plenty of free disk space. I'm using the latest CVS(cvs-20090902). Here are the logs+addr2line output: 2009-09-10 09:10:40 [27966] [18] PANIC: gwlib/octstr.c:2484: seems_valid_real: Assertion `ostr != NULL' failed. (Called from gwlib/octstr.c:874:octstr_compare.) 2009-09-10 09:10:40 [27966] [18] PANIC: bearerbox(gw_panic+0x15b) [0x4833db] 2009-09-10 09:10:40 [27966] [18] PANIC: bearerbox [0x483c59] 2009-09-10 09:10:40 [27966] [18] PANIC: bearerbox(octstr_compare+0x20) [0x488800] 2009-09-10 09:10:40 [27966] [18] PANIC: bearerbox [0x477292] 2009-09-10 09:10:40 [27966] [18] PANIC: bearerbox(gwlist_search+0x54) [0x4811d4] 2009-09-10 09:10:40 [27966] [18] PANIC: bearerbox(dict_get+0x35) [0x4772d5] 2009-09-10 09:10:40 [27966] [18] PANIC: bearerbox [0x4175f0] 2009-09-10 09:10:40 [27966] [18] PANIC: bearerbox [0x4177ef] 2009-09-10 09:10:40 [27966] [18] PANIC: bearerbox [0x417df5] 2009-09-10 09:10:40 [27966] [18] PANIC: bearerbox [0x47a2f5] 2009-09-10 09:10:40 [27966] [18] PANIC: /lib64/libpthread.so.0 [0x3781e06307] 2009-09-10 09:10:40 [27966] [18] PANIC: /lib64/libc.so.6(clone+0x6d) [0x37812d1ded] and here the addr2line output: addr2line -e /gateway-1.4.3_cvs_20090902/gw/bearerbox 0x4833db 0x483c59 0x488800 0x477292 0x4811d4 0x4772d5 0x4175f0 0x4177ef 0x417df5 0x47a2f5 0x3781e06307 0x37812d1ded /gateway-1.4.3_cvs_20090902/gwlib/log.c:541 /gateway-1.4.3_cvs_20090902/gwlib/octstr.c:2483 /gateway-1.4.3_cvs_20090902/gwlib/octstr.c:875 /gateway-1.4.3_cvs_20090902/gwlib/dict.c:103 /gateway-1.4.3_cvs_20090902/gwlib/list.c:472 /gateway-1.4.3_cvs_20090902/gwlib/dict.c:298 /gateway-1.4.3_cvs_20090902/gw/bb_store_file.c:196 /gateway-1.4.3_cvs_20090902/gw/bb_store_file.c:571 /gateway-1.4.3_cvs_20090902/gw/bb_store_file.c:236 /gateway-1.4.3_cvs_20090902/gwlib/gwthread-pthread.c:135 ??:0 ??:0 The line numbers seem to be the same as before. Regards, Dante 2009/9/10 Nikos Balkanas <[email protected]> Hi, No, this is the right place for debugger info. First make sure that your partition is not getting full and kannel has space to write the Q. Seems you are using spool type for Q storage and it runs out of unique hash strings. But I cannot be sure, since your addr2line output is from an older CVS and reports wrong line numbers. Please update to latest CVS and repost. BR, Nikos ----- Original Message ----- From: Dante Moreno To: [email protected] Sent: Thursday, September 10, 2009 5:20 PM Subject: Re: PANIC bearerbox cvs-20090902 Maybe I should post this to the users list? We are now facing this problem on a daily basis. Any help would be greatly appreciated. Regards, Dante 2009/9/8 Dante Moreno <[email protected]> Hi, We are using the latest CVS and have found this PANIC bugs. This has happened to us 3 times in around two weeks. We are not able to reproduce them....the only thing we know is that it seems to happen when the store size is very large(100,000+ messages). We are using the "file" store type. Below are the bug reports: The first one is: 2009-08-14 12:29:12 [4472] [15] DEBUG: boxc_receiver: sms received 2009-08-14 12:29:13 [4472] [14] PANIC: gwlib/octstr.c:2505: seems_valid_real: Assertion `ostr->data[ostr->len] == '\0'' failed. (Called from gwlib/octstr.c:343:octstr_len.) 2009-08-14 12:29:13 [4472] [14] PANIC: bearerbox(gw_panic+0x15b) [0x4830db] 2009-08-14 12:29:13 [4472] [14] PANIC: bearerbox [0x4837a5] 2009-08-14 12:29:13 [4472] [14] PANIC: bearerbox(octstr_len+0x1f) [0x483aef] 2009-08-14 12:29:13 [4472] [14] PANIC: bearerbox(octstr_hash_key+0x2f) [0x483b8f] 2009-08-14 12:29:13 [4472] [14] PANIC: bearerbox [0x476e8c] 2009-08-14 12:29:13 [4472] [14] PANIC: bearerbox(dict_get+0x1c) [0x476fbc] 2009-08-14 12:29:13 [4472] [14] PANIC: bearerbox [0x4175b0] 2009-08-14 12:29:13 [4472] [14] PANIC: bearerbox [0x4177af] 2009-08-14 12:29:13 [4472] [14] PANIC: bearerbox [0x417db5] 2009-08-14 12:29:13 [4472] [14] PANIC: bearerbox [0x479ff5] 2009-08-14 12:29:13 [4472] [14] PANIC: /lib64/libpthread.so.0 [0x3781e06307] 2009-08-14 12:29:13 [4472] [14] PANIC: /lib64/libc.so.6(clone+0x6d) [0x37812d1ded] addr2line -e /gateway-1.4.3/gw/bearerbox 0x4830db 0x4837a5 0x483aef 0x483b8f 0x476e8c 0x476fbc 0x4175b0 0x4177af 0x417db5 0x479ff5 0x3781e06307 0x37812d1ded /gateway-1.4.3/gwlib/log.c:541 /gateway-1.4.3/gwlib/octstr.c:2507 /gateway-1.4.3/gwlib/octstr.c:344 /gateway-1.4.3/gwlib/octstr.c:2468 /gateway-1.4.3/gwlib/dict.c:139 /gateway-1.4.3/gwlib/dict.c:294 /gateway-1.4.3/gw/bb_store_file.c:196 /gateway-1.4.3/gw/bb_store_file.c:571 /gateway-1.4.3/gw/bb_store_file.c:236 /gateway-1.4.3/gwlib/gwthread-pthread.c:135 ??:0 ??:0 And the second one which happened today: 2009-09-08 09:56:01 [25766] [18] PANIC: gwlib/octstr.c:2484: seems_valid_real: Assertion `ostr != NULL' failed. (Called from gwlib/octstr.c:874:octstr_compare.) 2009-09-08 09:56:02 [25766] [21] DEBUG: boxc_receiver: sms received 2009-09-08 09:56:02 [25766] [18] PANIC: bearerbox(gw_panic+0x15b) [0x4833db] 2009-09-08 09:56:02 [25766] [18] PANIC: bearerbox [0x483c59] 2009-09-08 09:56:02 [25766] [18] PANIC: bearerbox(octstr_compare+0x20) [0x488800] 2009-09-08 09:56:02 [25766] [18] PANIC: bearerbox [0x477292] 2009-09-08 09:56:02 [25766] [18] PANIC: bearerbox(gwlist_search+0x54) [0x4811d4] 2009-09-08 09:56:02 [25766] [18] PANIC: bearerbox(dict_get+0x35) [0x4772d5] 2009-09-08 09:56:02 [25766] [18] PANIC: bearerbox [0x4175f0] 2009-09-08 09:56:02 [25766] [18] PANIC: bearerbox [0x4177ef] 2009-09-08 09:56:02 [25766] [18] PANIC: bearerbox [0x417df5] 2009-09-08 09:56:02 [25766] [18] PANIC: bearerbox [0x47a2f5] 2009-09-08 09:56:02 [25766] [18] PANIC: /lib64/libpthread.so.0 [0x3781e06307] 2009-09-08 09:56:02 [25766] [18] PANIC: /lib64/libc.so.6(clone+0x6d) [0x37812d1ded] addr2line -e gateway-1.4.3_cvs_20090902/gw/bearerbox 0x4833db 0x483c59 0x488800 0x477292 0x4811d4 0x4772d5 0x4175f0 0x4177ef 0x417df5 0x47a2f5 0x3781e06307 0x37812d1ded gateway-1.4.3_cvs_20090902/gwlib/log.c:541 gateway-1.4.3_cvs_20090902/gwlib/octstr.c:2483 gateway-1.4.3_cvs_20090902/gwlib/octstr.c:875 gateway-1.4.3_cvs_20090902/gwlib/dict.c:103 gateway-1.4.3_cvs_20090902/gwlib/list.c:472 gateway-1.4.3_cvs_20090902/gwlib/dict.c:298 gateway-1.4.3_cvs_20090902/gw/bb_store_file.c:196 gateway-1.4.3_cvs_20090902/gw/bb_store_file.c:571 gateway-1.4.3_cvs_20090902/gw/bb_store_file.c:236 gateway-1.4.3_cvs_20090902/gwlib/gwthread-pthread.c:135 ??:0 ??:0 Also, for some strange reason, after the PANIC bearerbox restarts itself(parachute) but smsbox doesn't. Could anybody please hint me in how to solve this issues? Regards, Dante