[opennms-devel] Change default connection pool size to less than 100? Was: [opennms-discuss] 1.3.11 Hangs after hours of running

DJ Gregor <[email protected]>
Newsgroups gmane.network.opennms.bugs
Message-ID <[email protected]> 
Should we change the default connection pool size to something other  
than 100?

That happens to be _exactly_ the postgresql default, which seems  
particularly troubling (oh, you wanted to run 'psql' while opennms is  
running?  Sorry.).  Also, 100 concurrent connections just seems  
absolutely huge for all but the largest sites.


Begin forwarded message:

> From: Miakel Roberts <[email protected]>
> Date: February 28, 2008 8:58:34 PM EST
> To: <[email protected]>
> Subject: Re: [opennms-discuss] 1.3.11 Hangs after hours of running
> Reply-To: General OpenNMS Discussion <[email protected] 
> >
>

> after much work p-brane suggested I change pool size in the  
> c3p0.properties file from 50 to 100 at the 3 places it's defined.   
> Upon a OpenNMS restart I have been running for almost 24 hours  
> without a lock up !!
>
> Thanks for all the assistance !!
>
>
> Mike
>
>
>
> > From: [email protected]
> > To: [email protected]
> > Subject: RE: 1.3.11 Hangs after hours of running
> > Date: Wed, 27 Feb 2008 11:34:45 -0600
> >
> >
> > per request
> >
> > http://pastebin.ca/920833 - c3p0 properties
> >
> > http://pastebin.ca/920830 - postgresql.conf
> >
> >
> > ----------------------------------------
> > > From: [email protected]
> > > To: [email protected]
> > > Subject: RE: 1.3.11 Hangs after hours of running
> > > Date: Wed, 27 Feb 2008 09:24:43 -0600
> > >
> > >
> > > more info on the lock and activity and process
> > >
> > > [noc@canis etc]$ ps aux | grep 21915
> > > postgres 21915 0.0 0.0 423852 3324 ? S 09:22 0:00 postgres:  
> opennms opennms [local] idle
> > > noc 22100 0.0 0.0 3884 688 pts/2 S+ 09:23 0:00 grep 21915
> > > [noc@canis etc]$
> > >
> > > http://pastebin.ca/920662 - lock and activity
> > >
> > >
> > >
> > > ----------------------------------------
> > >> From: [email protected]
> > >> To: [email protected]
> > >> Subject: RE: 1.3.11 Hangs after hours of running
> > >> Date: Wed, 27 Feb 2008 08:59:28 -0600
> > >>
> > >>
> > >> 2 stack trace dumps taken while opennms is locked taken about  
> 20 minutes apart.
> > >>
> > >> http://pastebin.ca/920602
> > >>
> > >> I also checked pg_locks and pg_activity and results showed the  
> following
> > >> http://pastebin.ca/920610
> > >>
> > >> PID 7314 is postgres
> > >>
> > >>
> > >> ----------------------------------------
> > >>> From: [email protected]
> > >>> To: [email protected]
> > >>> Subject: RE: 1.3.11 Hangs after hours of running
> > >>> Date: Tue, 26 Feb 2008 12:17:33 -0600
> > >>>
> > >>>
> > >>> http://pastebin.ca/919237 - latest stack trace while the  
> previously post content was captured.
> > >>> ----------------------------------------
> > >>>> From: [email protected]
> > >>>> To: [email protected]
> > >>>> Subject: RE: 1.3.11 Hangs after hours of running
> > >>>> Date: Tue, 26 Feb 2008 11:56:52 -0600
> > >>>>
> > >>>>
> > >>>> I tried to post this to the list last evening but for some  
> reason it did not go.
> > >>>>
> > >>>> http://pastebin.ca/918151 - capsd while system is not  
> responding via webgui
> > >>>> http://pastebin.ca/918148 - poller.log while system is not  
> responding
> > >>>> http://pastebin.ca/918133 - pg_stat_Activity while system is  
> not responding
> > >>>> http://pastebin.ca/918134 - top while system is not responding
> > >>>> http://pastebin.ca/918132 - pg_locks while system is not  
> responding
> > >>>> http://pastebin.ca/918129 - vacuum.log while system is not  
> responding
> > >>>> http://pastebin.ca/917941 - some output from the postreslog,  
> "looks scary": joed
> > >>>>
> > >>>>
> > >>>> ----------------------------------------
> > >>>>> From: [email protected]
> > >>>>> To: [email protected]
> > >>>>> Subject: RE: 1.3.11 Hangs after hours of running
> > >>>>> Date: Mon, 25 Feb 2008 12:01:53 -0600
> > >>>>>
> > >>>>>
> > >>>>> I have been doing more digging and have come accross this in  
> the postgresql-Mon.log
> > >>>>>
> > >>>>> LOG: autovacuum: processing database "template1"
> > >>>>> LOG: autovacuum: processing database "postgres"
> > >>>>> LOG: autovacuum: processing database "test"
> > >>>>> ERROR: duplicate key violates unique constraint  
> "alarm_reductionkey_idx"
> > >>>>> ERROR: current transaction is aborted, commands ignored  
> until end of transaction block
> > >>>>> LOG: autovacuum: processing database "opennms"
> > >>>>> LOG: autovacuum: processing database "template1"
> > >>>>> LOG: autovacuum: processing database "postgres"
> > >>>>> LOG: autovacuum: processing database "test"
> > >>>>> LOG: autovacuum: processing database "opennms"
> > >>>>> LOG: autovacuum: processing database "template1"
> > >>>>> LOG: autovacuum: processing database "postgres"
> > >>>>> LOG: autovacuum: processing database "test"
> > >>>>> ERROR: duplicate key violates unique constraint  
> "alarm_reductionkey_idx"
> > >>>>> ERROR: current transaction is aborted, commands ignored  
> until end of transaction block
> > >>>>> LOG: autovacuum: processing database "opennms"
> > >>>>> LOG: autovacuum: processing database "template1"
> > >>>>> LOG: autovacuum: processing database "postgres"
> > >>>>>
> > >>>>>
> > >>>>> Mike
> > >>>>> ________________________________
> > >>>>>> From: [email protected]
> > >>>>>> To: [email protected]
> > >>>>>> Subject: RE: 1.3.11 Hangs after hours of running
> > >>>>>> Date: Mon, 25 Feb 2008 09:40:05 -0600
> > >>>>>>
> > >>>>>> This is the FULL log.
> > >>>>>>
> > >>>>>> http://pastebin.ca/917637
> > >>>>>>
> > >>>>>> sorry about the last 2 logs.
> > >>>>>>
> > >>>>>> Mike
> > >>>>>>
> > >>>>>>> From: [email protected]
> > >>>>>>> To: [email protected]
> > >>>>>>> Subject: RE: 1.3.11 Hangs after hours of running
> > >>>>>>> Date: Mon, 25 Feb 2008 09:19:37 -0600
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> I am running in full debug.
> > >>>>>>> David you are correct, that is not a full dump, I think I  
> must have lost some of the dump in the copy buffer.
> > >>>>>>> I'll try again.
> > >>>>>>>
> > >>>>>>> Here is the complete output.log from start of the opennms  
> (since last hang) to the next start after it hung.
> > >>>>>>> http://opennms.pastebin.org/21192
> > >>>>>>>
> > >>>>>>> Mike
> > >>>>>>> ________________________________
> > >>>>>>>> From: [email protected]
> > >>>>>>>> To: [email protected]
> > >>>>>>>> Subject: 1.3.11 Hangs after hours of running
> > >>>>>>>> Date: Mon, 25 Feb 2008 08:04:09 -0600
> > >>>>>>>>
> > >>>>>>>> Guys,
> > >>>>>>>> I have been having issues with 1.3.11 (don't know if it's  
> version related or not) ever since I went from 1.3.9 to 1.3.11.
> > >>>>>>>>
> > >>>>>>>> Symptoms are, after maybe 8 hours of running no one can  
> log in via the web interface, I can see the user / pass page but as  
> soon as I click login it just sits and hangs.
> > >>>>>>>> The system also at some point stops data collection /  
> polling after this lockup.
> > >>>>>>>> Restarting opennms fixes it for another 8 or so hours.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> I've tried numerous things, probably not all that I'll  
> remember here,
> > >>>>>>>> I've done a dumpall dropdb and restore. no dif
> > >>>>>>>> reinstall using bin/install. no diff
> > >>>>>>>>
> > >>>>>>>> I've done a kill -3 as per dj directions and will include  
> pastebins below.
> > >>>>>>>>
> > >>>>>>>> http://opennms.pastebin.org/21159 - full dump while  
> system not responding
> > >>>>>>>>
> > >>>>>>>> http://opennms.pastebin.org/21157 - pg_stat_activity &  
> pg_locks while system not responding
> > >>>>>>>>
> > >>>>>>>> http://opennms.pastebin.org/21158 - top while system is  
> not responding
> > >>>>>>>>
> > >>>>>>>> http://opennms.pastebin.org/21183 - iostat while system  
> is running fine
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Server stats are listed below
> > >>>>>>>>
> > >>>>>>>> 2 x 2.33ghz QuadCore
> > >>>>>>>> 16gb ram
> > >>>>>>>> 8 x ultra 320 SAS
> > >>>>>>>> raid1 - os and (just moved here from the raid10 to see if  
> this was the cause, it's not) the db
> > >>>>>>>> raid10 - rrd files
> > >>>>>>>> Centos 5.0 x86
> > >>>>>>>>
> > >>>>>>>> Opennms nodes
> > >>>>>>>> ~ 6000
> > >>>>>>>> ~ 1500 interfaces being polled for snmp data
> > >>>>>>>> ~ 900,000 event table
> > >>>>>>>>
> > >>>>>>>> Any assistance would be greatly appreciated !!
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Mike
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> ________________________________
> > >>>>>>>> Need to know the score, the latest news, or you need your  
> Hotmail®-get your "fix". Check it out.
> > >>>>>>>
> > >>>>>>>  
> _________________________________________________________________
> > >>>>>>> Connect and share in new ways with Windows Live.
> > >>>>>>> http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008
> > >>>>>>
> > >>>>>> ________________________________
> > >>>>>> Need to know the score, the latest news, or you need your  
> Hotmail®-get your "fix". Check it out.
> > >>>>>
> > >>>>>  
> _________________________________________________________________
> > >>>>> Connect and share in new ways with Windows Live.
> > >>>>> http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008
> > >>>>
> > >>>>  
> _________________________________________________________________
> > >>>> Need to know the score, the latest news, or you need your  
> Hotmail®-get your "fix".
> > >>>> http://www.msnmobilefix.com/Default.aspx
> > >>>
> > >>>  
> _________________________________________________________________
> > >>> Climb to the top of the charts! Play the word scramble  
> challenge with star power.
> > >>> http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan
> > >>
> > >> _________________________________________________________________
> > >> Need to know the score, the latest news, or you need your  
> Hotmail®-get your "fix".
> > >> http://www.msnmobilefix.com/Default.aspx
> > >
> > > _________________________________________________________________
> > > Need to know the score, the latest news, or you need your Hotmai 
> l®-get your "fix".
> > > http://www.msnmobilefix.com/Default.aspx
> >
> > _________________________________________________________________
> > Shed those extra pounds with MSN and The Biggest Loser!
> > http://biggestloser.msn.com/
>
> Shed those extra pounds with MSN and The Biggest Loser! Learn more.
> --- 
> ----------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-discuss mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom  
> of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-discuss

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-devel mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-devel
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.