Re: [opennms-devel] Change default connection pool size to less than 100? Was: [opennms-discuss] 1.3.11 Hangs after hours of running

50-75 seems to have worked quite well for me....
I usually turn this down, on some OS'es Like FBSD it
is actually by default way lower..

On Mon, Mar 3, 2008 at 3:58 AM, DJ Gregor <[email protected]> wrote:

> Should we change the default connection pool size to something other than
> 100?
>
> That happens to be _exactly_ the postgresql default, which seems
> particularly troubling (oh, you wanted to run 'psql' while opennms is
> running?  Sorry.).  Also, 100 concurrent connections just seems absolutely
> huge for all but the largest sites.
>
>
> Begin forwarded message:
>
> *From:* Miakel Roberts <[email protected]>
> *Date:* February 28, 2008 8:58:34 PM EST
> *To:* <[email protected]>
> *Subject:* *Re: [opennms-discuss] 1.3.11 Hangs after hours of running*
> *Reply-To:* General OpenNMS Discussion <
> [email protected]>
>
> after much work p-brane suggested I change pool size in the
> c3p0.properties file from 50 to 100 at the 3 places it's defined.  Upon a
> OpenNMS restart I have been running for almost 24 hours without a lock up !!
>
> Thanks for all the assistance !!
>
>
> Mike
>
>
>
> > From: <[email protected]>[email protected]
> > To: <[email protected]>
> [email protected]
> > Subject: RE: 1.3.11 Hangs after hours of running
> > Date: Wed, 27 Feb 2008 11:34:45 -0600
> >
> >
> > per request
> >
> > http://pastebin.ca/920833 - c3p0 properties
> >
> > http://pastebin.ca/920830 - postgresql.conf
> >
> >
> > ----------------------------------------
> > > From: <[email protected]>[email protected]
> > > To: <[email protected]>
> [email protected]
> > > Subject: RE: 1.3.11 Hangs after hours of running
> > > Date: Wed, 27 Feb 2008 09:24:43 -0600
> > >
> > >
> > > more info on the lock and activity and process
> > >
> > > [noc@canis etc]$ ps aux | grep 21915
> > > postgres 21915 0.0 0.0 423852 3324 ? S 09:22 0:00 postgres: opennms
> opennms [local] idle
> > > noc 22100 0.0 0.0 3884 688 pts/2 S+ 09:23 0:00 grep 21915
> > > [noc@canis etc]$
> > >
> > > http://pastebin.ca/920662 - lock and activity
> > >
> > >
> > >
> > > ----------------------------------------
> > >> From: <[email protected]>[email protected]
> > >> To: <[email protected]>
> [email protected]
> > >> Subject: RE: 1.3.11 Hangs after hours of running
> > >> Date: Wed, 27 Feb 2008 08:59:28 -0600
> > >>
> > >>
> > >> 2 stack trace dumps taken while opennms is locked taken about 20
> minutes apart.
> > >>
> > >> <http://pastebin.ca/920602>http://pastebin.ca/920602
> > >>
> > >> I also checked pg_locks and pg_activity and results showed the
> following
> > >> <http://pastebin.ca/920610>http://pastebin.ca/920610
> > >>
> > >> PID 7314 is postgres
> > >>
> > >>
> > >> ----------------------------------------
> > >>> From: <[email protected]>[email protected]
> > >>> To: <[email protected]>
> [email protected]
> > >>> Subject: RE: 1.3.11 Hangs after hours of running
> > >>> Date: Tue, 26 Feb 2008 12:17:33 -0600
> > >>>
> > >>>
> > >>> http://pastebin.ca/919237 - latest stack trace while the previously
> post content was captured.
> > >>> ----------------------------------------
> > >>>> From: <[email protected]>[email protected]
> > >>>> To: <[email protected]>
> [email protected]
> > >>>> Subject: RE: 1.3.11 Hangs after hours of running
> > >>>> Date: Tue, 26 Feb 2008 11:56:52 -0600
> > >>>>
> > >>>>
> > >>>> I tried to post this to the list last evening but for some reason
> it did not go.
> > >>>>
> > >>>> http://pastebin.ca/918151 - capsd while system is not responding
> via webgui
> > >>>> http://pastebin.ca/918148 - poller.log while system is not
> responding
> > >>>> http://pastebin.ca/918133 - pg_stat_Activity while system is not
> responding
> > >>>> http://pastebin.ca/918134 - top while system is not responding
> > >>>> http://pastebin.ca/918132 - pg_locks while system is not responding
>
> > >>>> http://pastebin.ca/918129 - vacuum.log while system is not
> responding
> > >>>> http://pastebin.ca/917941 - some output from the postreslog, "looks
> scary": joed
> > >>>>
> > >>>>
> > >>>> ----------------------------------------
> > >>>>> From: <[email protected]>[email protected]
> > >>>>> To: <[email protected]>
> [email protected]
> > >>>>> Subject: RE: 1.3.11 Hangs after hours of running
> > >>>>> Date: Mon, 25 Feb 2008 12:01:53 -0600
> > >>>>>
> > >>>>>
> > >>>>> I have been doing more digging and have come accross this in the
> postgresql-Mon.log
> > >>>>>
> > >>>>> LOG: autovacuum: processing database "template1"
> > >>>>> LOG: autovacuum: processing database "postgres"
> > >>>>> LOG: autovacuum: processing database "test"
> > >>>>> ERROR: duplicate key violates unique constraint
> "alarm_reductionkey_idx"
> > >>>>> ERROR: current transaction is aborted, commands ignored until end
> of transaction block
> > >>>>> LOG: autovacuum: processing database "opennms"
> > >>>>> LOG: autovacuum: processing database "template1"
> > >>>>> LOG: autovacuum: processing database "postgres"
> > >>>>> LOG: autovacuum: processing database "test"
> > >>>>> LOG: autovacuum: processing database "opennms"
> > >>>>> LOG: autovacuum: processing database "template1"
> > >>>>> LOG: autovacuum: processing database "postgres"
> > >>>>> LOG: autovacuum: processing database "test"
> > >>>>> ERROR: duplicate key violates unique constraint
> "alarm_reductionkey_idx"
> > >>>>> ERROR: current transaction is aborted, commands ignored until end
> of transaction block
> > >>>>> LOG: autovacuum: processing database "opennms"
> > >>>>> LOG: autovacuum: processing database "template1"
> > >>>>> LOG: autovacuum: processing database "postgres"
> > >>>>>
> > >>>>>
> > >>>>> Mike
> > >>>>> ________________________________
> > >>>>>> From: <[email protected]>[email protected]
> > >>>>>> To: <[email protected]>
> [email protected]
> > >>>>>> Subject: RE: 1.3.11 Hangs after hours of running
> > >>>>>> Date: Mon, 25 Feb 2008 09:40:05 -0600
> > >>>>>>
> > >>>>>> This is the FULL log.
> > >>>>>>
> > >>>>>> <http://pastebin.ca/917637>http://pastebin.ca/917637
> > >>>>>>
> > >>>>>> sorry about the last 2 logs.
> > >>>>>>
> > >>>>>> Mike
> > >>>>>>
> > >>>>>>> From: <[email protected]>[email protected]
> > >>>>>>> To: <[email protected]>
> [email protected]
> > >>>>>>> Subject: RE: 1.3.11 Hangs after hours of running
> > >>>>>>> Date: Mon, 25 Feb 2008 09:19:37 -0600
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> I am running in full debug.
> > >>>>>>> David you are correct, that is not a full dump, I think I must
> have lost some of the dump in the copy buffer.
> > >>>>>>> I'll try again.
> > >>>>>>>
> > >>>>>>> Here is the complete output.log from start of the opennms (since
> last hang) to the next start after it hung.
> > >>>>>>> <http://opennms.pastebin.org/21192>
> http://opennms.pastebin.org/21192
> > >>>>>>>
> > >>>>>>> Mike
> > >>>>>>> ________________________________
> > >>>>>>>> From: <[email protected]>[email protected]
> > >>>>>>>> To: <[email protected]>
> [email protected]
> > >>>>>>>> Subject: 1.3.11 Hangs after hours of running
> > >>>>>>>> Date: Mon, 25 Feb 2008 08:04:09 -0600
> > >>>>>>>>
> > >>>>>>>> Guys,
> > >>>>>>>> I have been having issues with 1.3.11 (don't know if it's
> version related or not) ever since I went from 1.3.9 to 1.3.11.
> > >>>>>>>>
> > >>>>>>>> Symptoms are, after maybe 8 hours of running no one can log in
> via the web interface, I can see the user / pass page but as soon as I click
> login it just sits and hangs.
> > >>>>>>>> The system also at some point stops data collection / polling
> after this lockup.
> > >>>>>>>> Restarting opennms fixes it for another 8 or so hours.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> I've tried numerous things, probably not all that I'll remember
> here,
> > >>>>>>>> I've done a dumpall dropdb and restore. no dif
> > >>>>>>>> reinstall using bin/install. no diff
> > >>>>>>>>
> > >>>>>>>> I've done a kill -3 as per dj directions and will include
> pastebins below.
> > >>>>>>>>
> > >>>>>>>> http://opennms.pastebin.org/21159 - full dump while system not
> responding
> > >>>>>>>>
> > >>>>>>>> http://opennms.pastebin.org/21157 - pg_stat_activity & pg_locks
> while system not responding
> > >>>>>>>>
> > >>>>>>>> http://opennms.pastebin.org/21158 - top while system is not
> responding
> > >>>>>>>>
> > >>>>>>>> http://opennms.pastebin.org/21183 - iostat while system is
> running fine
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Server stats are listed below
> > >>>>>>>>
> > >>>>>>>> 2 x 2.33ghz QuadCore
> > >>>>>>>> 16gb ram
> > >>>>>>>> 8 x ultra 320 SAS
> > >>>>>>>> raid1 - os and (just moved here from the raid10 to see if this
> was the cause, it's not) the db
> > >>>>>>>> raid10 - rrd files
> > >>>>>>>> Centos 5.0 x86
> > >>>>>>>>
> > >>>>>>>> Opennms nodes
> > >>>>>>>> ~ 6000
> > >>>>>>>> ~ 1500 interfaces being polled for snmp data
> > >>>>>>>> ~ 900,000 event table
> > >>>>>>>>
> > >>>>>>>> Any assistance would be greatly appreciated !!
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Mike
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> ________________________________
> > >>>>>>>> Need to know the score, the latest news, or you need your
> Hotmail(R)-get your "fix". Check it out.
> > >>>>>>>
> > >>>>>>>
> _________________________________________________________________
> > >>>>>>> Connect and share in new ways with Windows Live.
> > >>>>>>>
> <http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008>
> http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008
> > >>>>>>
> > >>>>>> ________________________________
> > >>>>>> Need to know the score, the latest news, or you need your
> Hotmail(R)-get your "fix". Check it out.
> > >>>>>
> > >>>>> _________________________________________________________________
> > >>>>> Connect and share in new ways with Windows Live.
> > >>>>>
> <http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008>
> http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008
> > >>>>
> > >>>> _________________________________________________________________
> > >>>> Need to know the score, the latest news, or you need your
> Hotmail(R)-get your "fix".
> > >>>> <http://www.msnmobilefix.com/Default.aspx>
> http://www.msnmobilefix.com/Default.aspx
> > >>>
> > >>> _________________________________________________________________
> > >>> Climb to the top of the charts! Play the word scramble challenge
> with star power.
> > >>>
> <http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan>
> http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan
> > >>
> > >> _________________________________________________________________
> > >> Need to know the score, the latest news, or you need your
> Hotmail(R)-get your "fix".
> > >> <http://www.msnmobilefix.com/Default.aspx>
> http://www.msnmobilefix.com/Default.aspx
> > >
> > > _________________________________________________________________
> > > Need to know the score, the latest news, or you need your Hotmail(R)-get
> your "fix".
> > > <http://www.msnmobilefix.com/Default.aspx>
> http://www.msnmobilefix.com/Default.aspx
> >
> > _________________________________________________________________
> > Shed those extra pounds with MSN and The Biggest Loser!
> > <http://biggestloser.msn.com/>http://biggestloser.msn.com/
>
> ------------------------------
> Shed those extra pounds with MSN and The Biggest Loser! Learn more.<http://biggestloser.msn.com/>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-discuss mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of
> this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-discuss
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-devel mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of
> this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-devel
>

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-devel mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-devel