Re: [opennms-devel] Change default connection pool size to less than 100? Was: [opennms-discuss] 1.3.11 Hangs after hours of running

If I'm not completely off bat - it crashes.

On Mon, Mar 3, 2008 at 1:55 PM, DJ Gregor <[email protected]> wrote:

> Since some OSes are even smaller, I wonder if there is a way we can
> check to see how many database connection slots are available.
>
> The PostgreSQL statistics collector will let you know how many
> connections exist to the database right now, but I don't think it lets
> us know how many slots are open:
>
> http://www.postgresql.org/docs/8.3/interactive/monitoring-stats.html
>
> What happens if you try to start the daemon and it can't get enough
> connections?  Does anyone know if it dies or hangs around trying for
> awhile/ever?
>
>
>        - djg
>
> On Mon, 3 Mar 2008 11:27:32 -0500, "Johan Edstrom" <[email protected]>
> said:
> > 50-75 seems to have worked quite well for me....
> > I usually turn this down, on some OS'es Like FBSD it
> > is actually by default way lower..
> >
> > On Mon, Mar 3, 2008 at 3:58 AM, DJ Gregor <[email protected]> wrote:
> >
> > > Should we change the default connection pool size to something other
> than
> > > 100?
> > >
> > > That happens to be _exactly_ the postgresql default, which seems
> > > particularly troubling (oh, you wanted to run 'psql' while opennms is
> > > running?  Sorry.).  Also, 100 concurrent connections just seems
> absolutely
> > > huge for all but the largest sites.
> > >
> > >
> > > Begin forwarded message:
> > >
> > > *From:* Miakel Roberts <[email protected]>
> > > *Date:* February 28, 2008 8:58:34 PM EST
> > > *To:* <[email protected]>
> > > *Subject:* *Re: [opennms-discuss] 1.3.11 Hangs after hours of running*
> > > *Reply-To:* General OpenNMS Discussion <
> > > [email protected]>
> > >
> > > after much work p-brane suggested I change pool size in the
> > > c3p0.properties file from 50 to 100 at the 3 places it's defined.
>  Upon a
> > > OpenNMS restart I have been running for almost 24 hours without a lock
> up !!
> > >
> > > Thanks for all the assistance !!
> > >
> > >
> > > Mike
> > >
> > >
> > >
> > > > From: <[email protected]>[email protected]
> > > > To: <[email protected]>
> > > [email protected]
> > > > Subject: RE: 1.3.11 Hangs after hours of running
> > > > Date: Wed, 27 Feb 2008 11:34:45 -0600
> > > >
> > > >
> > > > per request
> > > >
> > > > http://pastebin.ca/920833 - c3p0 properties
> > > >
> > > > http://pastebin.ca/920830 - postgresql.conf
> > > >
> > > >
> > > > ----------------------------------------
> > > > > From: <[email protected]>[email protected]
> > > > > To: <[email protected]>
> > > [email protected]
> > > > > Subject: RE: 1.3.11 Hangs after hours of running
> > > > > Date: Wed, 27 Feb 2008 09:24:43 -0600
> > > > >
> > > > >
> > > > > more info on the lock and activity and process
> > > > >
> > > > > [noc@canis etc]$ ps aux | grep 21915
> > > > > postgres 21915 0.0 0.0 423852 3324 ? S 09:22 0:00 postgres:
> opennms
> > > opennms [local] idle
> > > > > noc 22100 0.0 0.0 3884 688 pts/2 S+ 09:23 0:00 grep 21915
> > > > > [noc@canis etc]$
> > > > >
> > > > > http://pastebin.ca/920662 - lock and activity
> > > > >
> > > > >
> > > > >
> > > > > ----------------------------------------
> > > > >> From: <[email protected]>[email protected]
> > > > >> To: <[email protected]>
> > > [email protected]
> > > > >> Subject: RE: 1.3.11 Hangs after hours of running
> > > > >> Date: Wed, 27 Feb 2008 08:59:28 -0600
> > > > >>
> > > > >>
> > > > >> 2 stack trace dumps taken while opennms is locked taken about 20
> > > minutes apart.
> > > > >>
> > > > >> <http://pastebin.ca/920602>http://pastebin.ca/920602
> > > > >>
> > > > >> I also checked pg_locks and pg_activity and results showed the
> > > following
> > > > >> <http://pastebin.ca/920610>http://pastebin.ca/920610
> > > > >>
> > > > >> PID 7314 is postgres
> > > > >>
> > > > >>
> > > > >> ----------------------------------------
> > > > >>> From: <[email protected]>[email protected]
> > > > >>> To: <[email protected]>
> > > [email protected]
> > > > >>> Subject: RE: 1.3.11 Hangs after hours of running
> > > > >>> Date: Tue, 26 Feb 2008 12:17:33 -0600
> > > > >>>
> > > > >>>
> > > > >>> http://pastebin.ca/919237 - latest stack trace while the
> previously
> > > post content was captured.
> > > > >>> ----------------------------------------
> > > > >>>> From: <[email protected]>[email protected]
> > > > >>>> To: <[email protected]>
> > > [email protected]
> > > > >>>> Subject: RE: 1.3.11 Hangs after hours of running
> > > > >>>> Date: Tue, 26 Feb 2008 11:56:52 -0600
> > > > >>>>
> > > > >>>>
> > > > >>>> I tried to post this to the list last evening but for some
> reason
> > > it did not go.
> > > > >>>>
> > > > >>>> http://pastebin.ca/918151 - capsd while system is not
> responding
> > > via webgui
> > > > >>>> http://pastebin.ca/918148 - poller.log while system is not
> > > responding
> > > > >>>> http://pastebin.ca/918133 - pg_stat_Activity while system is
> not
> > > responding
> > > > >>>> http://pastebin.ca/918134 - top while system is not responding
> > > > >>>> http://pastebin.ca/918132 - pg_locks while system is not
> responding
> > >
> > > > >>>> http://pastebin.ca/918129 - vacuum.log while system is not
> > > responding
> > > > >>>> http://pastebin.ca/917941 - some output from the postreslog,
> "looks
> > > scary": joed
> > > > >>>>
> > > > >>>>
> > > > >>>> ----------------------------------------
> > > > >>>>> From: <[email protected]>[email protected]
> > > > >>>>> To: <[email protected]>
> > > [email protected]
> > > > >>>>> Subject: RE: 1.3.11 Hangs after hours of running
> > > > >>>>> Date: Mon, 25 Feb 2008 12:01:53 -0600
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> I have been doing more digging and have come accross this in
> the
> > > postgresql-Mon.log
> > > > >>>>>
> > > > >>>>> LOG: autovacuum: processing database "template1"
> > > > >>>>> LOG: autovacuum: processing database "postgres"
> > > > >>>>> LOG: autovacuum: processing database "test"
> > > > >>>>> ERROR: duplicate key violates unique constraint
> > > "alarm_reductionkey_idx"
> > > > >>>>> ERROR: current transaction is aborted, commands ignored until
> end
> > > of transaction block
> > > > >>>>> LOG: autovacuum: processing database "opennms"
> > > > >>>>> LOG: autovacuum: processing database "template1"
> > > > >>>>> LOG: autovacuum: processing database "postgres"
> > > > >>>>> LOG: autovacuum: processing database "test"
> > > > >>>>> LOG: autovacuum: processing database "opennms"
> > > > >>>>> LOG: autovacuum: processing database "template1"
> > > > >>>>> LOG: autovacuum: processing database "postgres"
> > > > >>>>> LOG: autovacuum: processing database "test"
> > > > >>>>> ERROR: duplicate key violates unique constraint
> > > "alarm_reductionkey_idx"
> > > > >>>>> ERROR: current transaction is aborted, commands ignored until
> end
> > > of transaction block
> > > > >>>>> LOG: autovacuum: processing database "opennms"
> > > > >>>>> LOG: autovacuum: processing database "template1"
> > > > >>>>> LOG: autovacuum: processing database "postgres"
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> Mike
> > > > >>>>> ________________________________
> > > > >>>>>> From: <[email protected]>[email protected]
> > > > >>>>>> To: <[email protected]>
> > > [email protected]
> > > > >>>>>> Subject: RE: 1.3.11 Hangs after hours of running
> > > > >>>>>> Date: Mon, 25 Feb 2008 09:40:05 -0600
> > > > >>>>>>
> > > > >>>>>> This is the FULL log.
> > > > >>>>>>
> > > > >>>>>> <http://pastebin.ca/917637>http://pastebin.ca/917637
> > > > >>>>>>
> > > > >>>>>> sorry about the last 2 logs.
> > > > >>>>>>
> > > > >>>>>> Mike
> > > > >>>>>>
> > > > >>>>>>> From: <[email protected]>[email protected]
> > > > >>>>>>> To: <[email protected]>
> > > [email protected]
> > > > >>>>>>> Subject: RE: 1.3.11 Hangs after hours of running
> > > > >>>>>>> Date: Mon, 25 Feb 2008 09:19:37 -0600
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> I am running in full debug.
> > > > >>>>>>> David you are correct, that is not a full dump, I think I
> must
> > > have lost some of the dump in the copy buffer.
> > > > >>>>>>> I'll try again.
> > > > >>>>>>>
> > > > >>>>>>> Here is the complete output.log from start of the opennms
> (since
> > > last hang) to the next start after it hung.
> > > > >>>>>>> <http://opennms.pastebin.org/21192>
> > > http://opennms.pastebin.org/21192
> > > > >>>>>>>
> > > > >>>>>>> Mike
> > > > >>>>>>> ________________________________
> > > > >>>>>>>> From: <[email protected]>[email protected]
> > > > >>>>>>>> To: <[email protected]>
> > > [email protected]
> > > > >>>>>>>> Subject: 1.3.11 Hangs after hours of running
> > > > >>>>>>>> Date: Mon, 25 Feb 2008 08:04:09 -0600
> > > > >>>>>>>>
> > > > >>>>>>>> Guys,
> > > > >>>>>>>> I have been having issues with 1.3.11 (don't know if it's
> > > version related or not) ever since I went from 1.3.9 to 1.3.11.
> > > > >>>>>>>>
> > > > >>>>>>>> Symptoms are, after maybe 8 hours of running no one can log
> in
> > > via the web interface, I can see the user / pass page but as soon as I
> click
> > > login it just sits and hangs.
> > > > >>>>>>>> The system also at some point stops data collection /
> polling
> > > after this lockup.
> > > > >>>>>>>> Restarting opennms fixes it for another 8 or so hours.
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> I've tried numerous things, probably not all that I'll
> remember
> > > here,
> > > > >>>>>>>> I've done a dumpall dropdb and restore. no dif
> > > > >>>>>>>> reinstall using bin/install. no diff
> > > > >>>>>>>>
> > > > >>>>>>>> I've done a kill -3 as per dj directions and will include
> > > pastebins below.
> > > > >>>>>>>>
> > > > >>>>>>>> http://opennms.pastebin.org/21159 - full dump while system
> not
> > > responding
> > > > >>>>>>>>
> > > > >>>>>>>> http://opennms.pastebin.org/21157 - pg_stat_activity &
> pg_locks
> > > while system not responding
> > > > >>>>>>>>
> > > > >>>>>>>> http://opennms.pastebin.org/21158 - top while system is not
> > > responding
> > > > >>>>>>>>
> > > > >>>>>>>> http://opennms.pastebin.org/21183 - iostat while system is
> > > running fine
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> Server stats are listed below
> > > > >>>>>>>>
> > > > >>>>>>>> 2 x 2.33ghz QuadCore
> > > > >>>>>>>> 16gb ram
> > > > >>>>>>>> 8 x ultra 320 SAS
> > > > >>>>>>>> raid1 - os and (just moved here from the raid10 to see if
> this
> > > was the cause, it's not) the db
> > > > >>>>>>>> raid10 - rrd files
> > > > >>>>>>>> Centos 5.0 x86
> > > > >>>>>>>>
> > > > >>>>>>>> Opennms nodes
> > > > >>>>>>>> ~ 6000
> > > > >>>>>>>> ~ 1500 interfaces being polled for snmp data
> > > > >>>>>>>> ~ 900,000 event table
> > > > >>>>>>>>
> > > > >>>>>>>> Any assistance would be greatly appreciated !!
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> Mike
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> ________________________________
> > > > >>>>>>>> Need to know the score, the latest news, or you need your
> > > Hotmail(R)-get your "fix". Check it out.
> > > > >>>>>>>
> > > > >>>>>>>
> > > _________________________________________________________________
> > > > >>>>>>> Connect and share in new ways with Windows Live.
> > > > >>>>>>>
> > > <
> http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008
> >
> > >
> http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008
> > > > >>>>>>
> > > > >>>>>> ________________________________
> > > > >>>>>> Need to know the score, the latest news, or you need your
> > > Hotmail(R)-get your "fix". Check it out.
> > > > >>>>>
> > > > >>>>>
> _________________________________________________________________
> > > > >>>>> Connect and share in new ways with Windows Live.
> > > > >>>>>
> > > <
> http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008
> >
> > >
> http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_012008
> > > > >>>>
> > > > >>>>
> _________________________________________________________________
> > > > >>>> Need to know the score, the latest news, or you need your
> > > Hotmail(R)-get your "fix".
> > > > >>>> <http://www.msnmobilefix.com/Default.aspx>
> > > http://www.msnmobilefix.com/Default.aspx
> > > > >>>
> > > > >>>
> _________________________________________________________________
> > > > >>> Climb to the top of the charts! Play the word scramble challenge
> > > with star power.
> > > > >>>
> > > <
> http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan
> >
> > >
> http://club.live.com/star_shuffle.aspx?icid=starshuffle_wlmailtextlink_jan
> > > > >>
> > > > >> _________________________________________________________________
> > > > >> Need to know the score, the latest news, or you need your
> > > Hotmail(R)-get your "fix".
> > > > >> <http://www.msnmobilefix.com/Default.aspx>
> > > http://www.msnmobilefix.com/Default.aspx
> > > > >
> > > > > _________________________________________________________________
> > > > > Need to know the score, the latest news, or you need your
> Hotmail(R)-get
> > > your "fix".
> > > > > <http://www.msnmobilefix.com/Default.aspx>
> > > http://www.msnmobilefix.com/Default.aspx
> > > >
> > > > _________________________________________________________________
> > > > Shed those extra pounds with MSN and The Biggest Loser!
> > > > <http://biggestloser.msn.com/>http://biggestloser.msn.com/
> > >
> > > ------------------------------
> > > Shed those extra pounds with MSN and The Biggest Loser! Learn more.<
> http://biggestloser.msn.com/>
> > >
> > >
> -------------------------------------------------------------------------
> > > This SF.net email is sponsored by: Microsoft
> > > Defy all challenges. Microsoft(R) Visual Studio 2008.
> > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > >
> > > _______________________________________________
> > > Please read the OpenNMS Mailing List FAQ:
> > > http://www.opennms.org/index.php/Mailing_List_FAQ
> > >
> > > opennms-discuss mailing list
> > >
> > > To *unsubscribe* or change your subscription options, see the bottom
> of
> > > this page:
> > > https://lists.sourceforge.net/lists/listinfo/opennms-discuss
> > >
> > >
> > >
> -------------------------------------------------------------------------
> > > This SF.net email is sponsored by: Microsoft
> > > Defy all challenges. Microsoft(R) Visual Studio 2008.
> > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > > _______________________________________________
> > > Please read the OpenNMS Mailing List FAQ:
> > > http://www.opennms.org/index.php/Mailing_List_FAQ
> > >
> > > opennms-devel mailing list
> > >
> > > To *unsubscribe* or change your subscription options, see the bottom
> of
> > > this page:
> > > https://lists.sourceforge.net/lists/listinfo/opennms-devel
> > >
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-devel mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of
> this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-devel
>

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-devel mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-devel