Re: OpenNMS High Availability Second Server (Standby) install - OpenNMS FAILS at startup - HTTP ERROR: 503
Blaine Boswell <[email protected]> Thu, 9 Oct 2014 22:23:54 +0000
| Newsgroups | gmane.network.opennms.install |
|---|---|
| Message-ID | <f90bd54cb9374f6eaf6d19e7fddfc8b3@BY2PR08MB268.namprd08.prod.outlook.com> |
From: Blaine Boswell Sent: Wednesday, October 08, 2014 4:23 PM To: '[email protected]' Subject: RE: OpenNMS High Availability Second Server (Standby) install - OpenNMS FAILS at startup - HTTP ERROR: 503 Okay, the Standby instance of OpenNMS is running on a server with pieces of Netcool that are also JAVA based (various different versions). We found that when we shutdown and restart the Netcool JAVA based processes, all is well with the OpenNMS Jetty login. OpenNMS seems to be working fine at the moment. Don't know yet if this will be a re-occurring issue or if it was a onetime thing after initial install. Has anyone else ever experienced anything like this? None of the ports configured for any of the JAVA processes conflict. I've been monitoring 'netstat -anp | grep 8980' and never seen anything else taking that port??? Regards, Blaine Boswell From: Blaine Boswell Sent: Wednesday, October 08, 2014 9:39 AM To: '[email protected]' Subject: RE: OpenNMS High Availability Second Server (Standby) install - OpenNMS FAILS at startup - HTTP ERROR: 503 Anyone on the Discuss List? I'm grasping for something to explain this behaviour Regards, Blaine Boswell From: Blaine Boswell Sent: Thursday, October 02, 2014 12:07 PM To: '[email protected]' Subject: OpenNMS High Availability Second Server (Standby) install - OpenNMS FAILS at startup Hello all, looking for assistance in which direction to go. I have a good Primary install of OpenNMS up and running. I'm installing the Standby instance and getting a failure at startup of OpenNMS. CentOS 5.10 OpenNMS 1.12.5-1 PostgreSQL 8.3.8 JAVA 1.7.0_67 jicmp-1.4.1 jicmp6-1.2.1 jrrd-1.0.7 iplike-2.0.6 Standby Startup /opt/opennms/logs/daemon/output.log below. I found a reference to " error occurred while attempting to start the "OpenNMS:Name=PollerBackEnd" in some posts from a few versions ago (NMS-5199) that suggests "If you are not using the remote poller then comment out the PollerBackEnd in service-configuration.xml". We are not using it so I commented it out and OpenNMS on Standby server starts successfully but I then get an HTTP ERROR: 503 trying to login to the WebUI. The "jetty.log" goes on for thousands of lines before coming to something that looks like an error. I found a more recent post related to 1.13 testing that indicated PollerBackEnd should not be commented out for some reason. Can anyone please point me in a direction to get the OpenNMS Standby server instance going? I've copied over all the /opt/opennms/etc files from the running Primary and modified opennms-datasources.xml and other files to access PostgresDB and other such files unique to the different server where I recognize them. I've apparently missed something but haven't been able to determine what it is. We are very interested in HA and have what looks like a solid Failover mechanism if someone can help me with what the issue is on the Standby server (either not starting successfully, or starting and getting an error with the WebUI if PollerBackEnd is commented out). Thanks for any assistance --------------------------------------------Web UI error if PollerBackEnd is comented and and OpenNMS starts successfully------------------------------------------ HTTP ERROR: 503 Problem accessing /opennms/login.jsp. Reason: Service Unavailable Powered by Jetty:// -------------------------------jetty.log when Standby OpenNMS starts with PollerBackEnd commented and the above WebUI error appears------------------------------------------------------------------------------------------- 2014-10-01 18:56:57,549 DEBUG [Main] JettyServer: SPRING: thread.classLoader=java.net.FactoryURLClassLoader@74ff0f4<mailto:thread.classLoader=java.net.FactoryURLClassLoader@74ff0f4> 2014-10-01 18:56:57,567 INFO [Main] JettyServer: OpenNMS.JettyServer initializing. 2014-10-01 18:56:57,598 DEBUG [Main] log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.jetty.util.log) via org.eclipse.jetty.util.log.Slf4jLog 2014-10-01 18:56:57,615 DEBUG [Main] Container: Container org.eclipse.jetty.server.Server@543b310c<mailto:org.eclipse.jetty.server.Server@543b310c> + [email protected]:8980<mailto:[email protected]:8980> as connector 2014-10-01 18:56:57,618 WARN [Main] JettyServer: adding context: /opennms -> /opt/opennms/jetty-webapps/opennms 2014-10-01 18:56:57,652 WARN [Main] JettyServer: adding context: /opennms-remoting -> /opt/opennms/jetty-webapps/opennms-remoting 2014-10-01 18:56:57,652 WARN [Main] JettyServer: adding context: / -> /opt/opennms/jetty-webapps/ROOT 2014-10-01 18:56:57,652 DEBUG [Main] Container: Container o.e.j.w.WebAppContext{/opennms,null},/opt/opennms/jetty-webapps/opennms + org.eclipse.jetty.servlet.ErrorPageErrorHandler@4053455d<mailto:org.eclipse.jetty.servlet.ErrorPageErrorHandler@4053455d> as error 2014-10-01 18:56:57,652 DEBUG [Main] Container: Container o.e.j.w.WebAppContext{/opennms-remoting,null},/opt/opennms/jetty-webapps/opennms-remoting + org.eclipse.jetty.servlet.ErrorPageErrorHandler@1aeefda9<mailto:org.eclipse.jetty.servlet.ErrorPageErrorHandler@1aeefda9> as error 2014-10-01 18:56:57,653 DEBUG [Main] Container: Container o.e.j.w.WebAppContext{/,null},/opt/opennms/jetty-webapps/ROOT + org.eclipse.jetty.servlet.ErrorPageErrorHandler@404f700f<mailto:org.eclipse.jetty.servlet.ErrorPageErrorHandler@404f700f> as error 2014-10-01 18:56:57,653 DEBUG [Main] Container: Container org.eclipse.jetty.server.handler.HandlerCollection@1f4609c5<mailto:org.eclipse.jetty.server.handler.HandlerCollection@1f4609c5> + o.e.j.w.WebAppContext{/opennms,null},/opt/opennms/jetty-webapps/opennms as handler 2014-10-01 18:56:57,653 DEBUG [Main] Container: Container org.eclipse.jetty.server.handler.HandlerCollection@1f4609c5<mailto:org.eclipse.jetty.server.handler.HandlerCollection@1f4609c5> + o.e.j.w.WebAppContext{/opennms-remoting,null},/opt/opennms/jetty-webapps/opennms-remoting as handler 2014-10-01 18:56:57,653 DEBUG [Main] Container: Container org.eclipse.jetty.server.handler.HandlerCollection@1f4609c5<mailto:org.eclipse.jetty.server.handler.HandlerCollection@1f4609c5> + o.e.j.w.WebAppContext{/,null},/opt/opennms/jetty-webapps/ROOT as handler 2014-10-01 18:56:57,653 DEBUG [Main] Container: Container org.eclipse.jetty.server.Server@543b310c<mailto:org.eclipse.jetty.server.Server@543b310c> + org.eclipse.jetty.server.handler.HandlerCollection@1f4609c5<mailto:org.eclipse.jetty.server.handler.HandlerCollection@1f4609c5> as handler 2014-10-01 18:56:57,653 INFO [Main] JettyServer: OpenNMS.JettyServer initialization complete. 2014-10-01 18:56:57,653 DEBUG [Main] JettyServer: SPRING: context.classLoader= java.net.FactoryURLClassLoader@74ff0f4<mailto:java.net.FactoryURLClassLoader@74ff0f4> 2014-10-01 18:56:59,119 INFO [Main] JettyServer: OpenNMS.JettyServer starting. ... 2014-10-01 18:59:24,943 DEBUG [qtp1591049279-1148 - /opennms/rtc/post/Trial] WebAppClassLoader: loaded class org.opennms.netmgt.xml.rtc.Category from java.net.FactoryURLClassLoader@74ff0f4<mailto:java.net.FactoryURLClassLoader@74ff0f4> 2014-10-01 18:59:24,945 DEBUG [qtp1591049279-1148 - /opennms/rtc/post/Trial] WebAppClassLoader: loaded class java.io.PrintWriter 2014-10-01 18:59:24,945 DEBUG [qtp1591049279-1148 - /opennms/rtc/post/Trial] WebAppClassLoader: loaded class java.io.PrintWriter from null 2014-10-01 18:59:24,950 DEBUG [qtp1591049279-1148 - /opennms/rtc/post/Trial] ContextHandler: scope null||/opennms/rtc/post/Trial @ o.e.j.w.WebAppContext{/opennms-remoting,file:/opt/opennms/jetty-webapps/opennms-remoting/},/opt/opennms/jetty-webapps/opennms-remoting 2014-10-01 18:59:24,950 WARN [qtp1591049279-1148 - /opennms/rtc/post/Trial] Response: Committed before 503 null 2014-10-01 18:59:24,950 WARN [qtp1591049279-1148 - /opennms/rtc/post/Trial] AbstractHttpConnection: /opennms/rtc/post/Trial java.lang.IllegalStateException: Committed at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1141) at org.eclipse.jetty.server.Response.sendError(Response.java:313) at org.eclipse.jetty.server.Response.sendError(Response.java:415) at org.eclipse.jetty.server.handler.ContextHandler.checkContext(ContextHandler.java:826) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:922) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:365) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:944) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1005) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) 2014-10-01 18:59:24,951 DEBUG [qtp1591049279-1148 - /opennms/rtc/post/Trial] AbstractGenerator: sendError on committed: 500 null 2014-10-01 18:59:24,951 DEBUG [qtp1591049279-1148] ChannelEndPoint: oshut SCEP@4b2845e5{l(/127.0.0.1:43810)<->r(/127.0.0.1:8980),s=1,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=0r}-{AsyncHttpConnection@4104e3ef,g=HttpGenerator{s=4,h=0,b=0,c=-1},p=HttpParser{s=0,l=4,c=7521},r=1}<mailto:SCEP@4b2845e5%7bl(/127.0.0.1:43810)%3c-%3er(/127.0.0.1:8980),s=1,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=0r%7d-%7bAsyncHttpConnection@4104e3ef,g=HttpGenerator%7bs=4,h=0,b=0,c=-1%7d,p=HttpParser%7bs=0,l=4,c=7521%7d,r=1%7d> 2014-10-01 18:59:24,951 DEBUG [qtp1591049279-1148] AsyncHttpConnection: Enabled read interest SCEP@4b2845e5{l(/127.0.0.1:43810)<->r(/127.0.0.1:8980),s=1,open=true,ishut=false,oshut=true,rb=false,wb=false,w=true,i=0r}-{AsyncHttpConnection@4104e3ef,g=HttpGenerator{s=4,h=0,b=0,c=-1},p=HttpParser{s=0,l=4,c=7521},r=1}<mailto:SCEP@4b2845e5%7bl(/127.0.0.1:43810)%3c-%3er(/127.0.0.1:8980),s=1,open=true,ishut=false,oshut=true,rb=false,wb=false,w=true,i=0r%7d-%7bAsyncHttpConnection@4104e3ef,g=HttpGenerator%7bs=4,h=0,b=0,c=-1%7d,p=HttpParser%7bs=0,l=4,c=7521%7d,r=1%7d> 2014-10-01 18:59:24,951 DEBUG [qtp1591049279-1148] HttpParser: filled 0/0 2014-10-01 18:59:24,998 DEBUG [qtp1591049279-1136] HttpParser: filled 6144/6144 2014-10-01 18:59:24,999 DEBUG [qtp1591049279-1136] ChannelEndPoint: close SCEP@4b2845e5{l(/127.0.0.1:43810)<->r(/127.0.0.1:8980),s=1,open=true,ishut=false,oshut=true,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@4104e3ef,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=2,l=5,c=18593},r=1}<mailto:SCEP@4b2845e5%7bl(/127.0.0.1:43810)%3c-%3er(/127.0.0.1:8980),s=1,open=true,ishut=false,oshut=true,rb=false,wb=false,w=true,i=1r%7d-%7bAsyncHttpConnection@4104e3ef,g=HttpGenerator%7bs=0,h=-1,b=-1,c=-1%7d,p=HttpParser%7bs=2,l=5,c=18593%7d,r=1%7d> 2014-10-01 18:59:24,999 DEBUG [qtp1591049279-1129 Selector1] nio: created SCEP@13e61c2d{l(/127.0.0.1:43818)<->r(/127.0.0.1:8980),s=0,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=0}-{AsyncHttpConnection@6843d88f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0}<mailto:SCEP@13e61c2d%7bl(/127.0.0.1:43818)%3c-%3er(/127.0.0.1:8980),s=0,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=0%7d-%7bAsyncHttpConnection@6843d88f,g=HttpGenerator%7bs=0,h=-1,b=-1,c=-1%7d,p=HttpParser%7bs=-14,l=0,c=0%7d,r=0%7d> 2014-10-01 18:59:24,999 DEBUG [qtp1591049279-1136] ChannelEndPoint: Exception while filling java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.eclipse.jetty.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:235) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.fill(SelectChannelEndPoint.java:366) at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1044) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:280) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) 2014-10-01 18:59:25,000 DEBUG [qtp1591049279-1128 Selector0] nio: destroyEndPoint SCEP@4b2845e5{l(null)<->r(0.0.0.0/0.0.0.0:8980),s=1,open=false,ishut=true,oshut=true,rb=false,wb=false,w=true,i=1!}-{AsyncHttpConnection@4104e3ef,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=2,l=5,c=18593},r=1}<mailto:SCEP@4b2845e5%7bl(null)%3c-%3er(0.0.0.0/0.0.0.0:8980),s=1,open=false,ishut=true,oshut=true,rb=false,wb=false,w=true,i=1!%7d-%7bAsyncHttpConnection@4104e3ef,g=HttpGenerator%7bs=0,h=-1,b=-1,c=-1%7d,p=HttpParser%7bs=2,l=5,c=18593%7d,r=1%7d> 2014-10-01 18:59:25,000 DEBUG [qtp1591049279-1136] HttpParser: filled -1/0 2014-10-01 18:59:25,000 DEBUG [qtp1591049279-1128 Selector0] AbstractHttpConnection: closed AsyncHttpConnection@4104e3ef,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=0,l=5,c=18593},r=1 2014-10-01 18:59:25,000 DEBUG [qtp1591049279-1136] AsyncHttpConnection: Disabled read interest while writing response SCEP@4b2845e5{l(null)<->r(0.0.0.0/0.0.0.0:8980),s=1,open=false,ishut=true,oshut=true,rb=false,wb=false,w=true,i=1!}-{AsyncHttpConnection@4104e3ef,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=0,l=5,c=18593},r=1}<mailto:SCEP@4b2845e5%7bl(null)%3c-%3er(0.0.0.0/0.0.0.0:8980),s=1,open=false,ishut=true,oshut=true,rb=false,wb=false,w=true,i=1!%7d-%7bAsyncHttpConnection@4104e3ef,g=HttpGenerator%7bs=0,h=-1,b=-1,c=-1%7d,p=HttpParser%7bs=0,l=5,c=18593%7d,r=1%7d> 2014-10-01 18:59:25,000 DEBUG [qtp1591049279-1148] HttpParser: filled 6144/6144 2014-10-01 18:59:25,001 DEBUG [qtp1591049279-1148 - /opennms/rtc/post/Overall+Service+Availability] Server: REQUEST /opennms/rtc/post/Overall+Service+Availability on AsyncHttpConnection@6843d88f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=2,l=5,c=18593},r=1 2014-10-01 18:59:25,001 DEBUG [qtp1591049279-1148 - /opennms/rtc/post/Overall+Service+Availability] ContextHandler: scope null||/opennms/rtc/post/Overall+Service+Availability @ o.e.j.w.WebAppContext{/opennms,file:/opt/opennms/jetty-webapps/opennms/},/opt/opennms/jetty-webapps/opennms 2014-10-01 18:59:25,001 DEBUG [qtp1591049279-1148 - /opennms/rtc/post/Overall+Service+Availability] ContextHandler: context=/opennms||/rtc/post/Overall+Service+Availability @ o.e.j.w.WebAppContext{/opennms,file:/opt/opennms/jetty-webapps/opennms/},/opt/opennms/jetty-webapps/opennms 2014-10-01 18:59:25,001 DEBUG [qtp1591049279-1148 - /opennms/rtc/post/Overall+Service+Availability] session: sessionManager=org.eclipse.jetty.server.session.HashSessionManager@10c32432<mailto:sessionManager=org.eclipse.jetty.server.session.HashSessionManager@10c32432> 2014-10-01 18:59:25,001 DEBUG [qtp1591049279-1148 - /opennms/rtc/post/Overall+Service+Availability] session: session=null --------------------------/opt/opennms/logs/daemon/output.log when PollerBackEnd is not commented and OpenNMS fails to start on Standby server---------------------------------------------------- Wed Oct 1 16:36:37 UTC 2014 begin ulimit settings: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 386164 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 20480 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 386164 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited end ulimit settings Executing command: /opt/opennms/bin/runjava -r -- -Djava.endorsed.dirs=/opt/opennms/lib/endorsed -Dopennms.home=/opt/opennms -Dcom.sun.management.jmxremote.port=18980 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xmx1536m -XX:+HeapDumpOnOutOfMemoryError -Xms1536m -XX:+UseParallelGC -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:MaxPermSize=256m -Djava.io.tmpdir=/opt/opennms/data/tmp -verbose:gc -jar /opt/opennms/lib/opennms_bootstrap.jar start [DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so. Attempting to load jicmp library from this location. [INFO] Successfully loaded jicmp library. [DEBUG] System property 'opennms.library.jicmp6' set to '/usr/lib64/libjicmp6.so. Attempting to load jicmp6 library from this location. [INFO] Successfully loaded jicmp6 library. 3.578: [GC [PSYoungGen: 393216K->16741K(458752K)] 393216K->16814K(1507328K), 0.0214200 secs] [Times: user=0.09 sys=0.01, real=0.03 secs] 6.630: [GC [PSYoungGen: 409957K->21738K(458752K)] 410030K->21818K(1507328K), 0.0275770 secs] [Times: user=0.19 sys=0.01, real=0.02 secs] 8.430: [GC [PSYoungGen: 414954K->23806K(458752K)] 415034K->23958K(1507328K), 0.0412640 secs] [Times: user=0.23 sys=0.00, real=0.04 secs] 16.190: [GC [PSYoungGen: 417022K->52923K(458752K)] 417174K->53083K(1507328K), 0.0281750 secs] [Times: user=0.26 sys=0.02, real=0.03 secs] 19.124: [GC [PSYoungGen: 431204K->65521K(458752K)] 431364K->86451K(1507328K), 0.0350300 secs] [Times: user=0.34 sys=0.03, real=0.03 secs] 19.159: [Full GC [PSYoungGen: 65521K->0K(458752K)] [ParOldGen: 20929K->84197K(1048576K)] 86451K->84197K(1507328K) [PSPermGen: 44806K->44632K(89600K)], 0.4190460 secs] [Times: user=3.25 sys=0.06, real=0.42 secs] An error occurred while attempting to start the "OpenNMS:Name=PollerBackEnd" service (class org.opennms.netmgt.poller.jmx.RemotePollerBackEnd). Shutting down and exiting. javax.management.RuntimeMBeanException: org.springframework.beans.factory.access.BootstrapException: Unable to return specified BeanFactory instance: factory key [pollerBackEndContext], from group with resource name [classpath*:beanRefContext.xml]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pollerBackEndContext' defined in URL [jar:file:/opt/opennms/lib/opennms-services-1.12.5.jar!/beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [org.springframework.context.support.ClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pollerBackEnd-rmi' defined in class path resource [META-INF/opennms/applicationContext-exportedPollerBackEnd-rmi.xml]: Invocation of init method failed; nested exception is java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: java.lang.ClassNotFoundException: org.springframework.remoting.rmi.RmiInvocationHandler (no security manager: RMI class loader disabled) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:821) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) at org.opennms.netmgt.vmmgr.Invoker.invoke(Invoker.java:316) at org.opennms.netmgt.vmmgr.Invoker.invokeMethods(Invoker.java:239) at org.opennms.netmgt.vmmgr.Starter.start(Starter.java:245) at org.opennms.netmgt.vmmgr.Starter.startDaemon(Starter.java:116) at org.opennms.netmgt.vmmgr.Controller.start(Controller.java:190) at org.opennms.netmgt.vmmgr.Controller.main(Controller.java:162) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.opennms.bootstrap.Bootstrap$3.run(Bootstrap.java:362) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.beans.factory.access.BootstrapException: Unable to return specified BeanFactory instance: factory key [pollerBackEndContext], from group with resource name [classpath*:beanRefContext.xml]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pollerBackEndContext' defined in URL [jar:file:/opt/opennms/lib/opennms-services-1.12.5.jar!/beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [org.springframework.context.support.ClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pollerBackEnd-rmi' defined in class path resource [META-INF/opennms/applicationContext-exportedPollerBackEnd-rmi.xml]: Invocation of init method failed; nested exception is java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: java.lang.ClassNotFoundException: org.springframework.remoting.rmi.RmiInvocationHandler (no security manager: RMI class loader disabled) at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:409) at org.opennms.core.utils.BeanUtils.getBeanFactory(BeanUtils.java:58) at org.opennms.core.utils.BeanUtils.getFactory(BeanUtils.java:100) at org.opennms.netmgt.daemon.AbstractSpringContextJmxServiceDaemon.init(AbstractSpringContextJmxServiceDaemon.java:94) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) ... 13 more Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pollerBackEndContext' defined in URL [jar:file:/opt/opennms/lib/opennms-services-1.12.5.jar!/beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [org.springframework.context.support.ClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pollerBackEnd-rmi' defined in class path resource [META-INF/opennms/applicationContext-exportedPollerBackEnd-rmi.xml]: Invocation of init method failed; nested exception is java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: java.lang.ClassNotFoundException: org.springframework.remoting.rmi.RmiInvocationHandler (no security manager: RMI class loader disabled) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:288) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1002) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:906) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:484) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:455) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:196) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1079) at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:397) ... 31 more Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [org.springframework.context.support.ClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pollerBackEnd-rmi' defined in class path resource [META-INF/opennms/applicationContext-exportedPollerBackEnd-rmi.xml]: Invocation of init method failed; nested exception is java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: java.lang.ClassNotFoundException: org.springframework.remoting.rmi.RmiInvocationHandler (no security manager: RMI class loader disabled) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:141) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:108) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:280) ... 41 more Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pollerBackEnd-rmi' defined in class path resource [META-INF/opennms/applicationContext-exportedPollerBackEnd-rmi.xml]: Invocation of init method failed; nested exception is java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: java.lang.ClassNotFoundException: org.springframework.remoting.rmi.RmiInvocationHandler (no security manager: RMI class loader disabled) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1422) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:518) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:455) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425) at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139) at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:105) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126) ... 43 more Caused by: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: java.lang.ClassNotFoundException: org.springframework.remoting.rmi.RmiInvocationHandler (no security manager: RMI class loader disabled) at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:400) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:255) at sun.rmi.transport.Transport$1.run(Transport.java:168) at java.security.AccessController.doPrivileged(AccessController.java:275) at sun.rmi.transport.Transport.serviceCall(Transport.java:164) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912) at java.lang.Thread.run(Thread.java:801) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:275) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:252) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:378) at sun.rmi.registry.RegistryImpl_Stub.rebind(Unknown Source) at org.springframework.remoting.rmi.RmiServiceExporter.prepare(RmiServiceExporter.java:291) at org.springframework.remoting.rmi.RmiServiceExporter.afterPropertiesSet(RmiServiceExporter.java:229) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1479) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1419) ... 59 more Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: java.lang.ClassNotFoundException: org.springframework.remoting.rmi.RmiInvocationHandler (no security manager: RMI class loader disabled) at sun.rmi.registry.RegistryImpl_Skel.dispatch(RegistryImpl_Skel.java:102) at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:390) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:255) at sun.rmi.transport.Transport$1.run(Transport.java:168) at java.security.AccessController.doPrivileged(AccessController.java:275) at sun.rmi.transport.Transport.serviceCall(Transport.java:164) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912) at java.lang.Thread.run(Thread.java:801) Caused by: java.lang.ClassNotFoundException: org.springframework.remoting.rmi.RmiInvocationHandler (no security manager: RMI class loader disabled) at sun.rmi.server.LoaderHandler.loadProxyClass(LoaderHandler.java:546) at java.rmi.server.RMIClassLoader$2.loadProxyClass(RMIClassLoader.java:640) at org.apache.geronimo.system.rmi.RMIClassLoaderSpiImpl.loadProxyClass(RMIClassLoaderSpiImpl.java:58) at java.rmi.server.RMIClassLoader.loadProxyClass(RMIClassLoader.java:306) at sun.rmi.server.MarshalInputStream.resolveProxyClass(MarshalInputStream.java:253) at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1519) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1482) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1718) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1324) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:362) at sun.rmi.registry.RegistryImpl_Skel.dispatch(RegistryImpl_Skel.java:98) ... 9 more Heap PSYoungGen total 458752K, used 12006K [0x00000000e0000000, 0x0000000100000000, 0x0000000100000000) eden space 393216K, 3% used [0x00000000e0000000,0x00000000e0bb9838,0x00000000f8000000) from space 65536K, 0% used [0x00000000f8000000,0x00000000f8000000,0x00000000fc000000) to space 65536K, 0% used [0x00000000fc000000,0x00000000fc000000,0x0000000100000000) ParOldGen total 1048576K, used 84197K [0x00000000a0000000, 0x00000000e0000000, 0x00000000e0000000) object space 1048576K, 8% used [0x00000000a0000000,0x00000000a5239500,0x00000000e0000000) PSPermGen total 89600K, used 44786K [0x0000000090000000, 0x0000000095780000, 0x00000000a0000000) object space 89600K, 49% used [0x0000000090000000,0x0000000092bbcbf0,0x0000000095780000) Regards, Blaine Boswell This message is confidential to Prodea unless otherwise indicated or apparent from its nature. This message is directed to the intended recipient only, who may be readily determined by the sender of this message and its contents. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient:(a)any dissemination or copying of this message is strictly prohibited; and(b)immediately notify the sender by return message and destroy any copies of this message in any form(electronic, paper or otherwise) that you have.The delivery of this message and its information is neither intended to be nor constitutes a disclosure or waiver of any trade secrets, intellectual property, attorney work product, or attorney-client communications. The authority of the individual sending this message to legally bind Prodea is neither apparent nor implied,and must be independently verified. ------------------------------------------------------------------------------ Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk _______________________________________________ Please read the OpenNMS Mailing List FAQ: http://www.opennms.org/index.php/Mailing_List_FAQ opennms-install mailing list To *unsubscribe* or change your subscription options, see the bottom of this page: https://lists.sourceforge.net/lists/listinfo/opennms-install