Opennms-1.8.3-1 start error

For the life of me, I can't get Opennms-1.8.3-1 to start after upgrading from 1.8.1. It's been few days now. Removed and reinstalled Opennms from scratch and with default everything it still reports the same error, so I don't think it's any etc/xxx configuration errors.

[root@ranlxnms01 /]# service opennms -v start
Starting OpenNMS: Could not connect to 127.0.0.1 on port 8181 (OpenNMS might not be running or could be starting up or shutting down): Connection refused
------------------------------------------------------------------------------
Fri Aug 27 10:26:58 EST 2010
begin ulimit settings:
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 268287
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 10240
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 268287
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
end ulimit settings
Executing command: /opt/opennms/bin/runjava -r  -- -Djava.endorsed.dirs=/opt/opennms/lib/endorsed   -Dopennms.home=/opt/opennms -Dcom.sun.management.jmxremote.port=18980 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xmx256m -Djava.net.preferIPv4Stack=true  -XX:MaxPermSize=128m -verbose:gc -jar /opt/opennms/lib/opennms_bootstrap.jar  start
[GC 3072K->811K(5376K), 0.0043310 secs]
Could not connect to 127.0.0.1 on port 8181 (OpenNMS might not be running or could be starting up or shutting down): Connection refused
[GC 3883K->1824K(8448K), 0.0047720 secs]
[Full GC 1824K->1813K(12608K), 0.0195510 secs]
[GC 7914K->2458K(12608K), 0.0059270 secs]
[GC 8602K->2987K(14784K), 0.0047380 secs]
[GC 11307K->3621K(14912K), 0.0029210 secs]
[DEBUG] System property 'opennms.library.jicmp' set to '/usr/lib64/libjicmp.so.  Attempting to load jicmp library from this location.
[INFO] Successfully loaded jicmp library.
[GC 12069K->4053K(18240K), 0.0029430 secs]
[GC 15061K->4315K(18432K), 0.0031370 secs]
[GC 15323K->4395K(21184K), 0.0030050 secs]
[GC 18155K->4555K(21312K), 0.0038170 secs]
[GC 18315K->4810K(23296K), 0.0033400 secs]
[GC 20490K->5002K(23488K), 0.0038160 secs]
[GC 20682K->6882K(25088K), 0.0080700 secs]
[Full GC 6882K->6654K(31552K), 0.0611840 secs]
[GC 23036K->8668K(32384K), 0.0127490 secs]
[GC 25756K->10263K(35520K), 0.0072220 secs]
[GC 30167K->10815K(36352K), 0.0082210 secs]
[GC 30719K->11837K(40384K), 0.0088270 secs]
[GC 35325K->13715K(41152K), 0.0103290 secs]
An error occurred while attempting to start the "OpenNMS:Name=Eventd" service (class org.opennms.netmgt.eventd.jmx.Eventd).  Shutting down and exiting.
-
-
-
Caused by: org.springframework.beans.factory.access.BootstrapException: Unable to return specified BeanFactory instance: factory key [eventDaemonContext], from group with resource name [classpath*:beanRefContext.xml]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'eventDaemonContext' defined in URL [jar:file:/opt/opennms/lib/opennms-services-1.8.3.jar!/beanRefContext.xml]: Cannot resolve reference to bean 'commonContext' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'commonContext' defined in URL [jar:file:/opt/opennms/lib/opennms-services-1.8.3.jar!/beanRefContext.xml]: Cannot resolve reference to bean 'daemonContext' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'daemonContext' defined in URL [jar:file:/opt/opennms/lib/opennms-daemon-1.9.0.jar!/beanRefContext.xml]: Cannot resolve reference to bean 'daoContext' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'daoContext' defined in URL [jar:file:/opt/opennms/lib/opennms-dao-1.8.3.jar!/beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [org.springframework.context.support.ClassPathXmlApplicationContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataCollectionConfig-init' defined in class path resource [META-INF/opennms/applicationContext-dao.xml]: Invocation of init method failed; nested exception is org.opennms.netmgt.dao.CastorDataAccessFailureException: Failed to marshal/unmarshal XML file while unmarshalling XML file for resource 'file [/opt/opennms/etc/datacollection-config.xml]': org.exolab.castor.xml.MarshalException: unable to find FieldDescriptor for 'include-collection' in ClassDescriptor of snmp-collection{File: file:/opt/opennms/etc/datacollection-config.xml; line: 12; column: 53}; nested exception is org.exolab.castor.xml.MarshalException: unable to find FieldDescriptor for 'include-collection' in ClassDescriptor of snmp-collection{File: file:/opt/opennms/etc/datacollection-config.xml; line: 12; column: 53}

I'm pulling my hair out here. Please advise as it's our management tool. I haven't made any changes to postgresql, no iptables running:

[root@ranlxnms01 /]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

 [root@ranlxnms01 /]# ps aux | grep post
postgres 24780  0.0  0.0 130884  5500 ?        S    Aug25   0:00 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data
postgres 24782  0.0  0.0 109904   760 ?        S    Aug25   0:00 postgres: logger process
postgres 24784  0.0  0.0 131000  1276 ?        S    Aug25   0:00 postgres: writer process
postgres 24785  0.0  0.0 110904  1704 ?        S    Aug25   0:00 postgres: stats buffer process
postgres 24786  0.0  0.0 110212   980 ?        S    Aug25   0:00 postgres: stats collector process
root     30466  0.0  0.0  61188   720 pts/1    S+   10:33   0:00 grep post

[root@ranlxnms01 /]# netstat -a | more
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State

tcp        0      0 *:postgres                  *:*                         LISTEN

All postgresql config as per the docs and it was working fine before the upgrade.

[root@ranlxnms01 /]# cat /proc/version
Linux version 2.6.18-194.11.1.el5 ([email protected]) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Tue Jul 27 05:45:06 EDT 2010

[root@ranlxnms01 /]# rpm -qa | grep java
java-1.5.0-sun-plugin-1.5.0.18-1jpp.1.el5
java-1.4.2-gcj-compat-src-1.4.2.0-40jpp.115
java-1.5.0-sun-jdbc-1.5.0.18-1jpp.1.el5
bsh-javadoc-1.3.0-9jpp.1
java-1.4.2-gcj-compat-devel-1.4.2.0-40jpp.115
java-1.5.0-sun-src-1.5.0.18-1jpp.1.el5
gcc-java-4.1.2-48.el5
java-1.5.0-sun-1.5.0.18-1jpp.1.el5
system-switch-java-1.1.2-2.el5_1
glib-java-0.2.6-3.fc6
java-1.5.0-sun-demo-1.5.0.18-1jpp.1.el5
java-1.4.2-gcj-compat-devel-1.4.2.0-40jpp.115
java-1.4.2-gcj-compat-1.4.2.0-40jpp.115
java-1.5.0-sun-devel-1.5.0.18-1jpp.1.el5
xmlrpc-javadoc-2.0.1-3jpp.1

[root@ranlxnms01 /]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
10.40.108.251   ranlxnms01.sesahs.nsw.gov.au    ranlxnms01

[root@ranlxnms01 /]# free -m
                                 total       used       free     shared    buffers     cached
Mem:                     32061       3123      28938          0        304       1793
-/+ buffers/cache:       1024      31036
Swap:                      8189          0       8189

All file permissions are root: no changes there either:
[root@ranlxnms01 /]# ls -l /opt/opennms
total 52
drwxr-xr-x 2 root root  4096 Aug 27 10:25 bin
drwxr-xr-x 8 root root  4096 Aug 27 10:25 contrib
drwxrwxr-x 8 root root 12288 Aug 27 10:26 etc
drwxr-xr-x 5 root root  4096 Aug 27 10:25 jetty-webapps
drwxr-xr-x 8 root root 24576 Aug 27 10:25 lib
lrwxrwxrwx 1 root root    16 Aug 27 10:25 logs -> /nms_lg/opennms/
drwxr-xr-x 6 root root  4096 Mar 23 14:38 share

And the same with sub folders, no problems with runjava or install:

[root@ranlxnms01 /]# /opt/opennms/bin/runjava -s
runjava: Looking for an appropriate JRE...
runjava: Checking for an appropriate JRE in JAVA_HOME...
runjava: skipping... JAVA_HOME not set
runjava: Checking JRE in user's path: "/usr/bin/java"...
runjava: found an appropriate JRE in user's path: "/usr/bin/java"
runjava: value of "/usr/bin/java" stored in configuration file

[root@ranlxnms01 /]# /opt/opennms/bin/install -dis
==============================================================================
OpenNMS Installer Version $Id$
==============================================================================

Configures PostgreSQL tables, users, and other miscellaneous settings.

- searching for jicmp:
  - trying to load /usr/lib/libjicmp.so: NO
  - trying to load /usr/lib64/libjicmp.so: OK
- searching for jrrd:
  - trying to load /usr/lib/libjrrd.so: OK
- using SQL directory... /opt/opennms/etc
- using create.sql... /opt/opennms/etc/create.sql
* using 'postgres' as the PostgreSQL user for OpenNMS
* using '' as the PostgreSQL password for OpenNMS
* using 'opennms' as the PostgreSQL database name for OpenNMS
- checking if database "opennms" is unicode... ALREADY UNICODE
- Checking for old import files in /opt/opennms/etc... DONE
- Migrating/creating database:
Aug 27, 2010 10:39:11 AM liquibase.database.template.JdbcTemplate comment
INFO: Lock Database
Aug 27, 2010 10:39:11 AM liquibase.lock.LockHandler acquireLock
INFO: Successfully acquired change log lock
Aug 27, 2010 10:39:12 AM liquibase.database.AbstractDatabase getRanChangeSetList
INFO: Reading from databasechangelog
Aug 27, 2010 10:39:19 AM liquibase.database.template.JdbcTemplate comment
INFO: Release Database Lock
Aug 27, 2010 10:39:19 AM liquibase.lock.LockHandler releaseLock
INFO: Successfully released change log lock
- checking if iplike is usable... YES
- checking for stale eventtime.so references... OK

And to repeat, no changes from 1.8.1 and it now completely crashes everytime.

Any ideas?




---------------------------------------------------------------------------------------------

SOUTH EASTERN SYDNEY AND ILLAWARRA AREA HEALTH SERVICE CONFIDENTIALITY NOTICE

This email, and the files transmitted with it, are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient, you are not permitted to distribute or use this email or any of its attachments in any way. We also request that you advise the sender of the incorrect addressing.

This email message has been virus-scanned. Although no computer viruses were detected, South Eastern Sydney and Illawarra Area Health Service accept no liability for any consequential damage resulting from email containing any computer viruses.

------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users 
worldwide. Take advantage of special opportunities to increase revenue and 
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d

_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-install mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-install