Discovery fails with TimeOutException

Thomas Brettinger <[email protected]>
Newsgroups gmane.network.opennms.general
Message-ID <[email protected]> 
Hi,

I hope this is not  an FAQ, I did some research and didn’t find anything that seems to match my issue.

I must admit that I’m new to opennms and just did a fresh install recently. 

The issue that I am currently struggling with is that discovery fails and I can’t nail down what the actual issue is. 
discovery.log tells me:

2018-05-02 23:16:48,925 ERROR [Camel (rpcClient) thread #8 - JmsReplyManagerOnTimeout[OpenNMS.A36.RPC.PING-SWEEP]] o.o.n.d.DiscoveryTaskExecutorImpl: An error occurred while processing job 12 of 13 at location A36 (on task #1). No newSuspect events will be generated.
java.util.concurrent.CompletionException: org.opennms.core.rpc.api.RequestTimedOutException: org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 30200 millis due reply message with correlationID: Camel-ID-nms-38215-1525295396929-0-203 not received on destination: temp-queue://ID:nms-37491-1525295388483-4:3:1 <temp-queue://ID:nms-37491-1525295388483-4:3:1>. Exchange[ID-nms-38215-1525295396929-0-202]
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_171]
	at org.opennms.core.rpc.camel.CamelRpcClientFactory$1$1.onFailure(CamelRpcClientFactory.java:107) ~[org.opennms.core.ipc.rpc.camel-21.1.0.jar:?]
	at org.apache.camel.impl.DefaultProducerTemplate.lambda$createCompletionFunction$0(DefaultProducerTemplate.java:488) ~[camel-core-2.19.1.jar:2.19.1]
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[?:1.8.0_171]
	at org.apache.camel.impl.AsyncCallbackToCompletableFutureAdapter.done(AsyncCallbackToCompletableFutureAdapter.java:53) ~[camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.impl.ProducerCache.lambda$doInAsyncProducer$2(ProducerCache.java:457) ~[camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.impl.EventNotifierCallback.done(EventNotifierCallback.java:47) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:258) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:258) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:147) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:258) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.RedeliveryErrorHandler$2.done(RedeliveryErrorHandler.java:553) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.management.InstrumentationProcessor$1.done(InstrumentationProcessor.java:86) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:147) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.MulticastProcessor.doDone(MulticastProcessor.java:925) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.MulticastProcessor$2.done(MulticastProcessor.java:802) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:258) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.RedeliveryErrorHandler.deliverToFailureProcessor(RedeliveryErrorHandler.java:1086) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.RedeliveryErrorHandler.processAsyncErrorHandler(RedeliveryErrorHandler.java:655) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.RedeliveryErrorHandler$2.done(RedeliveryErrorHandler.java:559) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.component.jms.reply.ReplyManagerSupport.processReply(ReplyManagerSupport.java:210) [camel-jms-2.19.1.jar:2.19.1]
	at org.apache.camel.component.jms.reply.TemporaryQueueReplyHandler.onTimeout(TemporaryQueueReplyHandler.java:62) [camel-jms-2.19.1.jar:2.19.1]
	at org.apache.camel.component.jms.reply.CorrelationTimeoutMap$1.run(CorrelationTimeoutMap.java:60) [camel-jms-2.19.1.jar:2.19.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: org.opennms.core.rpc.api.RequestTimedOutException: org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 30200 millis due reply message with correlationID: Camel-ID-nms-38215-1525295396929-0-203 not received on destination: temp-queue://ID:nms-37491-1525295388483-4:3:1 <temp-queue://ID:nms-37491-1525295388483-4:3:1>. Exchange[ID-nms-38215-1525295396929-0-202]
	... 30 more
Caused by: org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 30200 millis due reply message with correlationID: Camel-ID-nms-38215-1525295396929-0-203 not received on destination: temp-queue://ID:nms-37491-1525295388483-4:3:1 <temp-queue://ID:nms-37491-1525295388483-4:3:1>. Exchange[ID-nms-38215-1525295396929-0-202]
	at org.apache.camel.component.jms.reply.ReplyManagerSupport.processReply(ReplyManagerSupport.java:170) ~[camel-jms-2.19.1.jar:2.19.1]
	... 7 more
2018-05-02 23:16:58,925 ERROR [Camel (rpcClient) thread #8 - JmsReplyManagerOnTimeout[OpenNMS.A36.RPC.PING-SWEEP]] o.o.n.d.DiscoveryTaskExecutorImpl: An error occurred while processing job 13 of 13 at location A36 (on task #1). No newSuspect events will be generated.
java.util.concurrent.CompletionException: org.opennms.core.rpc.api.RequestTimedOutException: org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 9060 millis due reply message with correlationID: Camel-ID-nms-38215-1525295396929-0-218 not received on destination: temp-queue://ID:nms-37491-1525295388483-4:3:1 <temp-queue://ID:nms-37491-1525295388483-4:3:1>. Exchange[ID-nms-38215-1525295396929-0-217]
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_171]
	at org.opennms.core.rpc.camel.CamelRpcClientFactory$1$1.onFailure(CamelRpcClientFactory.java:107) ~[org.opennms.core.ipc.rpc.camel-21.1.0.jar:?]
	at org.apache.camel.impl.DefaultProducerTemplate.lambda$createCompletionFunction$0(DefaultProducerTemplate.java:488) ~[camel-core-2.19.1.jar:2.19.1]
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_171]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[?:1.8.0_171]
	at org.apache.camel.impl.AsyncCallbackToCompletableFutureAdapter.done(AsyncCallbackToCompletableFutureAdapter.java:53) ~[camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.impl.ProducerCache.lambda$doInAsyncProducer$2(ProducerCache.java:457) ~[camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.impl.EventNotifierCallback.done(EventNotifierCallback.java:47) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:258) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:258) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:147) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:258) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.RedeliveryErrorHandler$2.done(RedeliveryErrorHandler.java:553) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.management.InstrumentationProcessor$1.done(InstrumentationProcessor.java:86) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:147) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.MulticastProcessor.doDone(MulticastProcessor.java:925) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.MulticastProcessor$2.done(MulticastProcessor.java:802) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:258) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.RedeliveryErrorHandler.deliverToFailureProcessor(RedeliveryErrorHandler.java:1086) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.RedeliveryErrorHandler.processAsyncErrorHandler(RedeliveryErrorHandler.java:655) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.processor.RedeliveryErrorHandler$2.done(RedeliveryErrorHandler.java:559) [camel-core-2.19.1.jar:2.19.1]
	at org.apache.camel.component.jms.reply.ReplyManagerSupport.processReply(ReplyManagerSupport.java:210) [camel-jms-2.19.1.jar:2.19.1]
	at org.apache.camel.component.jms.reply.TemporaryQueueReplyHandler.onTimeout(TemporaryQueueReplyHandler.java:62) [camel-jms-2.19.1.jar:2.19.1]
	at org.apache.camel.component.jms.reply.CorrelationTimeoutMap$1.run(CorrelationTimeoutMap.java:60) [camel-jms-2.19.1.jar:2.19.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: org.opennms.core.rpc.api.RequestTimedOutException: org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 9060 millis due reply message with correlationID: Camel-ID-nms-38215-1525295396929-0-218 not received on destination: temp-queue://ID:nms-37491-1525295388483-4:3:1 <temp-queue://ID:nms-37491-1525295388483-4:3:1>. Exchange[ID-nms-38215-1525295396929-0-217]

However, the opennms host can indeed reach the network in question, and ping:sweep produces results as well (ping:sweep —pps 100 <begin> <end> almost instantly produces a list of responses). 

I suspected that the default of 1 pps combined with a larger number of IPs to scan might be the issue, and played with batch size and even edited the xml config to change the pps value - but it seems that this is not the problem, as I still get the above error (and nothing gets discovered).

Any hints where to look, moral support or any other useful input is welcome… ;)

Kind regards
Thomas

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.