[jgroups-users] S3_PING: Unexpected SocketException, nodes have their details written, but no cluster is established

Hello,

I would like to share an unusual socket error that has been a puzzle for
us for couple of days now.

The setup is an Infinispan 2-node cluster in invalidation mode, on EC2
with S3_PING configured.

Each node is able to write its S3 bucket entries, but they cannot see
one another, and no cluster is formed. So each node becomes the
coordinator. The following exception was recorded in the logs.

The same S3 bucket has been previously used with identical policy,
credentials and other settings for S3_PING and it has worked OK.

Any suggestions what might be going on?

It appears that the error is occurring during the TLS cipher suite
negotiation when S3_PING is trying to get the bucket entries.

The nodes however succeed in writing to it.

Cheers,

Vladimir

Infinispan 8.2.4
JGroups 3.6.7
Java 1.8.0_45




{
  "timeMillis": 1476203863483,
  "thread": "Timer-3,localhost-59909",
  "level": "ERROR",
  "loggerName": "org.jgroups.protocols.S3_PING",
  "message": "JGRP000185: failed reading addresses",
  "thrown": {
    "commonElementCount": 0,
    "localizedMessage": "Broken pipe",
    "message": "Broken pipe",
    "name": "java.net.SocketException",
    "extendedStackTrace": [
      {
        "class": "java.net.SocketOutputStream",
        "method": "socketWrite0",
        "file": "SocketOutputStream.java",
        "line": -2,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "java.net.SocketOutputStream",
        "method": "socketWrite",
        "file": "SocketOutputStream.java",
        "line": 109,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "java.net.SocketOutputStream",
        "method": "write",
        "file": "SocketOutputStream.java",
        "line": 153,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.OutputRecord",
        "method": "writeBuffer",
        "file": "OutputRecord.java",
        "line": 431,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.OutputRecord",
        "method": "write",
        "file": "OutputRecord.java",
        "line": 417,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.SSLSocketImpl",
        "method": "writeRecordInternal",
        "file": "SSLSocketImpl.java",
        "line": 864,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.SSLSocketImpl",
        "method": "writeRecord",
        "file": "SSLSocketImpl.java",
        "line": 835,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.SSLSocketImpl",
        "method": "writeRecord",
        "file": "SSLSocketImpl.java",
        "line": 705,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.Handshaker",
        "method": "sendChangeCipherSpec",
        "file": "Handshaker.java",
        "line": 1077,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.ClientHandshaker",
        "method": "sendChangeCipherAndFinish",
        "file": "ClientHandshaker.java",
        "line": 1191,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.ClientHandshaker",
        "method": "serverHelloDone",
        "file": "ClientHandshaker.java",
        "line": 1103,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.ClientHandshaker",
        "method": "processMessage",
        "file": "ClientHandshaker.java",
        "line": 344,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.Handshaker",
        "method": "processLoop",
        "file": "Handshaker.java",
        "line": 979,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.Handshaker",
        "method": "process_record",
        "file": "Handshaker.java",
        "line": 914,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.SSLSocketImpl",
        "method": "readRecord",
        "file": "SSLSocketImpl.java",
        "line": 1050,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.SSLSocketImpl",
        "method": "performInitialHandshake",
        "file": "SSLSocketImpl.java",
        "line": 1363,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.SSLSocketImpl",
        "method": "startHandshake",
        "file": "SSLSocketImpl.java",
        "line": 1391,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.security.ssl.SSLSocketImpl",
        "method": "startHandshake",
        "file": "SSLSocketImpl.java",
        "line": 1375,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.net.www.protocol.https.HttpsClient",
        "method": "afterConnect",
        "file": "HttpsClient.java",
        "line": 563,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class":
"sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection",
        "method": "connect",
        "file": "AbstractDelegateHttpsURLConnection.java",
        "line": 185,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.net.www.protocol.http.HttpURLConnection",
        "method": "getInputStream0",
        "file": "HttpURLConnection.java",
        "line": 1512,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.net.www.protocol.http.HttpURLConnection",
        "method": "getInputStream",
        "file": "HttpURLConnection.java",
        "line": 1440,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "java.net.HttpURLConnection",
        "method": "getResponseCode",
        "file": "HttpURLConnection.java",
        "line": 480,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "sun.net.www.protocol.https.HttpsURLConnectionImpl",
        "method": "getResponseCode",
        "file": "HttpsURLConnectionImpl.java",
        "line": 338,
        "exact": false,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "org.jgroups.protocols.S3_PING$ListBucketResponse",
        "method": "<init>",
        "file": "S3_PING.java",
        "line": 1193,
        "exact": false,
        "location": "S3_PING$ListBucketResponse.class",
        "version": "3.6.7.Final"
      },
      {
        "class": "org.jgroups.protocols.S3_PING$AWSAuthConnection",
        "method": "listBucket",
        "file": "S3_PING.java",
        "line": 529,
        "exact": false,
        "location": "S3_PING$AWSAuthConnection.class",
        "version": "3.6.7.Final"
      },
      {
        "class": "org.jgroups.protocols.S3_PING$AWSAuthConnection",
        "method": "listBucket",
        "file": "S3_PING.java",
        "line": 510,
        "exact": false,
        "location": "S3_PING$AWSAuthConnection.class",
        "version": "3.6.7.Final"
      },
      {
        "class": "org.jgroups.protocols.S3_PING",
        "method": "readAll",
        "file": "S3_PING.java",
        "line": 126,
        "exact": true,
        "location": "S3_PING.class",
        "version": "3.6.7.Final"
      },
      {
        "class": "org.jgroups.protocols.FILE_PING",
        "method": "findMembers",
        "file": "FILE_PING.java",
        "line": 108,
        "exact": true,
        "location": "FILE_PING.class",
        "version": "3.6.7.Final"
      },
      {
        "class": "org.jgroups.protocols.Discovery",
        "method": "findMembers",
        "file": "Discovery.java",
        "line": 244,
        "exact": true,
        "location": "Discovery.class",
        "version": "3.6.7.Final"
      },
      {
        "class": "org.jgroups.protocols.Discovery",
        "method": "down",
        "file": "Discovery.java",
        "line": 388,
        "exact": true,
        "location": "Discovery.class",
        "version": "3.6.7.Final"
      },
      {
        "class": "org.jgroups.protocols.FILE_PING",
        "method": "down",
        "file": "FILE_PING.java",
        "line": 103,
        "exact": true,
        "location": "FILE_PING.class",
        "version": "3.6.7.Final"
      },
      {
        "class": "org.jgroups.protocols.MERGE3$InfoSender",
        "method": "run",
        "file": "MERGE3.java",
        "line": 381,
        "exact": true,
        "location": "MERGE3$InfoSender.class",
        "version": "3.6.7.Final"
      },
      {
        "class": "org.jgroups.util.TimeScheduler3$Task",
        "method": "run",
        "file": "TimeScheduler3.java",
        "line": 291,
        "exact": true,
        "location": "TimeScheduler3$Task.class",
        "version": "3.6.7.Final"
      },
      {
        "class": "org.jgroups.util.TimeScheduler3$RecurringTask",
        "method": "run",
        "file": "TimeScheduler3.java",
        "line": 325,
        "exact": true,
        "location": "TimeScheduler3$RecurringTask.class",
        "version": "3.6.7.Final"
      },
      {
        "class": "java.util.concurrent.ThreadPoolExecutor",
        "method": "runWorker",
        "file": "ThreadPoolExecutor.java",
        "line": 1142,
        "exact": true,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "java.util.concurrent.ThreadPoolExecutor$Worker",
        "method": "run",
        "file": "ThreadPoolExecutor.java",
        "line": 617,
        "exact": true,
        "location": "?",
        "version": "1.8.0_45"
      },
      {
        "class": "java.lang.Thread",
        "method": "run",
        "file": "Thread.java",
        "line": 745,
        "exact": true,
        "location": "?",
        "version": "1.8.0_45"
      }
    ]
  },
  "endOfBatch": false,
  "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
  "threadId": 20,
  "threadPriority": 5
}




-- 
Vladimir Dzhuvinov

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users