Using different cipher suites in a BCFIPS JDK, triggers RSA key-reuse

We are using BCFIPS as the com.sun.net.ssl.internal.ssl.Provider
During SSL client / server handshake, our internal components choose TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA as cipher suite when talking to our HTTP-server.
An AWS ELB then comes along and selects TLS_RSA_WITH_AES_256_CBC_SHA as cipher suite. This results in the server failing during handshake with the following message:

org.bouncycastle.crypto.IllegalKeyException: Attempt to encrypt/decrypt with RSA modulus already used for sign/verify

If the healthcheck calls on our HTTP-server before our internal components, then it's the internal components that trigger a key reuse error on the server.

I have identified which cipher suites are incompatible with trial and error but really don't understand what is causing the issue, other than assuming that some cipher suites use the RSA key for encrypt / decrypt, while others are using it for signing / verifying.
Our way forward will be to restrict cipher suite uses in the server with all SSL handshakes we perform. I would love to set this restriction on JDK level though, to make sure that no new code "poisons" the JVM with the wrong cipher suite.
Does anyone else have any insights in this problem? Can this be locked down on JDK level (i.e. java.security / some kind of JSSE provider)?
Thanks
  Erik