Re: Intermittent TLS inbound failure behind HAProxy, Kamailio opens new TCP to HAProxy ephemeral port and gets RST

Hi again,

Following up with additional findings and a proposed fix.
I tested with tcp_connection_match=1. The alias overwrites and deletions still occur, and connections are still lost.
After reviewing the source code, this is expected. tcp_connection_match=1 only affects the lookup path in tcp_send() and msg_send_buffer(). It does not change how aliases are stored or how collisions are handled during alias creation.

The root of the problem is in _tcpconn_add_alias_unsafe() in tcp_main.c. 
When a new connection is added via tcpconn_add(), it creates 3 aliases at increasing specificity:
Alias 0: (peer_ip, peer_port, 0.0.0.0, 0) - wildcard, find any connection to this peer
Alias 1: (peer_ip, peer_port, local_ip, 0) - find connection to this peer from this local IP
Alias 2: (peer_ip, peer_port, local_ip, local_port) - fully specified
Aliases 0 and 1 are created with l_port=0. 
The hash function tcp_addr_hash() includes l_port in the XOR computation, so when two connections share the same peer IP:port but differ in local port (e.g., 5061 vs 50443), their aliases 0 and 1 hash to the same bucket (because l_port=0 for both), while alias 2 hashes to different buckets.

>From my logs:
SIP conn 5:    tcpconn_add(): hashes: 2726:2132:2998, 5     (local port 5061)
Health check:  tcpconn_add(): hashes: 2726:2132:3285, 384   (local port 50443)

Buckets 2726 and 2132 are shared. Bucket 2998 vs 3285 differ (alias 2).
The collision detection in _tcpconn_add_alias_unsafe() is:
if((a->parent->state != S_CONN_BAD) && (port == a->port)
        && ((l_port == 0) || (l_port == a->parent->rcv.dst_port))
        && (ip_addr_cmp(&c->rcv.src_ip, &a->parent->rcv.src_ip))
        && (is_local_ip_any
                || ip_addr_cmp(&a->parent->rcv.dst_ip, l_ip))) {

The condition (l_port == 0) acts as a wildcard. it matches any existing connection regardless of destination port. 
Combined with the default new_conn_alias_flags = TCP_ALIAS_REPLACE, the health check connection's wildcard aliases replace the SIP connection's wildcard aliases. When the health check closes, the aliases are deleted, and the SIP connection becomes permanently unreachable by peer address.

Proposed fix
When tcp_connection_match=1 (strict matching), tighten the collision detection in _tcpconn_add_alias_unsafe() so that two connections from the same peer but to different local ports are not considered collisions:
if((a->parent->state != S_CONN_BAD) && (port == a->port)
        && ((l_port == 0) || (l_port == a->parent->rcv.dst_port))
        && (ip_addr_cmp(&c->rcv.src_ip, &a->parent->rcv.src_ip))
        && (is_local_ip_any
                || ip_addr_cmp(&a->parent->rcv.dst_ip, l_ip))
        && (tcp_connection_match != TCPCONN_MATCH_STRICT
                || c->rcv.dst_port == 0
                || a->parent->rcv.dst_port == 0
                || c->rcv.dst_port == a->parent->rcv.dst_port)) {

The added condition:
When tcp_connection_match=0 (default): no behavior change: the new condition is always true
When tcp_connection_match=1: a collision is only detected if both connections have the same destination port (or either has dst_port=0 as a safety guard)
This means the health check connection (dst_port=50443) would no longer replace the SIP connection's aliases (dst_port=5061). Both sets of aliases coexist in the same hash bucket. 
When the health check closes, only its own aliases are removed. The SIP connection's aliases remain intact.
The change is ~3 lines in _tcpconn_add_alias_unsafe() in tcp_main.c, guarded behind the existing tcp_connection_match=1 opt-in, so default behavior is completely unchanged.

I am happy to submit a PR if this approach makes sense.

Any feedback or comments would be appreciated.

Thanks,
Joey.
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions -- [email protected]
To unsubscribe send an email to [email protected]
Important: keep the mailing list in the recipients, do not reply only to the sender!