Re: HAProxy and proxy protocol support

On 2024-06-13 01:01, Sam Varshavchik wrote:
> Nathan Phillip Brink writes:
>
>>> https://netty.io/wiki/tcp-fast-open.html
>>>
>>> "TFO cannot always be used, however, because it changes how TCP 
>>> behave: The receiving end might see this data duplicated due to 
>>> retransmission of the SYN packet. For this reason, TFO must only be 
>>> used when the data in the initial packet will be processed 
>>> idempotently. Whether this is the case, depends on the protocol and 
>>> the application."
>>>
>>>> That seems like a catastrophic bug in the TCP contract. I'd like to 
>>>> make some colleagues aware of the issue, ideally with a reference 
>>>> to some analysis of it.
>>>
>>> This seems like a well known, known issue with this.
>>
>> Doesn’t this just mean that a single connection could be seen by the 
>> server as two connections? As long as the initial data isn’t enough 
>> to actually enqueue an email, one of those ghost connections should 
>> just timeout and the other real connection actually process.
>
> I considered this interpretation while trying to wrap my own brain 
> around this. I don't believe this is the case, here's why.
>
> A TCP connection is defined by: source ip, source port, destination 
> ip, and destination port. My understanding of TCP is that it is 
> logically impossible for two logical connections exist at the same 
> time that have identical source/destination ip/ports. There is no 
> unique identifier, of some kind, that's assigned to each logical 
> connection. This quadruplet is what defines a unique TCP connection.
Thanks, this is what I was missing.
>
> So, a duplicated packet can't possibly create a new logical connection 
> with the same source and destination. It already exists. The only 
> possible interpretation of the above is that the payload gets 
> duplicated on a single connection.
>
>> I also apologize for any misunderstanding of SMTP as I am not very 
>> familiar with it. I would like some confirmation if I am interpreting 
>> the concern correctly. I do not think it was mentioned explicitly 
>> (unless I missed it). Is the concern the possibility of 
>> double-queuing a single email as I purported above?
>
> The concern is that the server expects and receives the first PROXY 
> statement, and sets things up accordingly. Now the SMTP session starts.
>
> The server then logically receives another PROXY command, as logical 
> data that's read from the socket, but the server is now talking SMTP. 
> And the sender did not actually send it. The sender sent the initial 
> SMTP command (I'm ignoring the deal with the initial SMTP banner, and 
> how it works, it's immaterial here). The sender is waiting for a 
> response to the initial command, which would be EHLO.
>
> But the server has logically read the PROXY command, that's logically 
> read from the socket, before the EHLO. That's not a valid SMTP 
> command, so the server sends an error. Which the client reads as the 
> response to the EHLO command.
>
> Stick a fork in this connection. It's done.
>
> The same deal applies to IMAP, POP3, or anything else for that matter 
> that works similarly.

Regardless of my lack of understanding of TCP, I am convinced that 
section 2.1 of RFC 7413 standard exhaustively describes the possible 
outcomes of SYN retransmission when TFO is turned on:

    a) the receiver host receives data in a duplicate SYN after it has
    forgotten it received the original SYN (e.g., due to a reboot);

    b) the duplicate is received after the connection created by the
    original SYN has been closed and the close was initiated by the
    sender (so the receiver will not be protected by the TIME-WAIT 2 MSL
    state).

Both of those outcomes are creation of an extra connection. It is not a 
concurrent connection. And both of those scenarios are only possible if 
the connection was forgotten or closed (and the time wait/linger had 
expired). If the connection is still open, a retransmitted SYN with data 
will be ignored just like any other SYN; the data will not be resent to 
the application. But if there isn’t a matching open connection, the 
retransmitted SYN will result in a new connection being created. The 
only thing that can affect the application is any data included in the 
initial SYN. I.e., this might cause a double-queue if an entire valid 
sequence which would enqueue a mailpiece were included in the initial SYN.

To be clear, this will *never* result in data being sent twice *within 
the same connection*. Such a thing would result in corruption for most 
application layer protocols—especially binary ones. What you are 
describing as a concern is not something that can (as best I understand) 
happen.

This is why the text in https://netty.io/wiki/tcp-fast-open.html 
mentions idempotency but not complete protocol corruption. As long as 
the data included in the initial SYN does not have side-effects, or the 
application layer uses cookies, or the application layer does its own 
handshake (netty mentions the TLS handshake), there is no concern. When 
experiencing this issue, you might just have one or two random spurious 
connections which end up timing out. If we had to worry about literal 
doubling of bytes, that would completely break every guarantee that TCP 
is providing the application layer. You might receive “PROXY «something» 
EHLPROXY «something»EHLO «server name»”, not “PROXY «something» PROXY 
«something» EHLO «server name»”. You can only guard against that type of 
thing if you somehow know *how many bytes* are actually included in the 
initial SYN and might be replayed, which is not something that I expect 
the socket APIs to provide to you (because you shouldn’t need to know 
that—you’re using a stream abstraction over a lower level 
packet-oriented layer).

> I'm adding code that will completely ignore any PROXY command that the 
> SMTP, IMAP, or the POP3 server appears to receive, initially. 
> Completely ignore it, and don't send any response. This should put 
> things back on the right track, or, more technically accurate: prevent 
> it from getting derailed in the first place.

Unless my understanding is wrong, such code would be unreachable. Also, 
doing this correctly would require you to know how many bytes are 
included in the SYN. If the sender is actually taking advantage of TFO, 
the repeated data would include the EHLO or other greeting command. It 
might even include part of a MAIL or RCPT command. So I advise against 
adding an implementation of PROXY in this way.

Thanks for taking the time to consider my thoughts and respond!

_______________________________________________
courier-users mailing list
[email protected]
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users