Re: Separate sockets for separate connections

(Sorry for not responding to this sooner, I was attending NixCon 2025)

On Fri, Aug 29, 2025, at 8:42 PM, Xin Long wrote:
> 
> I wrote some code to explore this:
> https://github.com/lxin/quic/pull/53
> 
> - A stream can be peeled off from a parent/connection socket using
>   getsockopt(QUIC_SOCKOPT_STREAM_PEELOFF) with a stream_id, similar to
>   SCTP's connection peeloff.
> 
> - For stream sockets, in addition to send(), recv(), and close(), support
>   for poll() and shutdown() is also implemented. Note that close() and
>   shutdown() send a FIN on the sending side, issue a STOP_SENDING on the
>   receiving side, or both for bidirectional streams, as applicable.
> 
> There's also a sample test:
> https://github.com/lxin/quic/blob/stream-peeloff/tests/peeloff_test.c
> 
> - Sender: Opens a stream with getsockopt(QUIC_SOCKOPT_STREAM_OPEN), peels
>   it off with getsockopt(QUIC_SOCKOPT_STREAM_PEELOFF), and sends data via
>   the new file descriptor.
> 
> - Receiver: Detects stream creation through a QUIC_EVENT_STREAM_UPDATE
>   event, peels off the stream with getsockopt(QUIC_SOCKOPT_STREAM_PEELOFF),
>   and receives data via the new file descriptor.

Fantastic! Thank you for investigating this; I really appreciate it.

> Please check out the stream socket interfaces in the PR above and comment
> on anything that you think could be improved.

Gladly! I looked at the linked PR and left some comment now, but they are only "mild questions". The basic design here looks like exactly like what I was hoping for. Hooray!

> I aimed to make a peeled-off stream socket fully independent to keep the
> design simple. However, since the connection socket may close at any time,
> the stream socket must hold a reference to it. To keep the relationship
> strictly one-way, the connection socket remains unaware of any peeled-off
> stream sockets.

That makes sense to me. Maybe someday someone will want "please close connection when the last stream socket is closed", but that can come later. (If I understand what `sock_hold` is doing with ref counts correctly, I'd hope that would be possible without any bidirectional references.)

> Right, providing it in libquic won't work for kernel consumers.

Just to be clear, I was thinking of one user-land process sending the stream socket to another. But yes, kernel consumer would face the exact same issues as other userspace processes without this.

> > For what it's worth, I would go further in fact and say that this
> > "stream peeloff" system call should not just be supported by QUIC, too.
> > It is very nice today how many code can be agnostic to TCP vs unix
> > domain sockets, for example. I would ideally want the same thing to be
> > true with QUIC too, via an "extended unix domain socket" that would
> > replicate the QUIC state machine(s) just as regular unix domain sockets
> > replicate the TCP state machine.
> >
> > I bring up such an "extended unix domain socket" not to indulge in scope
> > creep, but just to point out that a good litmus test for a new socket
> > interface is that multiple domains could meaningfully support it, and
> > that litmus test is met in this case.
> >
> That’s a good point.

Glad you like it! I had been thinking abstractly at that point. But putting on my hat as a Nix developer, yes 

> I originally based the QUIC stream API on SCTP’s model, but it now seems that
> most real-world use cases are closer to applications that traditionally
> relied on TCP rather than SCTP.

I don't really know enough about who is using SCTP to know for sure, but yes I think that it is right :).

Cheers,

John