Re: OTP socket.erl, unexpected interaction when receiving from errqueue

The VM team has now planned a task to extend the NIF API to handle
select/poll on error i.e enif_select_error() and ERL_NIF_SELECT_ERROR.

When that is done we can use that in the socket API, probably exactly as
you first suggested.

A question is if we should just add that, and all code using
the socket API has to be prepared for a new message, or if we need
an option on the 'otp' protocol level.

The Experimental status of the socket API allows for such a
backwards-incompatible change, but that does not mean that
we need to do one...

Until then I have just merged, into 'master', an optimization of
recv with Timeout =:= 0 so it skips the select/cancel select dance.

Cheers
/ Raimo


On Wed, Nov 25, 2020 at 02:45:22PM +0100, Raimo Niskanen wrote:
> On Wed, Nov 25, 2020 at 01:42:49PM +0100, Andreas Schultz wrote:
> > Am Mi., 25. Nov. 2020 um 11:37 Uhr schrieb Raimo Niskanen <
> > [email protected]>:
> > 
> > > Is it so that recvmsg(fd, &msg, MSG_ERRQUEUE) only receives from
> > > the error queue, and never any regular data?
> > >
> > 
> > That is my understanding from the man page. Experiments also confirm this.
> 
> The man page is a not entirely unambigous.  But I found a stackoverflow
> thread that also confirms this (also referring to experiments).
> 
>     https://stackoverflow.com/questions/17326913/linux-udp-socket-recvmsg-with-msg-errqueue
> 
> > 
> > Reading the errqueue is actually quite hard to test. The behavior for local
> > errors and remote errors (e.g. reception of ICMP errors) is sometimes
> > different.
> > Small sample:
> > 
> > 1> {ok, Socket} = socket:open(inet, dgram, udp).
> > {ok,{'$socket',#Ref<0.1577644963.140640257.162512>}}
> > 2> ok = socket:setopt(Socket, ip, recverr, true).
> > ok
> > 3> Dest = #{family => inet, addr => {127,0,0,1}, port => 1234}.
> > #{addr => {127,0,0,1},family => inet,port => 1234}
> > 4> socket:sendto(Socket, <<"Data">>, Dest, nowait).
> > ok
> > 5> socket:sendto(Socket, <<"Data">>, Dest, nowait).
> > {error,{econnrefused,[<<"Data">>]}}
> 
> Weird to get econnrefused from sendto() on an unconnected dgram socket.
> 
> > 
> > 
> > The first sendto returns `ok`, the error can be read in subsequent recvmsg.
> > The second sendto returns the error immediately because the kernel has
> > learned that the local endpoint does not exist.
> > 
> > 
> > 
> > > Today there is no VM support for a NIF do differ between POLLIN
> > > and POLLERR.  I have asked the VM guys to have a look at that.
> > >
> > > Without that you can as your response to receiving
> > > {'$socket',Socket,select,SelectHandle}
> > > call socket:recvmsg(Socket, [errqueue], 0), to poll, and then
> > > if the poll gave {error,timeout} call socket:recvmsg(Socket, 0, nowait).
> > >
> > 
> > If was actually thinking about doing `socket:recvmsg(Socket, [errqueue],
> > nowait)`
> > if
> >    a) just received a select message and
> >    b) socket:recvfrom returned a new select info instead of reading any data
> > 
> > That should capture the situation where only data in the errqueue is
> > present without having to
> > use the 0 timeout.
> 
> Might there not be a possibility to starve out error messages in the face of
> continously incoming data with this approach?
> 
> > 
> > Then we will have to optimize Timeout =:= 0, and maybe introduce
> > > Timeout =:= poll with a nicer return value for no data.
> > >
> > 
> > I like the general idea, but the `nowait` option now feels a  bit wrong. We
> > end up with `nowait == do a select` and `poll == just check if there is
> > data`.
> > It might be too late for changing `nowait`, but what about adding `select`
> > as an alias to `nowait` ?
> 
> I think it is too late to change 'nowait' now, and it is not entirely off.
> It states that the call will not wait, but not what happens instead.
> While 'select' states what to do, but only sometimes.
> 
> 'poll' also states what to do.  We could add 'select' as an alias for
> 'nowait', some might find it more appropriate.
> 
> We thought about having only polling operations and a dedicated select
> operation, but that would need minimum two NIF calls also when handling
> much data, so we decided on doing automatic select depending on the
> result of the operation.
> 
> / Raimo
> 
> 
> > 
> > Regards,
> > Andreas
> > 
> > Timeout =:= 0 today causes quite some overhead, but it should work.
> > >
> > > Cheers
> > > / Raimo
> > >
> > >
> > > On Mon, Nov 23, 2020 at 05:42:23PM +0100, Andreas Schultz wrote:
> > > > Hi,
> > > >
> > > > If setup a socket with:
> > > >
> > > >     socket:setopt(Socket, ip, recverr, true)
> > > >
> > > > If then started a asynchronous recvmsg with:
> > > >
> > > >     {select, SelectInfo} = socket:recvfrom(Socket, 0, [], nowait)
> > > >
> > > > When now something arrives in the error queue, I'll get a select info
> > > > message with:
> > > >
> > > >     {'$socket', Socket, select, SelectInfo}
> > > >
> > > > The problem is, nothing in there tells me to read from the error queue.
> > > The
> > > > underlying OS poll/epoll call would have this information, but it is lost
> > > > in the Erlang message.
> > > >
> > > > When I now try to read from the socket with:
> > > >
> > > >    socket:recvfrom(Socket, 0, [], nowait)
> > > >
> > > > all I get is another `{select, SelectInfo}` tuple, followed by another
> > > > `{'$socket', Socket, select, SelectInfo}` messages.
> > > > This can actually end up in an endless busy loop.
> > > >
> > > > To actually clear this I would to do a:
> > > >
> > > >     socket:recvmsg(Socket, [errqueue], nowait)
> > > >
> > > > On an POSIX socket, I would have to actually poll for POLLIN | POLLERR to
> > > > get a similar behavior. But the return of the poll would tell me whether
> > > it
> > > > was POLLIN or POLLERR (similar for epoll).
> > > > For the Erlang API it might sense to always poll for both conditions, but
> > > > we then should get an indication what exactly it was.
> > > >
> > > > Would it be possible to change the $socket message to something like:
> > > >
> > > >     {'$socket', Socket, error, SelectInfo}
> > > >
> > > > for POLLERR/EPOLLERR ???
> > > >
> > > > Regards
> > > > Andreas
> > > > --
> > > >
> > > > Andreas Schultz
> > >
> > > --
> > >
> > > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> > >
> > 
> > 
> > -- 
> > 
> > Andreas Schultz
> 
> -- 
> 
> / Raimo Niskanen, Erlang/OTP, Ericsson AB

-- 

/ Raimo Niskanen, Erlang/OTP, Ericsson AB