Re: [PATCH 1/1] tidbits: net-udp: solicit new client for server mode

Am 20.06.26 um 19:28 schrieb Philippe Gerum:
> Hannes Diethelm <[email protected]> writes:
> 
>> Am 01.06.26 um 11:10 schrieb Philippe Gerum:
>>> Philippe Gerum <[email protected]> writes:
>>>
>>>> Hannes Diethelm <[email protected]> writes:
>>>>
>>>>> This fixes random EINPROGRESS at init or during runtime.
>>>>>
>>>>> Also correct whitespaces and make stdout more consistent.
>>>>>
>>>>> Signed-off-by: Hannes Diethelm <[email protected]>
>>>>> ---
>>>>>    tidbits/oob-net-udp.c | 58 +++++++++++++++++++++++++++++++++++++------
>>>>>    1 file changed, 50 insertions(+), 8 deletions(-)
>>>>>
>>>>> diff --git a/tidbits/oob-net-udp.c b/tidbits/oob-net-udp.c
>>>>> index 11b8459..7af834f 100644
>>>>> --- a/tidbits/oob-net-udp.c
>>>>> +++ b/tidbits/oob-net-udp.c
>>>>> @@ -50,12 +50,12 @@ static void usage(void)
>>>>>    }
>>>>>      static void print_addr(char* text, struct sockaddr_in *addr){
>>>>> -    char ip_str[INET_ADDRSTRLEN+1];
>>>>> -    inet_ntop(AF_INET, &(addr->sin_addr), ip_str, sizeof(ip_str));
>>>>> -    evl_printf("%s--------\n", text);
>>>>> -    evl_printf("IP-Address: %s\n", ip_str);
>>>>> -    evl_printf("Port:       %d\n", ntohs(addr->sin_port));
>>>>> -    evl_printf("Family:     %d\n", addr->sin_family);
>>>>> +	char ip_str[INET_ADDRSTRLEN+1];
>>>>> +	inet_ntop(AF_INET, &(addr->sin_addr), ip_str, sizeof(ip_str));
>>>>> +	evl_printf("== %s\n", text);
>>>>> +	evl_printf("   ip-address: %s\n", ip_str);
>>>>> +	evl_printf("   port:       %d\n", ntohs(addr->sin_port));
>>>>> +	evl_printf("   family:     %d\n", addr->sin_family);
>>>>>    }
>>>>>      static void sender(int s, const char *text, int mcount,
>>>>> @@ -226,6 +226,8 @@ static void client(int s, const char *text, int mcount,
>>>>>    	free(tbuf);
>>>>>    }
>>>>>    +#define SERVER_ADDR_LIST_SIZE 64
>>>>> +
>>>>>    static void server(int s, const char *text, int mcount,
>>>>>    		struct sockaddr_in *addr, int iter)
>>>>>    {
>>>>> @@ -236,6 +238,9 @@ static void server(int s, const char *text, int mcount,
>>>>>    	ssize_t ret;
>>>>>    	char *tbuf;
>>>>>    	char rbuf[16384];
>>>>> +	in_addr_t addr_list[SERVER_ADDR_LIST_SIZE]={};
>>>>                                                     ^ missing whitespaces
>>>>> +	size_t addr_list_fill=0;
>>>>> +	bool solicit_done;
>>>>>      	tlen = (strlen(text) + 1) * mcount;
>>>>>    	tbuf = malloc(tlen);
>>>>> @@ -276,6 +281,39 @@ static void server(int s, const char *text, int mcount,
>>>>>    			evl_printf(" (TRUNCATED)");
>>>>>    		evl_printf(": %.*s\n", (int)ret, rbuf);
>>>>>    +		/*
>>>>> +		 * We need to call evl_net_solicit for each new
>>>>> +		 * client once before sending data. This will break
>>>>> +		 * realtime for the first response.
>>>>> +		 * If this is not done and the ARP address is not
>>>>> +		 * yet in cache or garbage-collected, oob_sendmsg
>>>>> +		 * will return EINPROGRESS on start or during runtime.
>>>>> +		 */
>>>>
>>>> We can happily send redundant solicit requests to the core, no need to
>>>> filter out cached addresses, evl_net_solicit() will do the right thing.
>>>>
>>> Except that doing so would always demote the caller to the in-band
>>> stage, which may not be what you want. The fact that we'd need to
>>> maintain a cache in apps in order to figure out whether solicitation
>>> should be done shows a shortcoming in the core, users should not have to
>>> do this dance.
>>> We should have an oob call for probing the route+arp caches with
>>> ipv4. I'll look into this asap.
>>>
>>
>> Yes, this is the reason I do it this way and print out a message if
>> this happens.
>>
>> I think server mode with multiple clients in real time is not something you
>> can just easily do without careful considerations what happens when multiple
>> clients send a message at the same time or TDMA behind. My intent in this example
>> is mostly to show that it is possible.
>>
>> A real application would need one of these options:
>> - A thread only for solicitation so the main thread doesn't get blocked
>> - evl_net_solicit() with a flag like MSG_DONTWAIT and then poll in the main loop if
>>    the client is ready
>> - Know the clients in advance
>> - A warm up phase during clients can connect
>> - ...
>>
>> However, a way for probing and reading the arp would shurely help.
> 
> Here is a general proposal to deal with the issue of sending UDP packets
> to unplanned destinations, which is basically what the server mode has
> to do, when receiving unsolicited messages.
> 
> We have a single invariant to cope with: the evl network stack wants to
> use the routing+arp information produced by the regular in-band stack,
> so that we can benefit from potentially complex routing logic without
> reinventing that wheel. Therefore, the route to any new/unsolicited
> destination must be resolved by the in-band stack once so that we can
> send UDP packets to that peer from the oob stage next.
> 
>  From the point above, and as you pointed out already, an oob sender must
> either plan for reaching a particular destination (e.g. soliciting it
> prior to entering time-critical code), or accept an initial delay for
> the route to be resolved if the destination is not yet known on first
> transmit (i.e. out of the oob caches).
> 
> So, I would say that evl could meet the requirements you stated above by
> implementing the following:
> 
> - Honor MSG_PROBE for oob_sendmsg(), so that only the general call
>    sanity and route resolution to the destination host is performed when
>    set in the request flags, without actually sending any data. On
>    success of such call, we would know that the routing information is
>    readily available from the oob caches, no offload to in-band would
>    have happened if we had not given this flag. The absence of routing
>    information to the destination from some oob cache would yield a
>    specific error, so that the caller may decide what to do next.

It seams the flag MSG_PROBE is kernel only? I did not find any occurrence
in /usr/include or in libevl.

> 
> - Extend the effect of receiving MSG_DONTWAIT (and more generally
>    O_NONBLOCK on fildes) to what we would do upon missing routing
>    information: if present, return with a specific error code _without_
>    relaying the packet to the in-band stack. The caller may then decide
>    to handle the case locally. Otherwise, proceed as usual (i.e. relay to
>    the in-band stack, then notify the caller with -EINPROGRESS).

So the oob-net-udp server code can be changed to use MSG_DONTWAIT and
if the return value is EWOULDBLOCK, call evl_net_solicit() and try again instead
of holding a list of IP's right?

> 
> - Provide a way to synchronize with the route resolution process in evl,
>    i.e. a syscall that would block until a given destination is available
>    from the oob cache. This call already exists, evl_net_solicit() can be
>    used for that purpose (if a resolution request is already in flight,
>    subsequent ones to the same destination won't cause any harm).
> 
> Points 1 and 2 are implemented in [1]. Feedback greatly appreciated and
> important when you have time. Usability for real-world applications is
> key.

I am a bit busy right now, but I would like to test it. However, It can
take a week or two.

> 
>> It would also be nice to have an "evl net" command to list the
>> entry's. I tried the normal arp command but it doesn't behave nicely
>> with evl.
> 
> Could you please elaborate on this issue? Currently, updates to the
> in-band cache are propagated to the oob cache (by registering a hook
> into the relevant notifier chain in the kernel). So I would expect all
> destinations of interest to oob which are visible from the in-band arp
> cache to be available from the evl map as well.
> 

The issue was on my side, I need to run "arp -n" so it does not resolve host
names. Otherwise, it takes quite some time. But this makes sense, with OOB
active, it will never receive the answer on the request, so it times out.

By the way: Since some time, all incoming network traffic goes to the OOB
stage when it is enabled. However, all outgoing traffic from the in-band stage
is still sent. Would it make sense to block all outgoing traffic after
"evl net -ei" except might be ARP requests if they are initialized by evl_net_solicit()?

At the moment, this is done using nftables for linuxcnc, this works fine
for preemt_rt and xenomai, so not really needed.

> This said, I agree that we need a way to inspect the oob cache
> specifically. Working on it.
> 
> [1] https://gitlab.com/Xenomai/xenomai4/linux-evl/-/tree/wip/net-solicit?ref_type=heads
>