Re: Help with external socket handling and xcb

On Wed, 8 Sep 2021 17:58:39 +0200
Uli Schlachter <[email protected]> wrote:

> Hi,

Hi! Great to see the man himself answer me :)
 
> Am 08.09.21 um 15:29 schrieb Carlo Wood:
> [...]
> > Unfortunately, not many libraries that provide protocol handling
> > provide such an API - most of them insist on also handling the
> > socket side of things; and as far as I can tell xcb is like that
> > too.  
> 
> This might be a bit off-topic, but how would such a library be
> thread-safe? xcb does quite a dance internally to provide a
> thread-safe API and hide all these implementation details.

I agree that this is off-topic, so bare with me, or skip ahead
to the real stuff (in the next email) if you don't have the time please.

What would be perfect is a library with the following API.

To set the stage right, we are talking about a single stream
of data (albeit bidirectional); one file descriptor. Having multiple
connections to the (X) server, and thus multiple fds is entirely
parallel and would not interfere with this.

Hence, what is needed is support a stream of bytes from the server
to the library, and from the library to the server. Also these two
streams are essentially in parallel and do not interfere, but they
are quite different.

Getting the data from the server to the library is easy: the application
(this is the requirement: not the library) monitors the fd for input
events and whenever the socket becomes readable, it reads bytes from
it into a buffer and then passes that buffer to the library.

[Even more off-topic, the way I normally do this is by means of
a protocol-aware class that can scan the incoming data in order to
know if it contains a decodable message; for example a protocol
that uses lines terminated by a new-line character requires a
search for a '\n' in newly read data; and only once a complete,
decodable message has been received it passes this to the decoder
(the library) without making a copy: it just passes the pointer
to the beginning and the length. If such a complete message is
not contiguous then indeed a copy has to be made; but that should
happen REALLY rarely if at all because the size of the blocks of
my buffer adjusts itself to be so large that normally a message
easily fits in one block, and as soon as all data has been decoded
and the buffer is empty, the write pointer is reset to the start of
the buffer (avoiding that we ever reach the end of the first block
of the buffer under normal circumstances)].

Allow me to point to an example that decodes html:

https://github.com/CarloWood/evio/blob/master/protocol/http.cxx#L13

This function returns 0 when there is nothing to decode yet,
otherwise it returns the number of bytes of the last received
data that is still part of the decodable message.
The complete message can be larger (if this function returned
zero before). The application (that is, this "evio" git submodule)
then makes the message contiguous if needed (which, again, is
very seldom the case, if at all) and pass it to the decoder,
in this case at line 31 in the above link - which only decodes
the header lines and then switches to a new decoder based on
the received Content-Type; but it could of course call a function
of the (xcb) library.

If finding end-of-messages is not feasible for some reason, then of course
the library would have to accept all data as passed to end_of_msg_finder
and then without copying that that would become a lot harder; aka,
the library would probably just have to copy it (short from providing
the end_of_message_finder function itself).

Note that only one thread (at a time) will be doing any of:
1) reading the socket, 2) calling end_of_msg_finder, 3) making sure it
is contiguous and 4) passing a complete message to the library.
Hence only one thread at a time calls the
incoming-message-decode-function
at a time for any given fd. 

My application only needs to call xcb_poll_for_event (in the 'idle'
loop that is - if I had one), so API requirements for the library would be:

1) provide a function like:
   void incoming_data(char const* buf, size_t len);
   OR
   int end_of_message_finder(char const* buf, size_t len) and
   bool decode_message(char const*, size_t len). Which can
   return false (or true) if the library needs to connection
   to be closed (ie, due to an unrecoverable parse error).

2) Provide a function that allows the application to set a callback
   function that will be called (from within `decode_message`)
   when an event has been received (passing the data that
   now is returned by xcb_poll_for_event).

The stream in the opposite direction, writing to the socket, is
even easier if the library really doesn't care about sockets:
when it has more data that it wants to write to the server, just
call a callback function (provided by the application) with the
data. Of course, the application then has to copy this data to
its own outgoing buffer. If you want to avoid that copy, then
you'll need more specialized output code that matches how evio
works. That is using the API of the class Dev2Buf
(https://github.com/CarloWood/evio/blob/master/StreamBuf.h#L1198)

In other words, the application should be able to set callback
functions for the following:

  // Return the number of bytes that can be written directly into memory at position dev2buf_ptr() at this moment.
  size_t dev2buf_contiguous();

  // Same as above, but doesn't return 0 unless out of memory or buffer full.
  size_t dev2buf_contiguous_forced();

  // Get pointer to put area (where the library can write the number of bytes to as returned by the above function).
  char* dev2buf_ptr();

  // After writing n bytes with the above function, call this one.
  void dev2buf_bump(int n);

  // A convenience function that copies data (ie, to write/append string literals).
  size_t sputn(char const* s, size_t const n);

All of the above functions may only be called by one thread at a time
(which I call the "producer" thread).

The normal procedure is that the producer thread first calls dev2buf_contiguous
and if that return > 0, write that amount of data and repeat (if there is
still more to write). Or, if it returns 0, call dev2buf_contiguous_forced
and do the same thing; if that returns 0 as well (should normally never 
happen) then we're out of luck (more API is needed to recover from that, but
lets not get into that now).

So, every time it knows how much it can write (> 0) then call dev2buf_ptr()
to get the area where it can write to, and write it. And then call
dev2buf_bump with the total amount written.

> For example, xcb_get_input_focus_reply() blocks until the reply was
> received from the X11 server (and does so in a way that does not block
> other threads from using the X11 connection). Sure, you can use
> xcb_poll_for_reply() to check whether the reply was received yet, but
> going down this route results in quite spaghetti-y code.

I probably should have read this before typing all of the above...

But ok, the way my application works is quite unique (at least, I invented
it myself over the past 20 years). I say application all the time, in order
to keep the word library reserved for the xcb library, but really it's
more of a (C++) library. It would be way way off topic to get into
20 years of design, but the short version is that everything is a "task"
and that tasks are designed to wait for things, and each other, without
EVER blocking a thread. No blocking. Ever.

So, having done all that effort it is a thorn in my eye to link with a library
that forces me to have some thread block/sleep all the time :p.

In the ideal case I'd write a task for the functionality of
xcb_get_input_focus_reply. The task would work as follows: you create it
(it can already be created, just reuse an object). Initialize it with
the parameters that xcb_get_input_focus_reply takes, and then 'run' the
task. When the waiting for the server reply is over the task "finishes"
which you can hook up in many ways to pick up from there (ie, a callback
function, or waking up another task). Note that a "waiting task" doesn't
mean a thread is sleeping or blocking. It means that the task object is
not using any CPU.

To make that work with the xcb library, xcb_get_input_focus_reply should
be non-blocking (and not return anything). While whenever it sees the
reply from the X server, call a callback function with the required
information.

Since this mail has become rather large, I will reply to the rest
in a separate mail.

Thanks,
Carlo