Re: Re: A more general question about curl

Martin Schanzenbach <[email protected]> Sun, 11 Sep 2022 18:20:40 +0000
Newsgroups	gmane.network.gnunet.devel
Message-ID	<1662920160-sup-9443@werkbank>
Hi,

I already talked to Tim a year ago about this.
Unfortunately since GNU was not participating in GSoC this did not get
any traction. Neither me or Tim have (had?) the bandwidth to do this.

But in terms of requirements what we need first is what curl multi API
offers _including_ the curl_multi_fdset functionality that exposes the
file descriptors so we can use our own scheduler.

This would allow us to replace curl everywhere except the GNS proxy.

The _secondary_ requirement is that we can access the SSL backend and
metadata on the TLS handshake: https://git.gnunet.org/gnunet.git/tree/src/gns/gnunet-gns-proxy.c#n957

BR
Martin

Excerpts from hyazinthe's message of 2022-09-11 18:36:11 +0200:
> Hello everyone,
> 
> 
> wget2 community is hearing our conversation and is reaching out to us for collaboration:
> 
> story - short:
> > On 10. Sep 2022, at 20:53, Tim Rühsen <[email protected]> wrote:
> >
> > just found this issue: https://gitlab.com/gnuwget/wget2/-/issues/550
> >
> > Feel free to add the details needed for GNUnet.
> 
> story - medium:
> > On 10. Sep 2022, at 20:47, Tim Rühsen <[email protected]> wrote:
> >
> > TL;DR the wget2 project is willing to help and a multi API may be easy
> > to implement. But someone has to write down the exact needs.
> 
> Full story, its rest - https://lists.gnu.org/archive/html/wget-dev/2022-09/msg00008.html :
> > On 10. Sep 2022, at 20:53, Tim Rühsen <[email protected]> wrote:
> > I try to explain what libwget does today and that it seems very straight forward to implement an API - and by the way - everybody is invited to do that :-)
> >
> > Libwget has several layers of abstraction of accessing the network stack(s).
> >
> > So you have indeed the synchronously, super simplified
> >   response = wget_http_get(...) like in example/http_get.c
> >
> > Then there is async HTTP API layer with more control over the details, see example/http_get2.c.
> >
> >   err = wget_http_open(&conn, url); // comes back immediately
> >   err = wget_http_send_request(conn, req); // comes back immediately
> >
> > resp = wget_http_get_response(conn); // waits until error/timeout or response
> >
> >   wget_http_close(&conn);
> >
> > While you could send several requests over a single connection, HTTP/1.1 has issues with it. But it works fine with HTTP/2. In this case wget_http_get_response(conn) can be called in a loop, returning the finished response in the order they came in. You can also have as many open connections as you like - but what is missing, if I understood correctly, is an API that fetches the responses from more than one connection at once, like
> >
> >   resp = wget_http_get_response(conn1, conn1, ..., NULL);
> >
> > Now there is also a TCP+SSL API (used by the above mentioned high level functions). This API is works asynchronously. It is like
> >
> >   tcp = wget_tcp_init()
> >   ... set all kind of configurations to the 'tcp' handle ...
> >   err = int wget_tcp_connect(tcp, host, port) // returns immediately
> >   wget_tcp_write() // returns after timeout or immediately if no timeout
> >   wget_tcp_read() // returns after timeout or immediately if no timeout
> >   wget_tcp_close()
> >   wget_tcp_deinit()
> >
> > Internally, this API uses select/poll, but just uses a single 'tcp' handle.
> >
> > Now, what a "multi"  API basically would look like is e.g.
> >
> > a function wget_tcp_select(array of tcp handles, timeout) which can be called in a loop and which returns an array of "ready" tcp handles (ready for write or read, configurable per tcp handle).
> >
> >
> > For me it looks like this is straight forward to implement (depends on the details / requirements).
> >
> >
> > Internally, wget_ready_2_transfer(int fd, int timeout, int mode) in
> > io.c just needs a companion function that takes a list/array of fds.
> >
> > If there really is interest from the GNUnet community, why not open an issue at https://gitlab.com/gnuwget/wget2/issues to discuss the details and the needs. Once we agree upon the details, the implementation can be done by anyone - whoever likes to pick it up.
> 
> 
> Greetings,
> Bastian Schmidt
signature.asc (application/pgp-signature, 833 B)