options for high-throughput connections between nodes
"Dave Cottlehuber" <[email protected]> Fri, 01 Apr 2022 10:01:01 +0000
| Newsgroups | gmane.comp.lang.erlang.general |
|---|---|
| Message-ID | <[email protected]> |
I'm investigating how to maximise pushing mostly loosely ordered erlang terms (via `term_to_binary/1` and friends) between a mesh of erlang nodes. A single TCP stream isn't sufficient, so I'm expecting using either UDP or multiple TCP streams, to push up throughput. I don't expect the usual erlang distribution protocol to be suitable, because of the risk of congestion and head-of-line blocking of distribution-related traffic. Assume LFN scenario (Long Fat Pipes), up to 3-10GiB/s bandwith, usual inter-continental latency as default. No numbers yet, just "as much as possible / closest to theoretical maximum bandwidth". - zeromq: https://github.com/zeromq/chumak | https://github.com/zeromq/erlzmq2 | https://github.com/lukaszsamson/erlzmq - osiris: https://github.com/rabbitmq/osiris - nanomsg: https://github.com/basho/enm - https://www.erlang.org/doc/man/gen_sctp.html Has anybody other suggestions to add to the list? thanks Dave