Re: options for high-throughput connections between nodes

--0000000000001185da05db9586c7
Content-Type: text/plain; charset="UTF-8"

Osiris is quite a different type of thing to the other ones in your list in
that it will always first write terms to disk and only then replicate them
(over TCP). That said it could do a decent replication job if you want a
local buffer to decouple the production of terms from the replication part.
Osiris does still need a dist erl connection for coordination messages and
you'd have to modify the quorum commit semantics to fit your use case (e.g.
a "leader" member on the production side and a "replica" member on the
other side).

Cheers
Karl

On Fri, 1 Apr 2022 at 11:01, Dave Cottlehuber <[email protected]> wrote:

> I'm investigating how to maximise pushing mostly loosely ordered erlang
> terms (via `term_to_binary/1` and friends) between a mesh of erlang nodes.
>
> A single TCP stream isn't sufficient, so I'm expecting using either UDP or
> multiple TCP streams, to push up throughput. I don't expect the usual
> erlang distribution protocol to be suitable, because of the risk of
> congestion and head-of-line blocking of distribution-related traffic.
>
> Assume LFN scenario (Long Fat Pipes), up to 3-10GiB/s bandwith, usual
> inter-continental latency as default. No numbers yet, just "as much as
> possible / closest to theoretical maximum bandwidth".
>
> - zeromq: https://github.com/zeromq/chumak |
> https://github.com/zeromq/erlzmq2 | https://github.com/lukaszsamson/erlzmq
> - osiris: https://github.com/rabbitmq/osiris
> - nanomsg: https://github.com/basho/enm
> - https://www.erlang.org/doc/man/gen_sctp.html
>
> Has anybody other suggestions to add to the list?
>
> thanks
> Dave
>


-- 
*Karl Nilsson*

--0000000000001185da05db9586c7
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Osiris is quite a different type of thing to the other one=
s in your list in that it will always first write terms to disk and only th=
en replicate them (over TCP). That said it could do a decent replication jo=
b if you want a local buffer to decouple the production of terms from the r=
eplication part. Osiris does still need a dist erl connection for coordinat=
ion messages and you&#39;d have to modify the quorum commit semantics to fi=
t your use case (e.g. a &quot;leader&quot; member on the production side an=
d a &quot;replica&quot; member on the other side).<div><br></div><div>Cheer=
s</div><div>Karl</div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr"=
 class=3D"gmail_attr">On Fri, 1 Apr 2022 at 11:01, Dave Cottlehuber &lt;<a =
href=3D"mailto:[email protected]">[email protected]</a>&gt; wrote:<br></div=
><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border=
-left:1px solid rgb(204,204,204);padding-left:1ex">I&#39;m investigating ho=
w to maximise pushing mostly loosely ordered erlang terms (via `term_to_bin=
ary/1` and friends) between a mesh of erlang nodes.<br>
<br>
A single TCP stream isn&#39;t sufficient, so I&#39;m expecting using either=
 UDP or multiple TCP streams, to push up throughput. I don&#39;t expect the=
 usual erlang distribution protocol to be suitable, because of the risk of =
congestion and head-of-line blocking of distribution-related traffic.<br>
<br>
Assume LFN scenario (Long Fat Pipes), up to 3-10GiB/s bandwith, usual inter=
-continental latency as default. No numbers yet, just &quot;as much as poss=
ible / closest to theoretical maximum bandwidth&quot;. <br>
<br>
- zeromq: <a href=3D"https://github.com/zeromq/chumak" rel=3D"noreferrer" t=
arget=3D"_blank">https://github.com/zeromq/chumak</a> | <a href=3D"https://=
github.com/zeromq/erlzmq2" rel=3D"noreferrer" target=3D"_blank">https://git=
hub.com/zeromq/erlzmq2</a> | <a href=3D"https://github.com/lukaszsamson/erl=
zmq" rel=3D"noreferrer" target=3D"_blank">https://github.com/lukaszsamson/e=
rlzmq</a><br>
- osiris: <a href=3D"https://github.com/rabbitmq/osiris" rel=3D"noreferrer"=
 target=3D"_blank">https://github.com/rabbitmq/osiris</a><br>
- nanomsg: <a href=3D"https://github.com/basho/enm" rel=3D"noreferrer" targ=
et=3D"_blank">https://github.com/basho/enm</a><br>
- <a href=3D"https://www.erlang.org/doc/man/gen_sctp.html" rel=3D"noreferre=
r" target=3D"_blank">https://www.erlang.org/doc/man/gen_sctp.html</a><br>
<br>
Has anybody other suggestions to add to the list?<br>
<br>
thanks<br>
Dave<br>
</blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"=
 class=3D"gmail_signature"><div dir=3D"ltr"><div><b>Karl Nilsson</b></div><=
/div></div>

--0000000000001185da05db9586c7--