Re: Connecting spread daemons with different segment configurations

> We just tried to define more than 20 segments in the spread configuration=
, but this is not valid due to the following ...

I think you meant:

#define         MAX_SEGMENTS             20

Line 66 of daemon/spread_params.h

> First of all, it seems a bit sloppy to me, that there is no "#ifndef MAX_=
PROC_NAME", so that one can define this macro variable during configuration=
 time with "./configure".

It was never intended for MAX_SEGMENTS to be so easily a reconfigurable con=
stant because of the following:

> On the other hand I want to know, why have you chosen such a small number?

Three main reasons.  =


First, it was expected that in LAN type environments that support broadcast=
 and multicast that this network capability would be exploited by putting a=
ll such daemons in the same segment, reducing the need for so many segments=
.  So, multiple segments was really meant to support WAN type deployments a=
nd not degenerate LAN deployments.  Getting a membership algorithm to conve=
rge across a WAN with more and more segment representatives in each-to-all =
communication requires exponentially more communication globally and typica=
lly requires ever less sensitive timeouts to allow convergence without memb=
ership =93flapping=94 causing needless membership reformations.

Second, the way the membership algorithm currently is implemented, all memb=
ership information for a potential ring needs to fit within a single Spread=
 packet, which is typically 1472 bytes minus overhead*.  Space is used on t=
he token for each segment and in the worst case there could be an entry for=
 each segment and every daemons=92 ID.  So there is a space constraint on h=
ow many of these things can be supported without reworking this mechanism. =
 Modifying this and similar membership constants without being aware of suc=
h constraints is dangerous and likely to fail.

Finally, we try to discourage people from using 1 daemon per segment config=
urations unless it is absolutely required because of the inefficient commun=
ication where each daemon needs to send each data message individually to e=
ach segment leader.

* - This was true before Spread 4.4.  In Spread 4.4, we allow the most prob=
lematic membership packets to go up to 64KiB by leaning on IP de/fragmentat=
ion.  If your membership info exceeds a single MTU then a warning will be l=
ogged and any underlying network loss will be magnified for these packets, =
but in low loss environments it may well go unnoticed.

> In you manual you describe a tested configuration of 60 daemons (which is=
 totally OK for us).

Yes, we have seen even larger deployments than that.  However, all such dep=
loyments, so far as I know, exploit the multi-daemon per segment (i.e. - ne=
twork multicast or broadcast) functionality for both performance and capabi=
lity reasons (i.e. - the above constraints).

> Are there any big disadvantages of huge segment setups and what happens i=
f I increase the number of daemons?

That depends on what you mean by =93huge segment setups.=94  =


If you mean one daemon per segment and many such segments, then, yes, there=
 are big disadvantages to such deployments versus multiple-daemon per segme=
nt configurations.  The main problems, which I alluded to above are the com=
munication costs, and membership stability (in particular on membership for=
mation).

On the other hand, adding more daemons to a segment has very low cost as th=
e main effect is simply to extend the control token ring by one more member=
.  Even so, token ring protocols don=92t scale up well to huge numbers of p=
articipants and trying to push a Spread configuration towards or even beyon=
d 100 daemons may make the system prone to daemon membership churn, even in=
 a LAN environment, without tailoring the membership timeouts appropriately.

> My first guess would be, that the communication overhead is increasing ex=
ponentially, due to the fact that the spread daemons always try to test the=
ir communication partners.

During membership formation, yes, globally that communication scales expone=
ntially with the number of segments.  Once a ring is formed, data communica=
tion (cpu overhead) scales linearly with the number of segments while the p=
rimary control communication scales linearly with the number of alive daemo=
ns per token ring cycle.

> So is it safe to just increase the "MAX_PROC_NAME" variable?

It may well work up to some point depending on how exactly many daemons and=
 segments you try to use and how many daemons are active at any given time,=
 up until you trip over the packet space constraint that I discussed above =
(pre Spread 4.4.) or underlying network loss gets magnified too much (Sprea=
d 4.4.).

Cheers!

-----
John Lane Schultz
Spread Concepts LLC
Cell: 443 838 2200

On Jan 29, 2015, at 10:21 AM, Timo Korthals <[email protected]=
.de> wrote:

Dear spread users,

we are trying to setup a huge (< 60) spread daemon network.
We just tried to define more than 20 segments in the spread =

configuration, but this is not valid due to the following line in =

"include/sp.h":
#define         MAX_PROC_NAME           20 /* largest possible size of =

process name of daemon */

First of all, it seems a bit sloppy to me, that there is no "#ifndef =

MAX_PROC_NAME", so that one can define this macro variable during =

configuration time with "./configure".
On the other hand I want to know, why have you chosen such a small number?
In you manual you describe a tested configuration of 60 daemons (which =

is totally OK for us).
Are there any big disadvantages of huge segment setups and what happens =

if I increase the number of daemons?

My first guess would be, that the communication overhead is increasing =

exponentially, due to the fact that the spread daemons always try to =

test their communication partners.
In fact, for our setup we just want to send a really small amount of =

data, and only want the advantage of the connectability.

So is it safe to just increase the "MAX_PROC_NAME" variable?

Kind regards,
Timo

Timo Korthals, M.Sc.
Universit=E4t Bielefeld
AG Kognitronik & Sensorik
Exzellenzcluster Cognitive Interaction Technology (CITEC)
Inspiration 1 (Zehlendorfer Damm 199)
33619 Bielefeld - Germany

Office  : 3.037
Phone   : +49 521 106-67368
eMail   : [email protected]
Internet: http://www.ks.cit-ec.uni-bielefeld.de/


_______________________________________________
Spread-users mailing list
[email protected]
http://lists.spread.org/mailman/listinfo/spread-users