bugreport: spread-5 crashing on conf reload, if number of daemons changed, because of virtual ID mess

--===============8446576305718422632==
Content-Type: multipart/alternative; boundary="0000000000009a0e430571e851e0"

--0000000000009a0e430571e851e0
Content-Type: text/plain; charset="UTF-8"

Hi John, hi all,

if an already running daemon is triggered to reload the spread.conf at
runtime, it can crash if the number of daemons has changed. There can be a
confusion in the internal VirtualID table. Apparently spread fetches the
wrong auto generated virtual id, if some daemon is removed/added in the
middle of the table at runtime.

Following steps lead to the problem:
- spread version 5.0.1
- No VirtualIDs are configured by us. VirtualIDs are auto-generated.
- All daemons are running.
- One Segment with one daemon is removed in the middle of spread.conf.
- The spread.conf is distributed to all hosts.
- spread is triggered to reload spread.conf by spmonitor r
- some spread daemon will abort with bad failure shown below. It is the
daemon behind the removed one in the list.

Here a conflict of virtual IDs is logged:
2018-07-20 16:12:27 GMT Auto-generated virtual ID = '1696126475' for daemon
'host-06a'
2018-07-20 16:12:27 GMT The virtual ID '1696126475' of 'host-06a' is
already in use by 'host-52'!  You will probably need to explicitly
reconfigure the daemons' virtual IDs so that they don't conflict.

One of the spread daemons complaining after reload:
2018-07-25 14:43:04 GMT Hash value for this configuration is: 4055467701
2018-07-25 14:43:04 GMT Finished configuration file.
2018-07-25 14:43:04 GMT Conf_load_conf_file: My name: host-05a, id:
3524380600, addr: 1.2.3.5, port: 9876
2018-07-25 14:43:04 GMT Conf_reload_initiate: daemon identity mapped to two
different old daemons: name ' host-06a' -> 0x7f57eccd8800, id '2876338096'
-> 0x7f57eccd8208! Partitioning to singleton!
2018-07-25 14:43:04 GMT Conf_reload_initiate: Return need_singleton = 1

One other spread daemon crashing after reload because of virtual ID chaos:
2018-07-25 14:43:04 GMT Hash value for this configuration is: 4055467701
2018-07-25 14:43:04 GMT Finished configuration file.
2018-07-25 14:43:04 GMT Conf_load_conf_file: My name: host-06a, id:
2876338096, addr: 1.2.3.7, port: 9876
2018-07-25 14:43:04 GMT Conf_reload_initiate: My daemon parameters have
changed! Exiting!
2018-07-25 14:43:04 GMT     Old: name 'host-06a', addr [1.2.3.7]:7654, id
'1696126475', num_ifs 1
2018-07-25 14:43:04 GMT     New: name 'host-06a', addr [1.2.3.7]:7654, id
'2876338096', num_ifs 1
Exit caused by Alarm!

Workaround:
- Configure a unique VirtualID in spread.conf explicitly for each daemon is
a solution, because this configuration is always reloaded including
configured virtualID.

Maybe the virtualID table in memory has to be cleared before the parser is
loading the spread.conf a second time.
The auto-generation of virtualID is done only if the found virtualID is
zero. This is not true for a reload.

No, I think this is not a problem of the internal hash algorithm delivering
ambiguous hashes.

Currently we have no fix for that bug, because the workaround is good
enough for us.

Best regards,
Martin

--0000000000009a0e430571e851e0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi John, hi all,<br></div><div><br></div><div>if an a=
lready running daemon is triggered to reload the spread.conf at runtime, it=
 can crash if the number of daemons has changed. There can be a confusion i=
n the internal VirtualID table. Apparently spread fetches the wrong auto ge=
nerated virtual id, if some daemon is removed/added in the middle of the ta=
ble at runtime. <br></div><div><br></div><div>Following steps lead to the p=
roblem:</div><div>- spread version 5.0.1<br></div><div>- No VirtualIDs are =
configured by us. VirtualIDs are auto-generated.<br></div><div>- All daemon=
s are running.</div><div>- One Segment with one daemon is removed in the mi=
ddle of spread.conf.</div><div>- The spread.conf is distributed to all host=
s.</div><div>- spread is triggered to reload spread.conf by spmonitor r<br>=
</div><div>- some spread daemon will abort with bad failure shown below. It=
 is the daemon behind the removed one in the list.<br></div><div><br></div>=
<div></div><div>Here a conflict of virtual IDs is logged:<br></div><div>201=
8-07-20 16:12:27 GMT Auto-generated virtual ID =3D &#39;1696126475&#39; for=
 daemon &#39;host-06a&#39;<br>2018-07-20 16:12:27 GMT The virtual ID &#39;1=
696126475&#39; of &#39;host-06a&#39; is already in use by &#39;host-52&#39;=
!=C2=A0 You will probably need to explicitly reconfigure the daemons&#39; v=
irtual IDs so that they don&#39;t conflict.<br><br>One of the spread daemon=
s complaining after reload:<br>2018-07-25 14:43:04 GMT Hash value for this =
configuration is: 4055467701<br>2018-07-25 14:43:04 GMT Finished configurat=
ion file.<br>2018-07-25 14:43:04 GMT Conf_load_conf_file: My name:=20
host-05a, id: 3524380600, addr: 1.2.3.5, port: 9876<br>2018-07-25 14:43:04 =
GMT Conf_reload_initiate: daemon identity mapped to two different old daemo=
ns: name &#39;
host-06a&#39; -&gt; 0x7f57eccd8800, id &#39;2876338096&#39; -&gt; 0x7f57ecc=
d8208! Partitioning to singleton!<br>2018-07-25 14:43:04 GMT Conf_reload_in=
itiate: Return need_singleton =3D 1<br><br>One other spread daemon crashing=
 after reload because of virtual ID chaos:<br>2018-07-25 14:43:04 GMT Hash =
value for this configuration is: 4055467701<br>2018-07-25 14:43:04 GMT Fini=
shed configuration file.<br>2018-07-25 14:43:04 GMT Conf_load_conf_file: My=
 name:=20
host-06a, id: 2876338096, addr: 1.2.3.7, port: 9876<br>2018-07-25 14:43:04 =
GMT Conf_reload_initiate: My daemon parameters have changed! Exiting!<br>20=
18-07-25 14:43:04 GMT =C2=A0=C2=A0=C2=A0 Old: name &#39;host-06a&#39;, addr=
 [1.2.3.7]:7654, id &#39;1696126475&#39;, num_ifs 1<br>2018-07-25 14:43:04 =
GMT =C2=A0=C2=A0=C2=A0 New: name &#39;host-06a&#39;, addr [1.2.3.7]:7654, i=
d &#39;2876338096&#39;, num_ifs 1<br>Exit caused by Alarm!<br></div><div><b=
r></div><div>Workaround:</div><div>- Configure a unique VirtualID in spread=
.conf explicitly for each daemon is a solution, because this configuration =
is always reloaded including configured virtualID.</div><div><br></div><div=
>Maybe the virtualID table in memory has to be cleared before the parser is=
 loading the spread.conf a second time.</div><div>The auto-generation of vi=
rtualID is done only if the found virtualID is zero. This is not true for a=
 reload.</div><div><br></div><div>No, I think this is not a problem of the =
internal hash algorithm delivering ambiguous hashes.<br></div><div><br></di=
v><div>Currently we have no fix for that bug, because the workaround is goo=
d enough for us.</div><div><br></div><div>Best regards,</div><div>Martin</d=
iv><div><br></div></div>

--0000000000009a0e430571e851e0--


--===============8446576305718422632==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Spread-users mailing list
[email protected]
http://lists.spread.org/mailman/listinfo/spread-users

--===============8446576305718422632==--