Re: bugreport: spread-5 crashing on conf reload, if number of daemons changed, because of virtual ID mess
Martin Schu <[email protected]> Wed, 1 Aug 2018 18:01:58 +0200
| Newsgroups | gmane.network.spread.user |
|---|---|
| Message-ID | <CAOSQKUXuhgtYcZ0fjA_==2n4yYncEd6Txzpocqeq6j2TBR9ELg@mail.gmail.com> |
--===============5515342163410090705==
Content-Type: multipart/alternative; boundary="00000000000080fead057261d091"
--00000000000080fead057261d091
Content-Type: text/plain; charset="UTF-8"
Hi John,
below is our old/new spread.conf. The host-53 is removed. Afterwards a
reload (spmonitor r) is triggered. Afterwards the spread daemon at host-06a
will crash with following log. This might be because host-06a takes
virtualid of deleted host-53, instead of autogenerating it again on reload.
Thanks,
Martin
2018-07-25 14:43:04 GMT Hash value for this configuration is: 4055467701
2018-07-25 14:43:04 GMT Finished configuration file.
2018-07-25 14:43:04 GMT Conf_load_conf_file: My name: host-06a, id:
2876338096, addr: 1.2.3.7, port: 9876
2018-07-25 14:43:04 GMT Conf_reload_initiate: My daemon parameters have
changed! Exiting!
2018-07-25 14:43:04 GMT Old: name 'host-06a', addr [1.2.3.7]:9876 , id
'1696126475', num_ifs 1
2018-07-25 14:43:04 GMT New: name 'host-06a', addr [1.2.3.7]:9876 , id
'2876338096', num_ifs 1
Exit caused by Alarm!
###################################################
# Section: Spread_Segments (old version)
###################################################
Spread_Segment 1.2.3.5:9876 {
host-05a 1.2.3.5 { 1.2.3.5 }
}
Spread_Segment 1.2.2.1:9876 {
host-51 1.2.2.1 { 1.2.2.1 }
}
Spread_Segment 1.2.2.2:9876 {
host-52 1.2.2.2 { 1.2.2.2 }
}
Spread_Segment 1.2.2.3:9876 {
host-53 1.2.2.3 { 1.2.2.3 }
}
Spread_Segment 1.2.3.7:9876 {
host-06a 1.2.3.7 { 1.2.3.7 }
}
###################################################
# Section: Spread_Segments (new version)
###################################################
Spread_Segment 1.2.3.5:9876 {
host-05a 1.2.3.5 { 1.2.3.5 }
}
Spread_Segment 1.2.2.1:9876 {
host-51 1.2.2.1 { 1.2.2.1 }
}
Spread_Segment 1.2.2.2:9876 {
host-52 1.2.2.2 { 1.2.2.2 }
}
##Spread_Segment 1.2.2.3:9876 {
## host-53 1.2.2.3 { 1.2.2.3 }
##}
Spread_Segment 1.2.3.7:9876 {
host-06a 1.2.3.7 { 1.2.3.7 }
}
On Wed, Aug 1, 2018 at 5:30 PM, John Lane Schultz <
[email protected]> wrote:
> Hi Martin,
>
> Would you please send me the two configuration files (old and new) that
> caused this issue, so I can more concretely understand your incident?
>
> Thanks,
> John
>
> On Jul 26, 2018, at 11:04 AM, Martin Schu <[email protected]>
> wrote:
>
> Hi John, hi all,
>
> if an already running daemon is triggered to reload the spread.conf at
> runtime, it can crash if the number of daemons has changed. There can be a
> confusion in the internal VirtualID table. Apparently spread fetches the
> wrong auto generated virtual id, if some daemon is removed/added in the
> middle of the table at runtime.
>
> Following steps lead to the problem:
> - spread version 5.0.1
> - No VirtualIDs are configured by us. VirtualIDs are auto-generated.
> - All daemons are running.
> - One Segment with one daemon is removed in the middle of spread.conf.
> - The spread.conf is distributed to all hosts.
> - spread is triggered to reload spread.conf by spmonitor r
> - some spread daemon will abort with bad failure shown below. It is the
> daemon behind the removed one in the list.
>
> Here a conflict of virtual IDs is logged:
> 2018-07-20 16:12:27 GMT Auto-generated virtual ID = '1696126475' for
> daemon 'host-06a'
> 2018-07-20 16:12:27 GMT The virtual ID '1696126475' of 'host-06a' is
> already in use by 'host-52'! You will probably need to explicitly
> reconfigure the daemons' virtual IDs so that they don't conflict.
>
> One of the spread daemons complaining after reload:
> 2018-07-25 14:43:04 GMT Hash value for this configuration is: 4055467701
> 2018-07-25 14:43:04 GMT Finished configuration file.
> 2018-07-25 14:43:04 GMT Conf_load_conf_file: My name: host-05a, id:
> 3524380600, addr: 1.2.3.5, port: 9876
> 2018-07-25 14:43:04 GMT Conf_reload_initiate: daemon identity mapped to
> two different old daemons: name ' host-06a' -> 0x7f57eccd8800, id
> '2876338096' -> 0x7f57eccd8208! Partitioning to singleton!
> 2018-07-25 14:43:04 GMT Conf_reload_initiate: Return need_singleton = 1
>
> One other spread daemon crashing after reload because of virtual ID chaos:
> 2018-07-25 14:43:04 GMT Hash value for this configuration is: 4055467701
> 2018-07-25 14:43:04 GMT Finished configuration file.
> 2018-07-25 14:43:04 GMT Conf_load_conf_file: My name: host-06a, id:
> 2876338096, addr: 1.2.3.7, port: 9876
> 2018-07-25 14:43:04 GMT Conf_reload_initiate: My daemon parameters have
> changed! Exiting!
> 2018-07-25 14:43:04 GMT Old: name 'host-06a', addr [1.2.3.7]:9876 , id
> '1696126475', num_ifs 1
> 2018-07-25 14:43:04 GMT New: name 'host-06a', addr [1.2.3.7]:9876 , id
> '2876338096', num_ifs 1
> Exit caused by Alarm!
>
> Workaround:
> - Configure a unique VirtualID in spread.conf explicitly for each daemon
> is a solution, because this configuration is always reloaded including
> configured virtualID.
>
> Maybe the virtualID table in memory has to be cleared before the parser is
> loading the spread.conf a second time.
> The auto-generation of virtualID is done only if the found virtualID is
> zero. This is not true for a reload.
>
> No, I think this is not a problem of the internal hash algorithm
> delivering ambiguous hashes.
>
> Currently we have no fix for that bug, because the workaround is good
> enough for us.
>
> Best regards,
> Martin
>
> _______________________________________________
> Spread-users mailing list
> [email protected]
> http://lists.spread.org/mailman/listinfo/spread-users
>
>
--00000000000080fead057261d091
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div>Hi John,</div><div><br></div><div>below is our old/ne=
w spread.conf. The host-53 is removed. Afterwards a reload (spmonitor r) is=
triggered. Afterwards the spread daemon at host-06a will crash with follow=
ing log. This might be because host-06a takes virtualid of deleted host-53,=
instead of autogenerating it again on reload.<br></div><div><br></div><div=
>Thanks,<br></div><div>Martin<br></div><div><br></div><div>
2018-07-25 14:43:04 GMT Hash value for this configuration is: 4055467701<br=
>
2018-07-25 14:43:04 GMT Finished configuration file.<br>
2018-07-25 14:43:04 GMT Conf_load_conf_file: My name: host-06a, id: 2876338=
096, addr: 1.2.3.7, port: 9876<br>
2018-07-25 14:43:04 GMT Conf_reload_initiate: My daemon parameters have cha=
nged! Exiting!<br>
2018-07-25 14:43:04 GMT=C2=A0 =C2=A0 =C2=A0Old: name 'host-06a', ad=
dr [1.2.3.7]:9876
, id '1696126475', num_ifs 1<br>
2018-07-25 14:43:04 GMT=C2=A0 =C2=A0 =C2=A0New: name 'host-06a', ad=
dr [1.2.3.7]:9876
, id '2876338096', num_ifs 1<br>
Exit caused by Alarm!
<br></div><div><br></div><div>#############################################=
######<br># Section: Spread_Segments (old version)<br>#####################=
##############################<br><br>Spread_Segment=C2=A0 <a href=3D"http:=
//1.2.3.5:9876">1.2.3.5:9876</a> {<br>=C2=A0=C2=A0=C2=A0 host-05a=C2=A0=C2=
=A0 1.2.3.5=C2=A0=C2=A0 { 1.2.3.5 }<br>}<br>Spread_Segment=C2=A0 <a href=3D=
"http://1.2.2.1:9876">1.2.2.1:9876</a> {<br>=C2=A0=C2=A0=C2=A0 host-51=C2=
=A0=C2=A0=C2=A0 1.2.2.1=C2=A0=C2=A0 { 1.2.2.1 }<br>}<br>Spread_Segment=C2=
=A0 <a href=3D"http://1.2.2.2:9876">1.2.2.2:9876</a> {<br>=C2=A0=C2=A0=C2=
=A0 host-52=C2=A0=C2=A0=C2=A0 1.2.2.2=C2=A0=C2=A0 { 1.2.2.2 }<br>}<br>Sprea=
d_Segment=C2=A0 <a href=3D"http://1.2.2.3:9876">1.2.2.3:9876</a> {<br>=C2=
=A0=C2=A0=C2=A0 host-53=C2=A0=C2=A0=C2=A0 1.2.2.3=C2=A0=C2=A0 { 1.2.2.3 }<b=
r>}<br>Spread_Segment=C2=A0 <a href=3D"http://1.2.3.7:9876">1.2.3.7:9876</a=
> {<br>=C2=A0=C2=A0=C2=A0 host-06a=C2=A0=C2=A0 1.2.3.7=C2=A0=C2=A0 { 1.2.3.=
7 }<br>}<br></div><div><br></div><div>
###################################################<br># Section: Spread_Se=
gments (new version)<br>###################################################=
<br><br>Spread_Segment=C2=A0 <a href=3D"http://1.2.3.5:9876">1.2.3.5:9876</=
a> {<br>=C2=A0=C2=A0=C2=A0 host-05a=C2=A0=C2=A0 1.2.3.5=C2=A0=C2=A0 { 1.2.3=
.5 }<br>}<br>Spread_Segment=C2=A0 <a href=3D"http://1.2.2.1:9876">1.2.2.1:9=
876</a> {<br>=C2=A0=C2=A0=C2=A0 host-51=C2=A0=C2=A0=C2=A0 1.2.2.1=C2=A0=C2=
=A0 { 1.2.2.1 }<br>}<br>Spread_Segment=C2=A0 <a href=3D"http://1.2.2.2:9876=
">1.2.2.2:9876</a> {<br>=C2=A0=C2=A0=C2=A0 host-52=C2=A0=C2=A0=C2=A0 1.2.2.=
2=C2=A0=C2=A0 { 1.2.2.2 }<br>}<br>##Spread_Segment=C2=A0 <a href=3D"http://=
1.2.2.3:9876">1.2.2.3:9876</a> {<br>## =C2=A0 host-53=C2=A0=C2=A0=C2=A0 1.2=
.2.3=C2=A0=C2=A0 { 1.2.2.3 }<br>##}<br>Spread_Segment=C2=A0 <a href=3D"http=
://1.2.3.7:9876">1.2.3.7:9876</a> {<br>=C2=A0=C2=A0=C2=A0 host-06a=C2=A0=C2=
=A0 1.2.3.7=C2=A0=C2=A0 { 1.2.3.7 }<br>}
<br></div><div><br></div><div class=3D"gmail_extra"><br><div class=3D"gmail=
_quote">On Wed, Aug 1, 2018 at 5:30 PM, John Lane Schultz <span dir=3D"ltr"=
><<a href=3D"mailto:[email protected]" target=3D"_blank">jschu=
[email protected]</a>></span> wrote:<br><blockquote class=3D"gmail_=
quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1=
ex">Hi Martin,<br>
<br>
Would you please send me the two configuration files (old and new) that cau=
sed this issue, so I can more concretely understand your incident?<br>
<span class=3D"im HOEnZb"><br>
Thanks,<br>
John<br>
<br>
On Jul 26, 2018, at 11:04 AM, Martin Schu <<a href=3D"mailto:martin.sc11=
[email protected]">[email protected]</a>> wrote:<br>
<br>
</span><div class=3D"HOEnZb"><div class=3D"h5">Hi John, hi all,<br>
<br>
if an already running daemon is triggered to reload the spread.conf at runt=
ime, it can crash if the number of daemons has changed. There can be a conf=
usion in the internal VirtualID table. Apparently spread fetches the wrong =
auto generated virtual id, if some daemon is removed/added in the middle of=
the table at runtime. <br>
<br>
Following steps lead to the problem:<br>
- spread version 5.0.1<br>
- No VirtualIDs are configured by us. VirtualIDs are auto-generated.<br>
- All daemons are running.<br>
- One Segment with one daemon is removed in the middle of spread.conf.<br>
- The spread.conf is distributed to all hosts.<br>
- spread is triggered to reload spread.conf by spmonitor r<br>
- some spread daemon will abort with bad failure shown below. It is the dae=
mon behind the removed one in the list.<br>
<br>
Here a conflict of virtual IDs is logged:<br>
2018-07-20 16:12:27 GMT Auto-generated virtual ID =3D '1696126475' =
for daemon 'host-06a'<br>
2018-07-20 16:12:27 GMT The virtual ID '1696126475' of 'host-06=
a' is already in use by 'host-52'!=C2=A0 You will probably need=
to explicitly reconfigure the daemons' virtual IDs so that they don=
9;t conflict.<br>
<br>
One of the spread daemons complaining after reload:<br>
2018-07-25 14:43:04 GMT Hash value for this configuration is: 4055467701<br=
>
2018-07-25 14:43:04 GMT Finished configuration file.<br>
2018-07-25 14:43:04 GMT Conf_load_conf_file: My name: host-05a, id: 3524380=
600, addr: 1.2.3.5, port: 9876<br>
2018-07-25 14:43:04 GMT Conf_reload_initiate: daemon identity mapped to two=
different old daemons: name ' host-06a' -> 0x7f57eccd8800, id &=
#39;2876338096' -> 0x7f57eccd8208! Partitioning to singleton!<br>
2018-07-25 14:43:04 GMT Conf_reload_initiate: Return need_singleton =3D 1<b=
r>
<br>
One other spread daemon crashing after reload because of virtual ID chaos:<=
br>
2018-07-25 14:43:04 GMT Hash value for this configuration is: 4055467701<br=
>
2018-07-25 14:43:04 GMT Finished configuration file.<br>
2018-07-25 14:43:04 GMT Conf_load_conf_file: My name: host-06a, id: 2876338=
096, addr: 1.2.3.7, port: 9876<br>
2018-07-25 14:43:04 GMT Conf_reload_initiate: My daemon parameters have cha=
nged! Exiting!<br>
2018-07-25 14:43:04 GMT=C2=A0 =C2=A0 =C2=A0Old: name 'host-06a', ad=
dr [1.2.3.7]:9876
, id '1696126475', num_ifs 1<br>
2018-07-25 14:43:04 GMT=C2=A0 =C2=A0 =C2=A0New: name 'host-06a', ad=
dr [1.2.3.7]:9876
, id '2876338096', num_ifs 1<br>
Exit caused by Alarm!<br>
<br>
Workaround:<br>
- Configure a unique VirtualID in spread.conf explicitly for each daemon is=
a solution, because this configuration is always reloaded including config=
ured virtualID.<br>
<br>
Maybe the virtualID table in memory has to be cleared before the parser is =
loading the spread.conf a second time.<br>
The auto-generation of virtualID is done only if the found virtualID is zer=
o. This is not true for a reload.<br>
<br>
No, I think this is not a problem of the internal hash algorithm delivering=
ambiguous hashes.<br>
<br>
Currently we have no fix for that bug, because the workaround is good enoug=
h for us.<br>
<br>
Best regards,<br>
Martin<br>
<br>
</div></div><div class=3D"HOEnZb"><div class=3D"h5">_______________________=
_______<wbr>_________________<br>
Spread-users mailing list<br>
<a href=3D"mailto:[email protected]">[email protected].=
org</a><br>
<a href=3D"http://lists.spread.org/mailman/listinfo/spread-users" rel=3D"no=
referrer" target=3D"_blank">http://lists.spread.org/<wbr>mailman/listinfo/s=
pread-users</a><br>
<br>
</div></div></blockquote></div><br></div></div>
--00000000000080fead057261d091--
--===============5515342163410090705==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Spread-users mailing list
[email protected]
http://lists.spread.org/mailman/listinfo/spread-users
--===============5515342163410090705==--