Re: [mh] Reliability & Redundancy (was "Voice commands with arbitrary parameters")
Giles Godart-Brown <[email protected]> Sun, 27 Jul 2025 10:22:24 +0100
| Newsgroups | gmane.comp.misc.misterhouse.user |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format.
--===============3063503808953881700==
Content-Type: multipart/alternative;
boundary="------------HHQam6kH1R0ODMhGZGFXyrk9"
Content-Language: en-GB
This is a multi-part message in MIME format.
--------------HHQam6kH1R0ODMhGZGFXyrk9
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Please see comments in-line.
One other thing on this topic, I run masters and backups on separate
UPSs in different parts of the house on separate Ethernet switches. Pis
use so little power a pretty basic UPS will keep them going for quite a
while.
Giles
On 26/07/2025 18:31, Brian M wrote:
> TL;DR: Giles described implementing fail-over. Brian describes
> diskless booting, need for replication, and split-brain issues.
>
> ----
>
> Good to hear from you.
>
> I changed the subject to aid others in finding this topic later,
> because I think you've got some really good information in your post
> (below) regarding MH reliability and redundancy. I think you've got a
> great set-up, and as a result of your post I may head in the same
> direction. Thank you!
>
> I have two critical RPis doing separate tasks, but really either one
> could handle both workloads. I could reconfigure them as primary and
> back-up along the lines of your set-up.
>
>
> Diskless
>
> Personally, I've always worried about the SD card failing. As a
> result, I've configured my two RPis to be diskless. They each PXE boot
> from a central server that runs mdadm-style RAIDed disks. Root, /var,
> MH's home, and everything else are provided via NFS.
GGB> Yes SD cards are very likely to fail eventually so I have moved
over to SSDs which have become very cheap now. I mount my Pis in a stack
with one live USB SSD connected and another cloned disk sat ready to
connect if the master fails. rpi-clone is great for building these. I'm
also experimenting with USB raid arrays like this
(https://www.maiwo360.com/2Bay/284.html) They seem to work in Raid1
mirroring, but I have had one disk fail that slowed the Pi down so much
it became almost unusable. The good news was that the good disk was OK,
and I was able to create a new mirror using dd and all is now well
(there are no docs I can find about how to recover from failures)
>
>
> Replication
>
> One nuisance issue I see with your set-up is that you need to make
> sure that MH's home directory is kept in sync between the two systems.
> A bug fix on one side needs to be applied to the other side, too.
> Presumably you're doing some automatic Rsync's or something to handle
> this. In my situation, I have another solution...
>
GGB> My own code is pretty stable now and I've moved a lot of things
into a MySQL database (temp set points, links between sensors and
actuators etc,) So I only rsync that on an ad hoc basis after I've made
and tested changes. The MH data directory is however volatile and I've
found for my installation an rsync every 2 minutes is sufficient. Note
I nfs mount the master SSD to do the sync, then unmount it.
> My MH home lives on a separate NFS share on the central server.
> Combining my set-up with what you've done, I wouldn't even need to
> replicate MH configuration and code data between the two RPi's data.
> The two RPis would still need separate /, /var, etc. to maintain their
> separate configurations, but either system could mount the MH home
> share whenever that system became the active system. The newly active
> system would have the latest, up-to-the-moment, MH data, right down to
> the log files. This same technique could be applied to my OpenHab
> system, or any other app I wanted to share.
GGB> My first NAS was quite unreliable so I tried to make the MH
installation as autonomous as possible. Given my time again with a very
reliable Synology NAS I would do the same as you.
>
> Split-brain
>
> The other issue that fail-over set-ups have classically run into is
> the "split-brain" situation, where some network issue makes the backup
> think the main system is down when it's not, and now you have two
> active systems trying to run the world. The DRBD project (ref:
> https://linbit.com/drbd/) recommends an alternate communication
> channel (another NIC, maybe some sort of usb-to-usb link, even just a
> couple of wires between GPIO pins) solely to allow the backup to
> confirm the main system is really out of service. MQTT's LWT ("last
> will and testament") function might fill this role, too, if it doesn't
> live in one of the RPIs.
>
GGB> Good idea. keepalived allows you to write your own code to detect
if a machine is up or down with a shell script that returns success or
failure so the options are endless. I do make extensive use of LWT as well.
> Split-brain is not going to be a day-to-day issue, but will probably
> happen to everyone with a failover set-up sooner or later. Have you
> run into this, and how have you handled it?
GGB> Not yet, but no doubt it will bite me one day.
>
>
> Home Assistant
>
> Re HA: I went with OpenHab because I perceived HA to have a "My way
> or the highway" attitude, and their installation models didn't match
> my use case. OpenHab has a smaller community, to be sure, but I'm
> pleased. I never used HA, so I don't know the reality of HA and I
> can't really compare them.
>
>
> ---
>
> Some day I'll learn to write a short post, but apparently not today. :-/
GGB> :-)
>
> Thanks again for your post. Great stuff there.
>
> -Brian M.
>
>
>
> On 7/26/25 03:14, Giles Godart-Brown wrote:
>> Hello Brian
>>
>> Long time.
>>
>> I don't think I can answer your question, but I have now changed the
>> way I use Misterhouse to increase reliability as follows;
>>
>> * Complex logic - keep in Misterhouse
>> * Simple logic (like timers and pirs switching lights) - use
>> Tasmota's rules wherever possible
>> * Comms protocol - Mostly MQTT via mosquitto
>> * UI - Home Assistant
>> * Interface to difficult devices (like tuya) - Home Assistant
>> * Voice commands - Alexa - to HA - to MH
>>
>> I've also discovered the pi keepalive daemon
>> (https://github.com/justinknguyen/Pi-Guide/blob/main/Pi-Guide/keepalived.md)
>> which allows me to run Misterhouse (and other services) on a pair of
>> Pis that automatically look after each other and switch if one fails
>> but keeps a common IP address.
>> For example I was worried about mosquitto being a really important
>> single point of failure, now I have two pis, the master is on
>> 192.168.100.23 and the backup on 192.168.100.24 but the common
>> address 192.168.100.25 is automatically associated with the currently
>> running server. All I have to do is set all my devices to look for
>> 192.168.100.25 and keepalived looks after the routing.
>> I use the same for Wireguard VPN and Misterhouse.
>> Unfortunately Home Assistant is dropping support for native Pi OS
>> installs, and only concentrating on HA OS which means I cannot use
>> keepalived. For now I'm thinking I will just have to have a cold
>> backup machine on standby with a remote power switch so I can power
>> it up manually if the master fails.
>> Regards
>> Giles
>>
>>
>> On 26/07/2025 08:11, Brian M wrote:
>>> I'm diving into so-called voice commands for the first time, though
>>> I'm really just using mhsend to send a command from another
>>> computer. The basics work fine, but in one case I'd like to pass an
>>> arbitrary file name, and have my code receive that file name.
>>> Something like "play file <filename>" and have my code be able to
>>> retrieve whatever was specified for <filename>. Can that be done,
>>> and if so, what's the syntax to define the voice command with an
>>> arbitrary string?
>>>
>>> -Brian M.
>>>
>>>
>>>
>>> ________________________________________________________
>>> To unsubscribe from this list, go to:
>>> https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>>>
>>
>
--------------HHQam6kH1R0ODMhGZGFXyrk9
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<font size="4">Please see comments in-line.<br>
One other thing on this topic, I run masters and backups on
separate UPSs in different parts of the house on separate Ethernet
switches. Pis use so little power a pretty basic UPS will keep
them going for quite a while.<br>
<br>
Giles<br>
</font><br>
<div class="moz-cite-prefix">On 26/07/2025 18:31, Brian M wrote:<br>
</div>
<blockquote type="cite"
cite="mid:[email protected]">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div class="moz-cite-prefix">TL;DR: Giles described implementing
fail-over. Brian describes diskless booting, need for
replication, and split-brain issues.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">----<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Good to hear from you. <br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">I changed the subject to aid others
in finding this topic later, because I think you've got some
really good information in your post (below) regarding MH
reliability and redundancy. I think you've got a great set-up,
and as a result of your post I may head in the same direction.
Thank you!</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">I have two critical RPis doing
separate tasks, but really either one could handle both
workloads. I could reconfigure them as primary and back-up along
the lines of your set-up.<br>
</div>
<h4 class="moz-cite-prefix">Diskless<br>
</h4>
<div class="moz-cite-prefix">Personally, I've always worried about
the SD card failing. As a result, I've configured my two RPis to
be diskless. They each PXE boot from a central server that runs
mdadm-style RAIDed disks. Root, /var, MH's home, and everything
else are provided via NFS.<br>
</div>
</blockquote>
GGB> Yes SD cards are very likely to fail eventually so I have
moved over to SSDs which have become very cheap now. I mount my Pis
in a stack with one live USB SSD connected and another cloned disk
sat ready to connect if the master fails. rpi-clone is great for
building these. I'm also experimenting with USB raid arrays like
this (<a class="moz-txt-link-freetext" href="https://www.maiwo360.com/2Bay/284.html">https://www.maiwo360.com/2Bay/284.html</a>) They seem to work in
Raid1 mirroring, but I have had one disk fail that slowed the Pi
down so much it became almost unusable. The good news was that the
good disk was OK, and I was able to create a new mirror using dd and
all is now well (there are no docs I can find about how to recover
from failures)<br>
<blockquote type="cite"
cite="mid:[email protected]">
<div class="moz-cite-prefix"> </div>
<h4 class="moz-cite-prefix">Replication<br>
</h4>
<div class="moz-cite-prefix">One nuisance issue I see with your
set-up is that you need to make sure that MH's home directory is
kept in sync between the two systems. A bug fix on one side
needs to be applied to the other side, too. Presumably you're
doing some automatic Rsync's or something to handle this. In my
situation, I have another solution...</div>
<div class="moz-cite-prefix"><br>
</div>
</blockquote>
GGB> My own code is pretty stable now and I've moved a lot of
things into a MySQL database (temp set points, links between sensors
and actuators etc,) So I only rsync that on an ad hoc basis after
I've made and tested changes. The MH data directory is however
volatile and I've found for my installation an rsync every 2 minutes
is sufficient. Note I nfs mount the master SSD to do the sync, then
unmount it.<br>
<blockquote type="cite"
cite="mid:[email protected]">
<div class="moz-cite-prefix"> </div>
<div class="moz-cite-prefix">My MH home lives on a separate NFS
share on the central server. Combining my set-up with what
you've done, I wouldn't even need to replicate MH configuration
and code data between the two RPi's data. The two RPis would
still need separate /, /var, etc. to maintain their separate
configurations, but either system could mount the MH home share
whenever that system became the active system. The newly active
system would have the latest, up-to-the-moment, MH data, right
down to the log files. This same technique could be applied to
my OpenHab system, or any other app I wanted to share.</div>
</blockquote>
GGB> My first NAS was quite unreliable so I tried to make the MH
installation as autonomous as possible. Given my time again with a
very reliable Synology NAS I would do the same as you.<br>
<br>
<blockquote type="cite"
cite="mid:[email protected]">
<h4 class="moz-cite-prefix">Split-brain</h4>
<div class="moz-cite-prefix">The other issue that fail-over
set-ups have classically run into is the "split-brain"
situation, where some network issue makes the backup think the
main system is down when it's not, and now you have two active
systems trying to run the world. The DRBD project (ref: <a
class="moz-txt-link-freetext" href="https://linbit.com/drbd/"
moz-do-not-send="true">https://linbit.com/drbd/</a>)
recommends an alternate communication channel (another NIC,
maybe some sort of usb-to-usb link, even just a couple of wires
between GPIO pins) solely to allow the backup to confirm the
main system is really out of service. MQTT's LWT ("last will and
testament") function might fill this role, too, if it doesn't
live in one of the RPIs. </div>
<div class="moz-cite-prefix"><br>
</div>
</blockquote>
GGB> Good idea. keepalived allows you to write your own code to
detect if a machine is up or down with a shell script that returns
success or failure so the options are endless. I do make extensive
use of LWT as well.<br>
<blockquote type="cite"
cite="mid:[email protected]">
<div class="moz-cite-prefix"> </div>
<div class="moz-cite-prefix">Split-brain is not going to be a
day-to-day issue, but will probably happen to everyone with a
failover set-up sooner or later. Have you run into this, and how
have you handled it?</div>
</blockquote>
GGB> Not yet, but no doubt it will bite me one day.<br>
<blockquote type="cite"
cite="mid:[email protected]">
<h4 class="moz-cite-prefix">Home Assistant</h4>
<div class="moz-cite-prefix">Re HA: I went with OpenHab because I
perceived HA to have a "My way or the highway" attitude, and
their installation models didn't match my use case. OpenHab has
a smaller community, to be sure, but I'm pleased. I never used
HA, so I don't know the reality of HA and I can't really compare
them.<br>
</div>
<h4 class="moz-cite-prefix">---<br>
</h4>
<div class="moz-cite-prefix">Some day I'll learn to write a short
post, but apparently not today. :-/<br>
</div>
</blockquote>
GGB> :-)<br>
<blockquote type="cite"
cite="mid:[email protected]">
<div class="moz-cite-prefix"> </div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Thanks again for your post. Great
stuff there.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"> -Brian M.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 7/26/25 03:14, Giles Godart-Brown
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:[email protected]">
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8">
<font size="4">Hello Brian<br>
<br>
Long time. <br>
<br>
I don't think I can answer your question, but I have now
changed the way I use Misterhouse to increase reliability as
follows;<br>
</font>
<ul>
<li><font size="4">Complex logic - keep in Misterhouse</font></li>
<li><font size="4">Simple logic (like timers and pirs
switching lights) - use Tasmota's rules wherever possible<br>
</font></li>
<li><font size="4">Comms protocol - Mostly MQTT via mosquitto</font></li>
<li><font size="4">UI - Home Assistant </font></li>
<li><font size="4">Interface to difficult devices (like tuya)
- Home Assistant</font></li>
<li><font size="4">Voice commands - Alexa - to HA - to MH<br>
</font></li>
</ul>
<font size="4">I've also discovered the pi keepalive daemon (<a
class="moz-txt-link-freetext"
href="https://github.com/justinknguyen/Pi-Guide/blob/main/Pi-Guide/keepalived.md"
moz-do-not-send="true">https://github.com/justinknguyen/Pi-Guide/blob/main/Pi-Guide/keepalived.md</a>)
which allows me to run Misterhouse (and other services) on a
pair of Pis that automatically look after each other and
switch if one fails but keeps a common IP address.<br>
For example I was worried about mosquitto being a really
important single point of failure, now I have two pis, the
master is on 192.168.100.23 and the backup on 192.168.100.24
but the common address 192.168.100.25 is automatically
associated with the currently running server. All I have to
do is set all my devices to look for </font><font size="4">192.168.100.25
and keepalived looks after the routing.<br>
I use the same for Wireguard VPN and Misterhouse.<br>
Unfortunately Home Assistant is dropping support for native Pi
OS installs, and only concentrating on HA OS which means I
cannot use keepalived. For now I'm thinking I will just have
to have a cold backup machine on standby with a remote power
switch so I can power it up manually if the master fails.<br>
Regards<br>
Giles<br>
</font><font size="4"><br>
</font><br>
<div class="moz-cite-prefix">On 26/07/2025 08:11, Brian M wrote:<br>
</div>
<blockquote type="cite"
cite="mid:[email protected]">I'm
diving into so-called voice commands for the first time,
though I'm really just using mhsend to send a command from
another computer. The basics work fine, but in one case I'd
like to pass an arbitrary file name, and have my code receive
that file name. Something like "play file <filename>"
and have my code be able to retrieve whatever was specified
for <filename>. Can that be done, and if so, what's the
syntax to define the voice command with an arbitrary string? <br>
<br>
-Brian M. <br>
<br>
<br>
<br>
________________________________________________________ <br>
To unsubscribe from this list, go to: <a
class="moz-txt-link-freetext"
href="https://lists.sourceforge.net/lists/listinfo/misterhouse-users"
moz-do-not-send="true">https://lists.sourceforge.net/lists/listinfo/misterhouse-users</a>
<br>
<br>
</blockquote>
<br>
</blockquote>
<p><br>
</p>
</blockquote>
<br>
</body>
</html>
--------------HHQam6kH1R0ODMhGZGFXyrk9--
--===============3063503808953881700==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
--===============3063503808953881700==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users
--===============3063503808953881700==--