Re: How to salvage a degraded mdadm RAID1 with as little data loss as possible?

On 6/21/26 12:46, Paul Leiber wrote:
> Am 21.06.26 um 06:47 schrieb David Christensen:
>> Do you own a power supply tester?
> 
> ... I will put a test of
> the power supply on the list of things to try.


Good.


>> Does your computer have ECC memory?
> 
> Yes, it does have ECC memory. 


Good.


> I will put the memory test on the list as well.


Choose your testing tool and methodology carefully -- memtest86 
(commercial) vs. memtest86+ (FOSS), memory correction report logging to 
motherboard firmware vs. operating system, etc..  See this thread and 
research carefully:

https://lists.debian.org/debian-user/2026/05/msg00386.html


>> Have you tested your hard disks?
> 
> Both disks are monitored via smartctl. Automated short and long tests 
> are being done regularly. There are no indications for hardware failure 
> in the smart data.


Good.


>> Have you validated the filesystem with fsck.xfs(8)?
> 
> I just did a check of the file system (using xfs_repair -n), with no 
> errors reported.


Good.


> 
>> Do you have streams of database transactions since the last known good 
>> backups?  If so, can they be replayed? 
> 
> Not exactly knowing what such a stream is, I guess I don't have one. But 
> I am not sure. Will check.


Good.


>> Can you switch the databases to read-only, shutdown, disconnect the 
>> first disk, boot, backup the database(s), shutdown, connect the first 
>> disk, disconnect the second disk, boot, backup the database(s), 
>> shutdown, and connect the second disk?
> 
> That's a good suggestion. I will need to check what booting with just 
> one disk could do to the BTRFS filesystem, but this might be a way to 
> force md to use the disk which is currently indicated as failed.


Is your OS on the btrfs mirror?  I have found that putting the OS on a 
dedicated SSD makes operations, maintenance, trouble-shooting, disaster 
preparedness/ recovery, etc., much easier.


On 6/21/26 14:45, Paul Leiber wrote:
 > I managed to rebuild the md RAID1 using the data on the seemingly
 > failed device (partition_1). First, I did a dd dump of partition_2
 > (currently in usage) in order not to lose data. Then, I recreated the
 > RAID1 using the data from partition_1:
 >
 > mdadm --stop /dev/md0 # This stops the degraded RAID1
 > mdadm --assemble --update=uuid /dev/md0 /dev/mapper/partition_1 # This
 > creates a new RAID1 using the partition_1, a new array UUID is
 > required
 > in order for --assembly to work
 > mdadm --manage --add /dev/md0 /dev/mapper/partition_2 # This adds
 > partition_2 to the RAID1, contents of partition_1 are replicated to
 > partition_2 automatically
 >
 > md is currently replicating the data (German movie descriptions in
 > Kodi, yay!) from partition_1 to partition_2. I might have to turn
 > partition_2 from "spare" to "active", but I'll let the replication
 > complete first.
 >
 > In any case, I set up mdmonitor to alert me if the RAID1 degrades
 > again.  That's something I should have thought of earlier.
 >
 > We'll see if this issue occurs again. I'll give an update if this is
 > the case.
 >
 > Thanks to everybody for trying to help me!
 >
 > Paul


Thank you for the curious problem.  We all learn when we work together 
on a solution.  Please let us know how it works out.


David