Re: Checksum mismatch in single repo

On Fri, Oct 27, 2023 at 6:42 AM Pierre Fourès <[email protected]>
wrote:

> Hi Felix,
>
> Your SMART data looks good to me, except for the hard drive temperature.
> Experiencing 53°C looks quite a lot to me. Yet, this should not be the
> cause of your corrupted data.
>
> Two data-corruption problems on the same server which looks independant
> from each other, and occured at a quite long time range interval from each
> other, reminds me of a server who caused me lots of trouble until I
> discovered it had memory defects. I suspected hard disk failure and/or hard
> drive data corruption, but couldn't nail it with smartctl nor with the
> badblocks utility. I eventually nailed the problem when doing extensive
> test with the stress utility, showing that in some runs, the memory was
> corrupting data (which ended up corrupting data on disk). I had to run the
> tests many times to spot the defect. Subtle defects are real hard to spot
> on.
>
> IMO, I would advice you to do a full scan of this server to spot where the
> problem is in order to file this trail of problems as definitively solved.
> In my situation, similar to your one, the problems occured too distantly
> from each other to commit resources to investigate thoroughly. This period
> of uncertaintly and intuitive distrust of the server caused us a hidden
> costs like stress and fatigue. Having experienced it, if that happened
> again, I would prefer to rule out this situation quickly instead of knowing
> it dormant.
>
> Here are some links which might be relevant to you :
>   - https://en.wikipedia.org/wiki/Badblocks
>   - https://wiki.archlinux.org/title/Badblocks
>   - https://man.archlinux.org/man/stress.1
>   - https://wiki.archlinux.org/title/Stress_testing
>   - https://www.memtest.org/
>
> Best Regards,
> Pierre.
>


I can speak to RAM corruption as well. In one instance, we were
experiencing the strangest problems and blamed just about everything until
I ran the above memtest utility and it showed tremendous numbers of memory
errors. When I opened up the hardware, I found dust on and around the
memory. I cleaned that very thoroughly, put the system back together, and
ran memtest overnight or over a weekend with zero errors. Evidently, dust
can be conductive enough to act like a bunch of resistors across pins that
shouldn't have resistors across them.

As trivial as that sounds, I recommend to check for things like dust, and
since heat was mentioned, I'd check for fans that don't spin very freely. I
also recommend running memtest over a weekend, and finally, I am with the
camp who believe that ECC RAM is a good idea, so I'd suggest to check
whether you are using ECC RAM.

Hope this helps,
Nathan