Logwatch.org outage

The main logwatch.org website, FTP site, mailing lists, and CVS
repository was down for the past 24 hours.  I suffered multiple disk
failures and the restoration process was a bit painful.  Any changes
that were checked in to CVS on Friday, Feb 6th were also lost.

On an unrelated note, it seems I may have had some DNS and/or sendmail
configuration issues for a little while now, so I'm not sure if the
lists have been working.

If you guys wouldn't mind:

1) Can *one* person reply to the list to verify that the lists are
working again?
2) Can everybody check out a fresh copy of the logwatch source code
and compare it against your local copy to make sure nothing was lost?

Thanks!

And some details in case anybody is interested in a good lesson on
backups.  This server is my main file server as well and had all of my
data on it.  I'd say I'm above average in my backup practices:
1) All data stored on RAID1 (2 drives)
2) Separate backup drive for monthly full backups
3) Another backup drive for nightly incremental
4) Nightly incrementals of the most important data that is not too
large are also sent to an offsite location (via SSH)
5) My photo collection (I'm an avid photographer) is backed up onto
CDs incrementally and stored in a safe (these are too big to replicate
remotely).
6) The rest of my data is largely static (archive data) and was only
stored on the 4 internal drives I previously mentioned (one copy on
each drive).

Approximately 2 months ago one of my backup drives failed but I was
too busy to deal with it.  Then a week ago the other backup drive
failed (they were both fairly old).  I ordered two new drives.  When I
went to put them in and rebooted, I was notified of some errors on one
of my RAID1 drives.  I was then alarmed to find out that the *other*
RAID1 drive had been removed from the array at some point in the
recent past for some unknown reason (I need to look into why mdadm
didn't notify me).  So I tried to bring the (hopefully good) drive
back into the array before the other drive got any worse.
Unfortunately the replication kept failing and restarting because of
the errors on the active drive.  Since this was my root (/) volume I
decided I would boot to rescue mode to see if I could work with it
more freely.  Upon reboot the drive that had been active and getting
errors was no longer recognized by BIOS.  So all I had left out of my
4 drives and 4 copies of my data was one drive that had somewhat old
data and had been partially synced from the other drive.

This remaining drive had old data and had many filesystem errors upon
fsck such that I lost thousands of files (still not sure which ones).
I rebuilt the system with the new drives and am still trying to
recover as much data as I can.  All of my most important data that is
frequently modified I was able to restore from the offsite storage.  I
was able to pull all of my large static data off of that remaining
drive.  And I can get my most recent photos from the backup CDs.
There are, however, plenty of files that were lost and I currently
have no idea how many were.  I'm hoping that the static data is more
likely to not have been corrupted by the replication process.

One thing I have learned is that I need to include a file listing with
sizes and checksums in my backups so I can at least determine what is
missing or has been corrupted.  I have also learned that I need to be
more proactive when disks fail.  I am also going to include hot spares
in my new RAID1 array and I think I may use RAID1 (no hot spares) for
one or more of my backup drives.  Finally, I am going to find a
solution to allow me to incrementally backup my large and mostly
static files to burned DVDs.


------------------------------------------------------
Kirk Bauer <[email protected]>
http://linux.kaybee.org | www.logwatch.org
Author, Automating UNIX & Linux Administration