Learning on backup systems

Last weekend, I had a great learning about backups from one of our
customers.

We had few servers attacked by rootkits.
Few binaries were installed in 5 servers and they started to consume
network bandwidth highly. A new mail server was installed recently and it
started to send spam mails. We missed to harden the mail server as it was a
test mail server and we thought of keep it alive for a week only.

These issues were reported by network monitoring team.

After trying to clean the rootkit attacked server, we realised that it is
better to reinstall the entire os, hoping to restore the data from backup.

We reinstalled all the servers. Then we checked for the backups to restore.
But found that the disks for backup server were full 1 month ago.

We missed to notice that. There was no monitoring client on that backup
server. :-(

Restrored  the available data and trying to collect the lost data from
users machine and other servers.

Lesson learned :

1. don't destroy existing servers without checking the backups

2. Don't missout any server from the eyes of monitoring system

3. Run mock runs for restoring data from backup often.

4. Do the hardening of any server as first task after installation, even
though the server is for one day use.

5. Setup intrusion detection systems for critical servers. I thought it was
boring and not an essential one. But understood the importance of them.

6. Having multiple backup server is really good. It is not waste of money
or effort. It can help on hard times.

Though we know about these already, unless we suffer, we don't realise the
effects.

Requesting all the sys admins to make sure about their backup systems and
security.

Good backup and security systems will give you peace of mind and reduce
high tensions.
_______________________________________________
ILUGC Mailing List:
http://www.ae.iitm.ac.in/mailman/listinfo/ilugc
ILUGC Mailing List Guidelines:
http://ilugc.in/mailinglist-guidelines