Re: Learning on backup systems

On Wed, Dec 23, 2015 at 1:34 PM, Shrinivasan T <[email protected]> wrote:
> Last weekend, I had a great learning about backups from one of our
> customers.
>
> We had few servers attacked by rootkits.
> Few binaries were installed in 5 servers and they started to consume
> network bandwidth highly. A new mail server was installed recently and it
> started to send spam mails. We missed to harden the mail server as it was a
> test mail server and we thought of keep it alive for a week only.
>
> These issues were reported by network monitoring team.
>
> After trying to clean the rootkit attacked server, we realised that it is
> better to reinstall the entire os, hoping to restore the data from backup.
>
> We reinstalled all the servers. Then we checked for the backups to restore.
> But found that the disks for backup server were full 1 month ago.
>
> We missed to notice that. There was no monitoring client on that backup
> server. :-(
>
> Restrored  the available data and trying to collect the lost data from
> users machine and other servers.
>
> Lesson learned :
>
> 1. don't destroy existing servers without checking the backups
>
> 2. Don't missout any server from the eyes of monitoring system
>
> 3. Run mock runs for restoring data from backup often.
>
> 4. Do the hardening of any server as first task after installation, even
> though the server is for one day use.
>
> 5. Setup intrusion detection systems for critical servers. I thought it was
> boring and not an essential one. But understood the importance of them.
>
> 6. Having multiple backup server is really good. It is not waste of money
> or effort. It can help on hard times.
>
> Though we know about these already, unless we suffer, we don't realise the
> effects.
>
> Requesting all the sys admins to make sure about their backup systems and
> security.
>
> Good backup and security systems will give you peace of mind and reduce
> high tensions.

All great points and they should be the "due diligence" list for any
sys admin worth his/her weight in salt.

Frankly, I am bit surprised that many items listed in your post fell
through the cracks by your sys admins.

You missed one critical point -- keep your production servers (at a
minimum open to the WAN) up to date with security and bug fixes.

Beside IDS/IPS  like aide, fail2ban etc, 'logwatch' is a wonderful
tool.  Install it and have the reports emailed to you; more
importantly read them and you might get a head start on impending
problems in your servers.

Cheers,
-- Arun Khan
_______________________________________________
ILUGC Mailing List:
http://www.ae.iitm.ac.in/mailman/listinfo/ilugc
ILUGC Mailing List Guidelines:
http://ilugc.in/mailinglist-guidelines