Re: Learning on backup systems

On Thu, Dec 24, 2015 at 1:04 AM, Shrinivasan T <[email protected]>
wrote:

> Last weekend, I had a great learning about backups from one of our
> customers.
>
> We had few servers attacked by rootkits.
> Few binaries were installed in 5 servers and they started to consume
> network bandwidth highly. A new mail server was installed recently and it
> started to send spam mails. We missed to harden the mail server as it was a
> test mail server and we thought of keep it alive for a week only.
>
> These issues were reported by network monitoring team.
>
> After trying to clean the rootkit attacked server, we realised that it is
> better to reinstall the entire os, hoping to restore the data from backup.
>
> We reinstalled all the servers. Then we checked for the backups to restore.
> But found that the disks for backup server were full 1 month ago.
>
> We missed to notice that. There was no monitoring client on that backup
> server. :-(
>
> Restrored  the available data and trying to collect the lost data from
> users machine and other servers.
>
> Lesson learned :
>
> 1. don't destroy existing servers without checking the backups
>
> 2. Don't missout any server from the eyes of monitoring system
>
> 3. Run mock runs for restoring data from backup often.
>
> 4. Do the hardening of any server as first task after installation, even
> though the server is for one day use.
>
> 5. Setup intrusion detection systems for critical servers. I thought it was
> boring and not an essential one. But understood the importance of them.
>
> 6. Having multiple backup server is really good. It is not waste of money
> or effort. It can help on hard times.
>
> Though we know about these already, unless we suffer, we don't realise the
> effects.
>
> Requesting all the sys admins to make sure about their backup systems and
> security.
>
> Good backup and security systems will give you peace of mind and reduce
> high tensions.
>

Thanks sir for this write-up and your insights on real-world scenario. I am
sure everyone has something to learn in this post.

And, I request everyone in this forum and to those who deals day-in and
day-out on real world IT issues to write-up about their experiences so that
everyone will get a chance to learn new things. This could help new-comers
and those who want to learn. :)
I know this may be time-consuming for them but a weekend post may work. If
we get 5posts like this then eventually we will have a good knowledge base
with the passage of time.

Thank you.


-- 

*Regards,Sahil ModGill*
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
This
email has been sent from a virus-free computer protected by Avast.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#151e6a84b51843f9_DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
_______________________________________________
ILUGC Mailing List:
http://www.ae.iitm.ac.in/mailman/listinfo/ilugc
ILUGC Mailing List Guidelines:
http://ilugc.in/mailinglist-guidelines