Re: bandwidth and space required to sync a basic mirror

anarcat <[email protected]> Tue, 03 Jun 2014 23:25:40 -0400
Newsgroups	gmane.linux.debian.user.mirrors
Message-ID	<[email protected]>
On 2013-11-20 13:56:09, Carsten Otto wrote:
> ftp2.de.debian.org reporting in :)

Hi!

Sorry for the immensely late response, but I got caught up on other
things. :)

> On Wed, Nov 20, 2013 at 01:31:00PM -0500, anarcat wrote:
>> Munin/Cacti graph of the disk usage from an existing mirror for the last
>> year would be awesome here, for example.
>
> Debian

[...]

That's exactly what I was looking for, thanks!!!

Out of curiosity, which hardware are you using to host this? a dedicated
server? something with a SAN behind?

>> Bandwidth for updates
>> =====================
>> 
>> It is unclear how much data is transfered on a daily or (ideally)
>> monthly basis for updates to the archive. I found a graph[2] which seem
>> to show the size transfered varies between 2 and 15 GB on a daily basis,
>> but that data is old (2012-08-12) and it's hard to get a good idea of
>> what it actually means - what's the average, for example? Or better,
>> what's the 95th percentile?
>
> The red lines (rsync) correspond to downloads we actively do using rsync
> (after being triggered).
>
> http://ftp.halifax.rwth-aachen.de/~cotto/rrd/debian-backports%23month.png
> http://ftp.halifax.rwth-aachen.de/~cotto/rrd/debian-cd%23month.png
> http://ftp.halifax.rwth-aachen.de/~cotto/rrd/debian-volatile%23month.png
> http://ftp.halifax.rwth-aachen.de/~cotto/rrd/debian%23month.png
>
> The graphical representation is not too helpful, but I also have the
> corresponding raw (Java-)RRD data. Just message me if you need something.

It would be great to have a 95th percentile of that red line, actually.

One of the things I am wondering is if it is worthwhile to keep a
private mirror running for clusters that have lots of machines and want
to spare the bandwidth costs yet do not want to offer the public hosting
commitment...

>> Bandwidth for users
>> ===================
>> 
>> I assume this is the hardest part to figure out, as this will vary
>> wildly according to the region you're in and the relative popularity of
>> your mirror.
>> 
>> The official mirror page[3] states that you need a "T1 or better" (for
>> the record and those who weren't around in those days, a T1 is
>> 1.5Mbit/s). I hardly think this requirement is fit for a public mirror
>> these days. :) The official page does mention this requirement is
>> "higher" for "developped countries", but fails to clarify that
>> requirement.
>
> Have a look at the graphs above. With 51 MBit/sec on average, Debian is
> not very demanding (although we're not ftp.de.debian.org, just ftp2).

I would consider 50mbps a respectable usage. :) I'm curious to see what
kind of pain ftp.de is suffering... ;)

Thank you so much for this again (and everyone else that answered!),
it's great to see those graphs.

A.

-- 
L'art n'est pas un bureau d'anthropométrie.
                        - Léo Ferré, "Préface"
signature.asc (application/pgp-signature, 818 B)