Discussion on P2Portage and other ways to lower mirror bandwidth problem

This is a multi-part message in MIME format.

------=_NextPart_000_0013_01C361FF.3F2A08E0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

hi,

yesterday night, we have a quite long brainstorming time on the very prob=
lem
any distribution knows : the bandwidth problem (on #zynot freenode chan)

full log attached

contains ideas about P2Portage (like in Wiki) and diff download

have fun reading it

++

St=E9phane LOEUILLET / LeRoutier

------=_NextPart_000_0013_01C361FF.3F2A08E0
Content-Type: text/plain;
	name="P2Portage.txt"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
	filename="P2Portage.txt"

<LeRoutier> any of the guy interrested in the P2Portage idea around ?
<Lovechild> LeRoutier, shoot !
<LeRoutier> Lovechild: if you're one of those people, i suppose you =
already took a look at existing P2P FileSharing protocols, right ?
<Lovechild> not really in detail - I'm more interested if it can be done =
safely - I had an idea involving FreeNet
<Lovechild> but if you have an idea I would be more than willing to lend =
an ear.
<LeRoutier> FreeNet is secured, encrypted (good points),  it is also =
damn slow and only implementation (reference implementation is in java
<Lovechild> nothing wrong with java as such, it's portable and it =
works..=20
<LeRoutier> but we'll depend on a VM
<Lovechild> only if the user elects to use that feature in Porteus - if =
there's any reason not pick it (who knows...) we have fall backs in =
Porteus' getFile commands.
<DieMumiee> afai understood the users of freenet do not know what they =
share ..
<DieMumiee> so zynot users would mostly share foreign data ..
<DieMumiee> or we create our own zynot only freenet.. =20
<DieMumiee> but as LeRoutier mentioned it is slow ..=20
<Lovechild> yes they do, but they don't know who's sharing it, and =
there's protection for bogus files - as a Zynot users would would search =
for a keyfrase (and string of ints as I recall) and that would be the =
file you need and there's no way of faking it
<Lovechild> of course the builds are signed by developers, so we already =
know the frase to search for.
<LeRoutier> the reliability point, we have to know digests by another =
way than the P2P network.
<LeRoutier> as far as we have the MD5 (or any other reliable digest), we =
can fetch the file from the P2P network
<DieMumiee> .. still corruption is possible
<DieMumiee> u just have to implement a client that sends corrupted data =
.. and the end user will have to delete the download ..
<LeRoutier> no prob, every recent P2P protocol handles it (via the =
global digest of the file and also via subparts)
<DieMumiee> yes..=20
<LeRoutier> in fact, the "partionning" of shared files has 2 goals : =
help with corruption (as seen before), help at fast spreading of files =
too
<LeRoutier> another thing it helps : you can download parts from =
different people and the same time
<LeRoutier> s/and/at/
<LeRoutier> BitTorrent is very fast, reference implementation in python, =
but there is no BitTorrent network. no server is connected to another =
one
<DieMumiee> what network / system would you suggest?
<LeRoutier> in fact, one of the more feature-full is eDonkey/OverNet but =
its design (protocol i mean) sucks
<Lovechild> bittorrent scales better for files >100 megs - I doubt many =
of Zynots packages are that big.
<LeRoutier> in fact, most actual P2P systems are good for big files, =
because their main purpose is to share albums, films, games, ISO, ...
<DieMumiee> hmm the big ones are between 20 - 120 mb ..=20
<Lovechild> I have a package of OOo ximian edition 1.1rc2 and that's =
about the biggest our packages get - it's about 70megs
<LeRoutier> our big ones have those sizes, usual edonkey/bittorrent/... =
ones are 700MB up to 2GB
<LeRoutier> first P2P implementations where usually only dealing with =
small files under 10MB, (mostly MP3 and so), but they were unreliable, =
no CRC/digest check and the like
<DieMumiee> so we will have to write our own impl?=20
<Lovechild> but I think it would be hard to adapt BT to our use.
<Lovechild> but it's a good protocol, I really enjoy it.
<LeRoutier> yep, BitTorrent is fast but is a bad one for us, because it =
lacks the 'rsync like' part between official mirrors
<DieMumiee> why is the size so important for bitorent?
<LeRoutier> because it works with some kind of credit system
<DieMumiee> oh
<LeRoutier> if you UP, you can DL
<Lovechild> so which protocol gives us the things we need?
<LeRoutier> those that don't UP can somehow DL but at really poor speed
<LeRoutier> in fact, we need edonkey features, but with little tweaks, =
like a better chunk/block size
<Lovechild> I max out my upload and get only 30kb/s down on BT - and =
it's kinda annoying to see t3 users disconnect after having leeched
<DieMumiee> and maybe user auth
<LeRoutier> only few P2P systems have user auth
<LeRoutier> proprietary ones (where you pay) have it
<DieMumiee> if we find corrupted files it would be nice to find the =
users=20
<DieMumiee> that share it
<LeRoutier> also new 'private' P2P networks, which are small groups and =
auth via a key/certificate system
<Lovechild> Well, it would be really nice to be able to track people so =
we can kick off people who trick the network into sharing warez - those =
idiots deserve what's coming
<DieMumiee> well we can define what files get shared ...=20
<DieMumiee> if we have servers for the "rsync" stuff we can define what =
files maybe searched and redistributed ..
<LeRoutier> in fact, MD5 lists have to be fetched via a direct protocol, =
from a trusted host
<Lovechild> trusted host - that just sounds so... spoofable.
<DieMumiee> :)
<LeRoutier> you always have to trust at least someone
<Lovechild> I would like to see a ring of trust - if 2-3 devs have =
signed a package then I'll trust it... if only one did then I would be =
kinda scared.
<LeRoutier> for the warez problem, a mandatory auth system would make =
them go away. what they are looking for nowadays is anonymous P2P with =
encryption all over the place
<Lovechild> unless it's thorbjorn in which case I'll worship the package =
like it was god
<LeRoutier> but where do you fetch this digest from ? you have to trust =
the website where it is stored
<LeRoutier> what we need too is a protocol that handle both IPv4 and =
IPv6
<LeRoutier> a good feature should be also to redirect users to their =
nearest peer/mirror, using the free geoiplib for example
<LeRoutier> seems a large piece of cake to cook
<DieMumiee> well we have time ..=20
<LeRoutier> yep
<DieMumiee> and meanwhile other sources of dowload can be used
<LeRoutier> and it won't be for us alone, other distro/sites would be =
pleased if their user would lower their bandwitdth costs by using P2P
<LeRoutier> lets dream, SourceForge, Savanah, kernel.org, gnome.org =
using it
<DieMumiee> yap
<DieMumiee> we could reduce a lot of costs=20
<Lovechild> nice.. and then if someone would figure out a clever way to =
download tarballs and patches up to the current version instead of =
getting the entire thing ecah time... that would be SO nice.
<LeRoutier> the main problem with P2P those days comes that most =
broadband users have an async connexion, so they could say, download at =
1MB/s and only upload at 128KB/s
<LeRoutier> like a auto-diff system ?
<Lovechild> yeah, but forcing people to use it is the hard bit
<LeRoutier> say, i have kernel vanilla 2.4.21; i want 2.4.22, give me =
the diff
<Lovechild> yeah...
<Lovechild> that would save SO much bw, mirror maintainers would be =
thankful
<LeRoutier> yep, just with kernel, we only DL around 1MB or such instead =
of what, 26MB or more (bz2, not gz or .Z)
<Lovechild> that's how many procent saved... just think about it... and =
people say it's not worth the time developing such technologies????????
<LeRoutier> such diffs between, say 4 latest versions, should be =
available on every 'big' package
<LeRoutier> (kernel, mozilla, evolution, openoffice, xfree, ...)
<Lovechild> OpenOffice diffs would be so great...
<Lovechild> that would be like 50megs saved.. try that on 56k bastards
<LeRoutier> big too, even for a diff, openoffice release arent that =
often as kernel releases
<LeRoutier> i know it, i have 56k here (hum, lets say 48k)
<Lovechild> who cares would you rather download 50megs or 110 ?
<Lovechild> or kernels for that matter - 1 meg or 26 ?
<DieMumiee> yap .. gentoo never was an option for small bandwidth dialup =
users
<LeRoutier> only possible (but hard) with tricks like installing stage3 =
from another running distro
<Lovechild> and I must say diff downloads is one of my main priorities =
for zynot, I want that to happen, even if I have to learn how to code =
python just to do it.
<DieMumiee> :)
<Lovechild> I would like to do it in C# and .NET though, since I started =
looking at it and found it very nice.. but no matter what we need that =
feature.
<LeRoutier> diffs =3D more disk space required / far less bandwidth, and =
nowadays, disks are cheap, bandwidth is not
<Lovechild> even if I have to be nall's obdian servant for the rest of =
my natural life
<Lovechild> 120gb HDs are cheap as candy these days..
<Lovechild> trasferring 120gb ain't
<LeRoutier> kernel.org now : Current bandwidth utilization 84.53 Mbit/s=20
<Lovechild> and that's just the main site, the mirrors probably have =
some nice loads too
<LeRoutier> 120Gb is what kernel.org transfers in ... 25 minutes
<LeRoutier> 120GB=3D3h
<LeRoutier> i don't want such a telco bill
<Lovechild> OUCH!!!
<LeRoutier> yep, better take some vaseline
<Lovechild> mommy mommy, why do the aliens keep probing my ass... man =
that's insane, who knew Linux was that popular - I don't want to see =
SF.net's bw usage graphs
<LeRoutier> should be amazing
<LeRoutier> but on sf.net, you can have to choose a mirror to DL, on =
kernel.org, must people don't mind looking for a mirror
<Lovechild> just think how much $$$ we could save people by enforcing =
diff downloading...
<LeRoutier> s/can/
<LeRoutier> in fact, if this diff download is transparent, it is better
<Lovechild> yeah, it should be...
<LeRoutier> oh, i saw kernel-2.4.21.tar.bz2 in your shared file, MD5 is =
ok, you want 2.4.22, i fetch the diff, uncompress both, patch and check =
new manifest/digest
<LeRoutier> s/file/directory/
<Lovechild> that could work - but the Disk IO could be avoided if we =
simply signed the patches as well
<LeRoutier> yep, trusting the patch is one thing, but better have a =
clean old-version installed
<Lovechild> unpacking 2 kernels and packing the second again to check =
md5, just to unpack it afterwards - and then deleting them both...
<LeRoutier> havy IO, true
<LeRoutier> heavy
<Lovechild> we should avoid that approach I think
<LeRoutier> other proposition : download a patch which we know the =
digest, then downloading the full version on,ly if patching fails
<Lovechild> but of course, unpacking the old tarball and patching it to =
the new version and packing it would allow us to have the newest tarball =
in the DISTFILES dir - so we could then delete the old package and thus =
saving diskspace
<LeRoutier> and in the case of P2P, automatically share the new version
<Lovechild> yup
<LeRoutier> or without the repacking, share the patch
<Lovechild> or both
<Lovechild> what ever the result will be shifting the load from the =
server to the client which is endlessly more fair
<LeRoutier> at least, one thing we all agree tonight : beeing able to =
download diffs/patches instead of full heavy tarballs is far more a =
priority than using any P2P layer as the DL protocol


------=_NextPart_000_0013_01C361FF.3F2A08E0
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Zynot-dev mailing list
[email protected]
http://lists.zynot.org/mailman/listinfo/zynot-dev

------=_NextPart_000_0013_01C361FF.3F2A08E0--