Replacement client for pep381client

Christian Theune <[email protected]> Wed, 20 Mar 2013 16:59:21 -0700
Newsgroups	gmane.comp.python.catalog
Message-ID	<[email protected]>
This is a multi-part message in MIME format.

--===============0719565562==
Content-Type: multipart/alternative; boundary=--------------3141024881606143750

This is a multi-part message in MIME format.

----------------3141024881606143750
Content-Type: text/plain; charset=iso-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable

Hi,

as you might be aware, I've done my share on bitching about my mirror=20
(f.pypi.python.org) breaking.

I have picked pep381client apart yesterday and rebuilt it - mostly from=20
ground up.

You can find a working version here:
https://bitbucket.org/ctheune/bandersnatch

The focus has been on making it a lot more robust and a lot easier to=20
repair a mirror when it's known to be broken. To achieve that I:

- refactored the code, trying to make it more intentional, less mechanica=
l
- stop parsing the simple pages' html and make more use of the XML-RPC AP=
I
- add Tarek's worker/queue approach for parallelizing it
- keep as little state as possible on the client
- switch form timestamps to serial counters for checking what and how=20
much to update
- handle locking of concurrent runs more gracefully

I think I have a good grasp of what's going on now so that I can keep=20
maintining this in the future.

I'm currently re-initializing my own mirror. This basically can be run=20
in-place by just removing the existing state data and calling my sync=20
script (bsn-mirror) instead of pep381run with the same parameters.

Tomorrow I'll update the documentation, make it use a config file and=20
put some lipstick on the main entry point. After that I should be ready=20
for a release.

If you want to give it a try already, you just do this:

$ hg clone https://bitbucket/org/ctheune/bandersnatch
$ cd bandersnatch
$ virtualenv-2.7 .
$ bin/python bootstrap.py
$ bin/buildout
$ bin/bsn-mirror /my/mirror/path

Cheers,
Christian
----------------3141024881606143750
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/h=
tml4/strict.dtd">
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-885=
9-1">
<meta http-equiv=3D"Content-Style-Type" content=3D"text/css">
<title></title>
<meta name=3D"Generator" content=3D"Cocoa HTML Writer">
<meta name=3D"CocoaVersion" content=3D"1187.37">
<style type=3D"text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; line-height: 14.0px; font: 12.0px =
Helvetica; min-height: 14.0px}
p.p3 {margin: 0.0px 0.0px 0.0px 0.0px; line-height: 14.0px; font: 12.0px =
Helvetica}
</style>
</head>
<body>
<p class=3D"p1">Hi,</p>
<p class=3D"p2"><br></p>
<p class=3D"p3">as you might be aware, I've done my share on bitching abo=
ut my mirror (f.pypi.python.org) breaking.</p>
<p class=3D"p2"><br></p>
<p class=3D"p3">I have picked pep381client apart yesterday and rebuilt it=
 - mostly from ground up.</p>
<p class=3D"p2"><br></p>
<p class=3D"p3">You can find a working version here:</p>
<p class=3D"p3"><a href=3D"https://bitbucket.org/ctheune/bandersnatch">ht=
tps://bitbucket.org/ctheune/bandersnatch</a></p>
<p class=3D"p2"><br></p>
<p class=3D"p3">The focus has been on making it a lot more robust and a l=
ot easier to repair a mirror when it's known to be broken. To achieve tha=
t I:</p>
<p class=3D"p2"><br></p>
<p class=3D"p3">- refactored the code, trying to make it more intentional=
, less mechanical</p>
<p class=3D"p3">- stop parsing the simple pages' html and make more use o=
f the XML-RPC API</p>
<p class=3D"p3">- add Tarek's worker/queue approach for parallelizing it<=
/p>
<p class=3D"p3">- keep as little state as possible on the client</p>
<p class=3D"p3">- switch form timestamps to serial counters for checking =
what and how much to update</p>
<p class=3D"p3">- handle locking of concurrent runs more gracefully</p>
<p class=3D"p2"><br></p>
<p class=3D"p3">I think I have a good grasp of what's going on now so tha=
t I can keep maintining this in the future.</p>
<p class=3D"p2"><br></p>
<p class=3D"p3">I'm currently re-initializing my own mirror. This basical=
ly can be run in-place by just removing the existing state data and calli=
ng my sync script (bsn-mirror) instead of pep381run with the same paramet=
ers.</p>
<p class=3D"p2"><br></p>
<p class=3D"p3">Tomorrow I'll update the documentation, make it use a con=
fig file and put some lipstick on the main entry point. After that I shou=
ld be ready for a release.</p>
<p class=3D"p2"><br></p>
<p class=3D"p3">If you want to give it a try already, you just do this:</=
p>
<p class=3D"p2"><br></p>
<p class=3D"p3">$ hg clone https://bitbucket/org/ctheune/bandersnatch</p>
<p class=3D"p3">$ cd bandersnatch</p>
<p class=3D"p3">$ virtualenv-2.7 .</p>
<p class=3D"p3">$ bin/python bootstrap.py</p>
<p class=3D"p3">$ bin/buildout</p>
<p class=3D"p3">$ bin/bsn-mirror /my/mirror/path</p>
<p class=3D"p2"><br></p>
<p class=3D"p3">Cheers,</p>
<p class=3D"p3">Christian</p>
</body>
</html>
----------------3141024881606143750--



--===============0719565562==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Catalog-SIG mailing list
[email protected]
http://mail.python.org/mailman/listinfo/catalog-sig

--===============0719565562==--