Re: Known working sets II [was: Eggification redux]

Tres Seaver <[email protected]> Thu, 27 Sep 2007 19:40:11 -0400

Newsgroups	gmane.comp.web.zope.zope3
Message-ID	<[email protected]>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tres Seaver wrote:

> Anybody running against the Cheeseshop today is *more* on the bleeding
> edge than a sysadmin whose production boxes are running 'sid':  Debian
> has cultural constraits, even for that distro, which are vastly more
> restricted than the Wild West which is PyPI.
> 
> The only solution I can see is to create filtered subsets / mirrors of PyPI.

<snip>

> 
> Exactly.  Without some way to impose a "gatekeeper" role on the package
> pool from which a given deployment draws, we can't have any
> deterministic outcomes when installing packages.

OK, here is a sample "gatekeeper" script, intended to be run from within
a directory full of source distributions.  E.g.:

  $ cd /path/to/dist.example.com
  $ ls
  abc-1.2.3.tar.gz  abc-1.2.4.tar.gz  ghijk-2.3.4.tar.gz
  $ python /tmp/makeindex.py *.gz
  Parsing: abc-1.2.3.tar.gz
  Parsing: abc-1.2.4.tar.gz
  Parsing: ghijk-2.3.4.tar.gz
  Project: abc
    --> 1.2.3  abc-1.2.3.tar.gz
    --> 1.2.4  abc-1.2.4.tar.gz
  Project: ghijk
    --> 2.3.4  ghijk-2.3.4.tar.gz

Assuming that the directory is the root of an Apache virtual domain,
'dist.example.com', the script creates a 'simple' subdirectory, with
an index listing the projects corresponding to the tarballs.  Each
project ('abc', 'ghijk') gets a subdirectory with an index pointing to
its tarballs.

At this point, from a fresh virtualenv, you can install those packages
without risk of pulling anything from the Cheeseshop:

  $ bin/easy_install --index-url=http://dist.example.com/simple ghijk

Total effort involved in maintaining the "gated community" then becomes
keeping a set of tarballs available at some web-downloadable location,
and re-running the script after adding / removing them to regenerate
the index.

Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          [email protected]
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG/D9a+gerLs4ltQ4RAtZrAJwPrSe+vAaLTNF+XrrdyPY6bFXgTgCgzqOV
ssgeiDB9/whhld4DyylsQxA=
=f2tL
-----END PGP SIGNATURE-----

makeindex.py (application/x-httpd-cgi, 2.9 KB)

import os
import setuptools
import shutil
import subprocess
import sys
import tarfile
import tempfile

_tempdir = None

def _extractNameVersion(filename):
    global _tempdir
    if _tempdir is None:
        _tempdir = tempfile.mkdtemp()
    print 'Parsing:', filename
    tgz = tarfile.TarFile.gzopen(filename, 'r')
    try:
        names = tgz.getnames()
        for name in names:

            if name.endswith('PKG-INFO'):

                project, version = None, None

                for line in tgz.extractfile(name).readlines():
                    key, value = line.split(':', 1)

                    if key == 'Name':
                        project = value.strip()
                        if version is not None:
                            return project, version

                    elif key == 'Version':
                        version = value.strip()
                        if project is not None:
                            return project, version


            elif name == 'setup.py':
                tgz.extract(name, _tempdir)

        # no PKG-INFO found, do it the hard way.
        command = ('cd %s/%s && %s setup.py --name --version'
                                % (_tempdir, names[0], sys.executable))
        popen = subprocess.Popen(command,
                                 stdout=subprocess.PIPE,
                                 shell=True,
                                )
        output = popen.communicate()[0]
        return output.splitlines()[:2]
    finally:
        tgz.close()


def main(argv=None):
    if argv is None:
        argv = sys.argv[1:]

    projects = {}
    for arg in argv:
        try:
            project, revision = _extractNameVersion(arg)
            projects.setdefault(project, []).append((revision, arg))
        except:
            continue

    items = projects.items()
    items.sort()

    os.makedirs('simple')
    top = open('simple/index.html', 'w')
    top.writelines(['<html>\n',
                    '<body>\n',
                    '<h1>Package Index</h1>\n',
                    '<ul>\n'])

    for key, value in items:
        print 'Project: %s' % key
        dirname = 'simple/%s' % key
        os.makedirs(dirname)
        top.write('<li><a href="%s">%s</a>\n' % (key, key))

        sub = open('simple/%s/index.html' % key, 'w')
        sub.writelines(['<html>\n',
                        '<body>\n',
                        '<h1>%s Distributions</h1>\n' % key,
                        '<ul>\n'])

        for revision, archive in value:
            print '  -> %s, %s' % (revision, archive)
            sub.write('<li><a href="/%s">%s</a>\n' % (archive, archive))

        sub.writelines(['</ul>\n',
                        '</body>\n',
                        '</html>\n'])

    top.writelines(['</ul>\n',
                    '</body>\n',
                    '</html>\n'])
    top.close()

if __name__ == '__main__':
    main()
    if _tempdir is not None:
        shutil.rmtree(_tempdir)