Re: Converting PyPI to feeds

Hi Bastian,

I will make them publicly available. I don't have any solid hosting myself
(and for people to rely on these feeds instead of PyPI, it should be), so I
planned on hosting the feeds on github.io with GitHub Pages. I've avoided
modifying PyPI's distribution archives (the tgz, zip files) so that it can
simply link to PyPI for those (about 160GB in total; but I bet it's better
to reuse PyPI's CDN instead of hosting it ourselves). So, the only thing to
host are the feed files. GitHub pages have a monthly bandwidth limit of
100GB, which should be plenty for just feeds.

I will also need a script to be run every hour to update the feeds. I was
planning to use Travis. This update is incremental and needs to download
all archives that have been added since the last update; they are discarded
at the end of the run. Travis has a time limit of 30 min per run, probably
sufficient for an hourly update. I will run the initial 'update' locally on
my machine, to jump start it, as it would exceed Travis/GitHub's limits. I
may be able to avoid running setup.py, to avoid a malicious setup.py from
messing with the server or the generated feeds; if not, I'd need docker or
something with sufficient isolation.

Providing an interface like maven.0install.net sounds good to me, i.e.
https://pypi.0install.net/{pypi_package_name}. I'm not sure whether they
should be hosted on 0install.net or on GitHub, Travis is convenient though.

Cheers,
Tim

On 20 February 2017 at 19:45, Bastian Eicher <[email protected]> wrote:

> Hi Tim,
>
>
>
> this sounds very interesting! Unfortunately I do not have much Python
> expertise, so I probably cannot offer any valuable feedback on the nuts and
> bolts of the design.
>
>
>
> Do you plan on publicly hosting the feeds your converter generates? This
> would seem to be somewhat similar in spirit to my Maven-to-ZeroInstall
> webservice hosted at http://maven.0install.net/ . Perhaps a directory of
> bindings between Zero Install and other package managers would be in order
> (hoping the number increases :) ).
>
>
>
> Regards
>
> Bastian
>
>
>
> *From:* Tim Diels [mailto:[email protected]]
> *Sent:* Samstag, 18. Februar 2017 00:18
> *To:* The Zero Install system <[email protected]>
> *Subject:* Re: [Zero-install-devel] Converting PyPI to feeds
>
>
>
> I've made a draft on how to convert PyPI packages to ZI feeds (most of the
> documentation section):
>
>
>
> https://github.com/timdiels/pypi-to-0install#documentation
>
>
>
> I'd appreciate any comments/review of it before going ahead and
> implementing it. This approach assumes setup.py has the typical interface
> that setuptools.setup would, no need for docker this way.
>
>
>
> On Wed, Feb 1, 2017 at 5:14 PM, Tim Diels <[email protected]> wrote:
>
> Dear all,
>
>
>
> I'm considering automatically converting the Python packages listed in the
> Python Packaging Index (PyPI) to Zero Install feeds; Python packaging has a
> few problems which Zero Install would solve. One main issue with this is
> that Python packages are installed by executing a package's setup.py
> script. While this file usually is implemented by calling setuptools.setup,
> this is not always the case.
>
>
>
> One approach would be to convert them to source packages. Its compilation
> step would call setup.py and the changes it makes to the system would be
> recorded; just like an AUR package would. Then, a build server should
> compile these source packages into binary packages. By default, the build
> server assumes the binary package will work on any system. This assumption
> can be corrected later, for individual source packages, by manually
> specifying a list of environments to build in/for. Alternatively, if Zero
> Install can conveniently install something with source dependencies these
> days, it might be better not to automatically generate binary packages.
>
>
>
> Has anyone else looked into doing this? Any suggestions?
>
>
>
> Best regards,
>
>
>
> Tim
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Zero-install-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/zero-install-devel
>
>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

_______________________________________________
Zero-install-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/zero-install-devel