Re: Feedback Request: yum-plugin-multithread

On Thu, 18 Feb 2010, Michael Schultz wrote:

> Hello all,
>
> In my "spare time," I wrote a simple yum-plugin that uses the pycurl
> library to download the RPM packages as a predownload hook.  The goal
> is to better use the client connection bandwidth by downloading multiple
> packages at a time.  I looked through the history of this and the dev list,
> but only saw an attempt by someone hacking something into yum itself.  This
> was shot down, with the suggestion to write it as a plugin and make use of
> pycurl to handle the downloads.  I saw no follow-ups, so here I am.
>
> From a high-level, this plugin uses pycurl and downloads all the packages
> max_connections (16) at a time before the yum client processes downloads.
> Once all packages are downloaded, control is returned to yum, which sees
> that it has all the packages and begins the transactions as normal.
>
> Right now the plugin is still in its infancy, so I am looking for quick
> feedback.  You can download the plugin from:
> http://dev.beyond-syntax.com/yum/multithread.py
> (for an example config file just replace .py with .conf).  There are a few
> things I would like to fix up, but haven't figured out how:
>
>  - Better output would be nice, but I'm not sure how to get there.
>  - It should integrate with fastestmirror (since it orders the repos), but
>    it would be nice to integrate with presto too, not sure how.
>  - Reordering downloads, starting biggest ones first and letting smaller
>    ones fill the download during that time.
>  - Distribute the downloads over more than just one mirror (if not already).
>
> In my completely unscientific testing, I took a VM snapshot and ran yum
> without presto and specified '--downloadonly'.  The update was 94 packages,
> totaling 102 Megabytes.  (Both runs use the same mirror.)
>  - Multithread plugin downloaded all packages in 34 seconds.
>  - Normal yum downloaded all packages in 53 seconds.
>
> Obviously, this is strongly dependent on the connection and server
> capabilities, but I would assume any Fedora/RPM mirror will have
> bandwidth >> client bandwidth.  By creating more connections, multithread
> consumes more bandwidth to speed up the overall throughput.
>
> Questions, comments, observations?

Great!

This work notwithstanding - if you'd like to work on adding curlMulti() 
support generally to urlgrabber - you'd be MOST welcome.

-sv