download strategy, was Re: apt, yum, up2date, smartpm, ... ?

Good evening, Les,

On Wed, 21 Sep 2005, Les Mikesell wrote:

> On Wed, 2005-09-21 at 14:53, William Stearns wrote:
>
>>> Altogether, yum and utils has been one of the more infuriating set of
>>> packages I've ever used, and seems to have been designed with barely a
>>> thought for anyone not on a nice fast permanent connection.
>>
>>  	Yum-pull is the glue you need to handle that problem, IMHO.
>
> Does it know to only download the packages you need or are you
> trading being proxy-cache-unfriendly for making copies of
> stuff you won't ever need.

 	For the repositories and distributions you request it defaults 
to getting all of their files.  That said, you can:

- place any files you already have in the tree in advance so they're not 
downloaded again

- limit bandwidth used

- run yum-pull on off-hours to reduce the impact on normal operations

- place fileglobs of packages you don't want to download in the 
"user-exclude" file in the top of the tree.  For example, I might know 
I'll never use the kde or openoffice internationalizations, so I add these 
lines:
openoffice.org-i18n-*
kde-i18n-*


 	In an environment where multiple people are using the yum-pull 
repository server, it seems to me there are two possible approaches:

- download all files in advance.  This is yum-pull's strategy; it uses 
bandwidth for files that may never be used, but means that all packages 
and package requirements will be available even if the line or Internet 
servers are down.

- only download needed files on demand.  This is what yum does by default, 
but suffers from the problems discussed in this thread.

 	If it's simply a matter of wanting to avoid duplicate downloads, 
it would seem that using proxy such as squid would avoid duplicate 
downloads from a given internet server; the "proxy" option in the "[main]" 
section of /etc/yum.conf would allow you to specify the location of the 
squid proxy to use (man yum.conf).  One consideration is that squid only 
caches objects up to a certain size.  From /etc/squid/squid.conf:

#  TAG: maximum_object_size     (bytes)
#       Objects larger than this size will NOT be saved on disk.  The
#       value is specified in kilobytes, and the default is 4MB.  If
#       you wish to get a high BYTES hit ratio, you should probably
#       increase this (one 32 MB object hit counts for 3200 10KB
#       hits).  If you wish to increase speed more than your want to
#       save bandwidth you should leave this low.


 	Here's an open question.  If we're not getting all packages in 
advance like yum-pull, but we want to minimize the bandwidth used, is 
there an approach better than "let squid cache requests so the second 
workstation doesn't have to download the file again"?  A couple of people 
have used phrases like your "only download the packages you need", and I'd 
be sincerely interested to hear about ways to do that are friendly to 
modem-based users.  Is there a third option that's an alternative to "get 
all files in advance" and "wait for the files to come down after the user 
requests a package install or upgrade"?
 	Would "download only new versions of packages I currently have 
installed" be the third option you want?  That would speed up patches, but 
wouldn't help with installing new software and might run into problems if 
a patch rpm requires libraries you don't currently have installed.
 	Cheers,
 	- Bill

---------------------------------------------------------------------------
 	"Absence of evidence is not evidence of absence."
 	-- SETI, the Search for Extra-Terrestrial Intelligence
--------------------------------------------------------------------------
William Stearns ([email protected]).  Mason, Buildkernel, freedups, p0f,
rsync-backup, ssh-keyinstall, dns-check, more at:   http://www.stearns.org
--------------------------------------------------------------------------