Re: Problem sharing apt cache through NFS

Em Tue, Jul 20, 2004 at 07:02:28AM -0300, Andre Costa escreveu:
> Hi Sven,
> 
> On Tue, 20 Jul 2004 10:22:36 +0200
> Sven Hoexter <[email protected]> wrote:
> 
> > On Mon, Jul 19, 2004 at 02:18:10PM -0300, Andre Costa wrote:
> > > Hi,
> > > 
> > > this is more a basic UNIX question, but I am pretty sure you guys will
> > > be able to help me out with it: we would like to share
> > > /var/cache/apt/archives among a group of machines, with one of them
> > > acting as the "master repository" from which the rest would mount this
> > > directory, so that we would only need to download packages once.
> > > 
> > > PS: Is there any better way of sharing apt-get downloads?


Try this package, based lua scripting, by Andre Ruiz.

http://espelhos.edugraf.ufsc.br/conectiva/10/i386/RPMS.extras/apt-netcache-1.0-54567cl.noarch.rpm

its README:

Maintaining a network cache for Apt
-----------------------------------

by Andre Ruiz <[email protected]>
Wed Mar 10 03:24:26 BRT 2004


It is very common to have a network with many machines running a
apt-enabled distribution of linux like Conectiva Linux, and depend on
external repositories to upgrade packages on all those hosts. If more than
one machine on that network is running the same distribution release, it's
expected that they all benefit from a single download, but it's not what
happens without a little effort of the sysadmin.

After many nightmares with "scp"ing the cache from machine to machine in my
small home network every day, I started searching for a more robust solution
to this problem. In my research I analised apt-proxy and apt-move, both
very interesting projects, but they were lacking simplicity or were bloated
with functions I did not really thought were needed.

Enters "rpm-dir" and "lua scripting". If you don't know what those mean,
you should read about them at https://moin.conectiva.com.br/AptRpm
Conectiva leaded the effort to port apt-get to use rpm backend, and it's
still actively maintained and improved. Two of those improvements are:

rpm-dir: the ability to use single directories as repositories, or, in
other words, you point sources.list to a directory with a bunch of RPM
files and apt will use them if possible. As easy as that.

lua scripting: the ability to run scripts that are plugged to "slots", or,
in other words, you can run scripts driven by a number of events you choose

So, here is the hole idea:

1) you create and export via nfs some directory, any name, any place, it
just need to be reachable (at least for reading) to anyone intereted in
participating; let's call this directory the "netcache"

2) every machine will look in this netcache before looking in any other
source (you set an rpm-dir entry on sources.list on them, pointing to
this dir, see examples below)

3) if that machine wants to download a package it has not found in the
netcache, it will do so and right after downloading it will copy this
file to the netcache (if it has write access); this is done by the lua
script called netcache-copy.lua

NOTE: You are not forced to use NFS here, it's just the easiest to setup.
If you want another file sharing system, go ahead, use another method in
the "rpm-dir" entry and do whatever is necessary to make the lua script
copy the locally downloaded file to that place. If you do so, please send
me a message and the modifications :)

Even if you download the files in "download only" mode, they will be copied
to cache. The script plugs in slot "Scripts::Acquire::Archive::Done", which
runs just after the download finishes. So, if you download everything in
download only mode and "apt-get clean" right after, it will still use the
cached files next time. Other hosts will also be able to use the cached
files just after they are downloaded by any host on the network.

Install
-------

As said above, the instalation is simple.

1) Set up a central place to hold the files. Give write access to those you
trust (they could erase your cache, which could be mean more downloads and
more money to spend). Mount this directory in all the workstations. I use
automount, but you can use a static mount. It's not needed to be mounted in
the same path in all hosts, you'll just have to adjust the config files on
each machine accordingly. In the next examples, I used the dir:

/mnt/storage/system/apt-cache/RPMS.cache

This is exported in the server and mounted in exactly the same dir on the
workstations. The path is composed by the following parts:

/mnt/storage/system   <----  this could be any path, you choose
apt-cache   <--------------  this is what we'll call our "distribution"
RPMS.cache   <-------------  this is the repository (or "component")

You may create separate RPMS.foobar for each group of machines you want to
separate from each other (like groups of distro releases, for example), but
I prefer to let them all together. That could be:

/mnt/storage/system/apt-cache/RPMS.cl8
/mnt/storage/system/apt-cache/RPMS.cl9
/mnt/storage/system/apt-cache/RPMS.cl10

And each machine would write/read in the cache directory that represents
it. Of course the names are examples.

2) On every host, configure sources.list so that it will check this
directory for rpms before the external repositories. This can be done
adding a line like this, before the others:

rpm-dir file:///mnt/storage/system apt-cache cache

As you may know, the above line is concatenated to form the real path where
the RPM files are, with the last word being prepended with "RPMS.", so:

	file:///mnt/storage/system apt-cache cache

	turns in

	/mnt/storage/system/apt-cache/RPMS.cache

3) Configure each host to write the downloaded files to the cache, if they
can write there. To do that, you just adjust the options in the apt
configuration file "/etc/apt/apt.conf.d/netcache.conf". They are:

APT::Get::NetCache::Copy "true";

This option turns on/off the execution of the lua script. For those
curious, the script is located at "/usr/lib/apt/scripts/netcache-copy.lua".
This is set to "false" by default.

APT::Get::NetCache::Verbose "true";

This option makes the script be verbose and say which file is being copied
to which place, so you can be sure if and when it's doing it (and see if
there are eny error messages following it).

Dir::NetCache::Root "/mnt/storage/system/apt-cache/RPMS.cache";

Here you specify the netcache directory, the same as you exported (or one
of them, in case you chose to have many). The script will run an external
"cp" command to this directory, so nothing fancy here, it's a path.


Using the system
----------------

The usage should be almost transparent, if you have set verbose mode off.
The only difference that should be noted is that files will not be
downloaded again if they are already in cache.

You can run apt-get update / upgrade on each machine as allways, and you
can run an apt-get clean just after finishing each one, the files are
already in the netcache.


Erasing the cache
-----------------

In this system, there is not an easy way to flush the central cache. Which
can be a good thing, depending on your case. The goal is that only the
administrator should care about that. Every user can "clean" their personal
machine, and the admin can "rm -rf *" in the cache whenever he thinks it's
old enough or when he knows every machine has already upgraded and then is
a good time to flush it. Generally the cache will not grow too fast so to
become a problem.

If you have a sugestion, please send it to me. I'll try to improve this on
the next versions. One could always mount that netcache dir at
/var/cache/apt/archives and use an "apt-get autoclean" to keep things under
control, but it's too much work yet. And this does not work well if you mix
more than one distro release together (autoclean would clean up-to-date
files from older distros).

eof