Design decision about the cache

I have received complaints about the file cache growing and never being
cleaned up.  This has been an item in the todo list for a while, but it
should be handled now.

It's not so simple to decide how to handle this.  Therefore I have
written a design decision for it.  Please take a look at it and give
comments.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
How to cleanup cached files

When files are downloaded they are cached. After using Aap for a while
the cache is filled with files, but they are not cleaned up without
manual intervention. Many people do not even know about the cache and
never clean it up.

Alternatives:

Add a command to cleanup the cache

    The main drawback is that many people do not know this is to be
    done, and people who do know about it might want to run it often,
    upto after each download (if they know the chached files won't be
    used again). That would be annoying. 

Automatically cleanup the cache

    This could be done when Aap is run for downloading and detects that
    some files in the cache are older than a certain age (e.g. a month).
    Since deleting a file doesn't take much time the user will hardly
    notice the extra cleanup action.
    The main drawback is that it is quite unpredictable how long a
    cached file is valid. Some files may never be used a second time,
    some remain valid for years. Since the cached files might not appear
    in the recipe currently being executed, information about the
    usefulness of the file (e.g., the "constant" attribute) is currently
    missing. If that information is available it should be added in the
    cache index file.
    Once this is done the user could set a limit for the size of the
    cache. 

Never use the cache

    This is a simple solution, quite a bit of code can be removed. The
    disadvantage would be that the same file may need to be downloaded
    several times. How often does this occur?

	* Most often an explicit ":fetch" command or using the "fetch"
	  target invokes a download anyway, bypassing the cache.
	* When downloading through a method where the timestamp can't be
	  obtained the cache will not be used, because the remote file
	  may already have changed. Unless the "contstant" attribute is
	  used.
	* A situation where the cache would actually be used is when a
	  file doesn't exist for one user and has already been
	  downloaded by another user. But this requires a directory
	  writable for a group of users, which has security problems
	  (someone could put a trojan horse in the cache), thus can only
	  be used on a trusted system. And never happens on a system
	  with one Aap user.
	* Another situation is when a user cleans up everything and
	  starts building a program all over again.

    Summary: Removing the use of the cache would cause extra downloads
    for some kind of files. This may be a problem for someone who uses
    many of these files, he would have to do some caching manually. 

Only store a file in the cache when useful

    This is a more complicated solution. The principle would be that
    when the downloading is done the file is stored in the cache if the
    cache would be used when the current operation is repeated. When a
    forced fetch is done, the cached file won't be used next time, thus
    don't write it in the cache. When fetching outdated files (using the
    ":mkdownload" command) cached files will also not be used, thus
    don't cache them. What remains are files that are downloaded when
    missing - if the download method allows checking the timestamp - and
    for files with a "constant" attribute. 


Choice: Use a mix: Only store a file in the cache when useful, add a way
to manually cleanup the cache when desired and automatically delete
files when the cache uses more than a certain amount of space. 

-- 
He who laughs last, thinks slowest.

 /// Bram Moolenaar -- [email protected] -- http://www.Moolenaar.net   \\\
///        Sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\              Project leader for A-A-P -- http://www.A-A-P.org        ///
 \\\  Help AIDS victims, buy here: http://ICCF-Holland.org/click1.html  ///


-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn