Re: Making CIFS FS-Cache capable

On Tue, 12 May 2009 10:31:20 +0100
David Howells <[email protected]> wrote:

> 
> Hi Steve,
> 
> To make CIFS FS-Cache capable, we need to consider a number of things:
> 
>  (1) We need an index hierarchy through which we can find cached data when a
>      file is reopened after being dropped from the inode cache.
> 
>  (2) We need coherency data so that we can tell whether the cache for a file
>      we're reopening is still valid.
> 
>  (3) We need a way to tell CIFS that it should use FS-Cache for a particular
>      mount.
> 
> 
> INDEX HIERARCHY
> ===============
> 
> As far as (1) goes, that doesn't need to be a very deep hierarchy.  NFS's has
> three levels:
> 
>       - Server (nfs_fscache_server_index_def)
>       - Exported FS (nfs_fscache_super_index_def)
>       - Inode (nfs_fscache_inode_object_def)
> 
> The objects of the first type are keyed by NFS version, address and port;
> objects of the second type are keyed by the FSID and various mount parameters,
> including an optional uniquifier string; and objects of the third type are
> keyed on NFS file handles.
> 
> AFS, on the other hand, has a four level hierarchy:
> 
> 	- Cell (afs_cell_cache_index_def)
> 	- Volume location (afs_vlocation_cache_index_def)
> 	- Volume instance (afs_volume_cache_index_def)
> 	- Inode (afs_vnode_cache_index_def)
> 
> Keyed by cell name, volume name, volume type and vnode ID respectively.
> 
> 
> For CIFS, (1) might be a little more complicated.  We need to identify a set of
> files, and then the files within that set.  Perhaps a three level hierarchy:
> 
> 	- Server
> 	- Share
> 	- File
> 
> Where server might be the IP address and name of a server, and share may be the
> name of a share on that server.  Files are more interesting, since, as far as I
> know, the primary way of identifying files on a server is by filename.
> 
> However, we could limit cacheable CIFS files to just those that have a
> UniqueId.
> 
> Failing that, we'd have to create an index for every directory, and install the
> files into the appropriate directory index.  Note that it is permissible to (a)
> mix different types of objects within an index, and (b) have coherency data
> attached to an index.  Files and directories would then be keyed by their
> names.
> 
> 
> COHERENCY DATA
> ==============
> 
> With respect to coherency data, NFS uses i_size, i_mtime and i_ctime plus
> change_attr if it's available.  AFS uses the FID uniquifier and the data
> version number.
> 
> CIFS could use CreationTime, LastModificationTime/LastWriteTime,
> LastStatusChange/ChangeTime, NumOfBytes and EndOfFile.
> 
> 
> CONFIGURING CIFS
> ================
> 
> The obvious way to enable FS-Caching for a CIFS mount would be to add a mount
> option in a similar way to NFS (NFS has -o fsc or -o fsc=<uniquifier>).
> 
> Finer grained caching configuration is something on my to-do list, and will
> probably be available on two levels:
> 
>  (a) By directory (where the config is inherited by the children of that dir).
> 
>  (b) By simple pattern match on names of regular files.
> 
> 
> Thoughts?

(cc'ing JRA here too)

A couple...

First, the patches I posted the other day to fix up the hardlink
handling should make this work a bit easier I think since they clean up
the way that inodes are created and managed within CIFS.

Second, we have a bit of a dilemma with CIFS here. Currently, CIFS
caches data even when it doesn't have an oplock and does a very
NFS-like thing to revalidate it -- it checks the LastWriteTime (aka
mtime) and the size. If those have changed then it invalidates the
cache. This is all well and good but for one small problem that JRA
pointed out to me:

If a client sets the LastWriteTime on a file, then most CIFS servers
(windows and samba included) will not update the mtime on the file for
subsequent writes to that filehandle (up until the fh is closed). This
makes it impossible to reliably detect changes to a file based on
LastWriteTime changes.

The only reliable way to work with cached data in CIFS apparently is to
hold an oplock, but that's only good as long as you have a file open.

In practice, this is rarely an issue. Changing mtime is usually
only done by stuff like "tar" or "cp" and in those cases, concurrent
access by other machines isn't common, but it is a current data
corruption exposure in CIFS.

One small worry -- could fscache make that more of a problem due to the
fact that the cache may be persistent across reboots?

-- 
Jeff Layton <[email protected]>