Re: Zero-byte file in cache

Ryan,

> On Jan 12, 2020, at 3:03 PM, Ryan C. Underwood <[email protected]> wrote:
> 
> I have a reproducible scenario where the cache manager populates the
> cache with a zero-byte file entry.  This happens when populating the
> cache with a large directory tree of a web application.  The client
> and fileserver communicate over the internet, so sporadic packet loss
> may be implicated.

If you can supply a zipped directory tree (or an AFS volume dump) that reproduces
the problem, others can try to reproduce this under a debugger.

> 
> The only resolution I have found so far is to stop the AFS client,
> delete the cache and restart it and attempt cache population again.

It's worth trying 'fs flushall' as a workaround if you haven't tried that yet.
If it works, it will save you a client restart.  And if it works, you could
try narrowing things down a bit with 'fs flushvol' or 'fs flush'.

> Without going into too much detail immediately, is there a recommended
> way to go about debugging this sort of issue?  

Some type of trace might be helpful, certainly.  What's your host operating
system?  Presuming a Linux distro, systemtap may be helpful.   There's also
the OpenAFS built-in fstrace, which is sometimes helpful, sometimes not so much.
But I think attempting to duplicate the problem elsewhere will get answers more quickly. 
I could supply some trace suggestions if no one else can reproduce this problem.

> This particular issue
> has been observed with 1.6.x clients and with the most recent 1.8
> clients as well.

Which 1.8.x clients have you tried?  There are some fixes in 1.8.3,
and possibly 1.8.4, that may be relevant to this problem.
I'm thinking specifically of 'afs: Do not ignore errors in afs_CacheFetchProc' in 1.8.3.

I also see a few candidates that have been merged to master but haven't yet been ported to 1.8.x:
13436 afs: Check dcache size when checking DVs
13435 afs: Introduce afs_IsDCacheFresh

Regards,
--
Mark Vitale
Sine Nomine Associates
20 Years of Customer Success