Performance issues with Git repositories (or in general with many small files workloads)

Ciprian Dorin Craciun <[email protected]>
Newsgroups gmane.comp.file-systems.openafs.general
Message-ID <CA+Tk8fxp6kR1QgPHikzy1xndT6Y1Wu=eYupzyQj5+oN96ygsiA@mail.gmail.com>
Hello all!

I'm trying to use AFS to backup various Git repositories.  By "backup"
actually mean `git push --mirror /afs/.cell/some-path/repository.git`,
which has the following behaviour:  it writes many small files in the
`.git/objects` folder fanned by the first two hex digits of the object
hash.

In fact this pattern can be found in many applications that handle
lots of small files.  For example `rsync`, build systems, etc.
Moreover the pattern I'm describing is single-threaded, as in these
files are not created concurrently by multiple threads / processes.

Unfortunately the performance is abysmal, I mean what should take
perhaps 1-2 seconds on a normal drive it takes perhaps up-to a minute
on AFS;  for example `git-push` reports an bandwidth of only ~20
KiB/s.

Looking at the CPU usage, the `dafileserver` seems to be at ~95%,
although the system has 4 cores and is lightly used.

I can eliminate the following causes:
* network issues (both bandwidth or latency), because this behaviour
occurs even if I mount AFS on the same server where the file server
lives, thus everything happens over loopback;
* encryption -- it is off;
* synchronous close -- I've tried to set `fs storebehind -allfiles
16384 -verbose`;
* disks backing AFS cache -- it's a NVMe disk capable of ~3GiB/s;
* disks backing AFS file server -- it's a RAID5 of 3 top-of-the-line
(Gold) WD S-ATA drives;
* I can achieve good throughput for large files, or if accessing
medium sized files from multiple threads / processes;

My OpenAFS deployment is on Linux 5.3.18, OpenSUSE Leap 15.2, and the
following are the arguments of the file server and cache manager:

~~~~
/usr/lib/openafs/dafileserver -syslog -sync onclose \
-p 128 -b 524288 -l 524288 -s 1048576 -vc 4096 \
-cb 1048576 -vhandle-max-cachesize 32768 \
-udpsize 67108864 -sendsize 67108864 \
-rxpck 4096 -rxmaxmtu 1400 -busyat 65536
~~~~

~~~~
/usr/sbin/afsd -blocks 67108864 -chunksize 17 -files 524288 \
-files_per_subdir 4096 -dcache 524288 \
-stat 524288 -volumes 4096 \
-splitcache 90/10 \
-afsdb -dynroot-sparse -fakestat-all \
-inumcalc md5 -backuptree \
-daemons 8 -rxmaxfrags 8 -rxmaxmtu 1400 \
-rxpck 4096 -nosettime
~~~~

BTW, initially I was using the old `fileserver`-based setup, and
even though I've switched to `dafileserver` the performance seems to
stay unchanged.

Thanks for the help,
Ciprian.
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.