Re: Migration and slow AFS performance

Hi Dan,

Since no one from the OpenAFS community has replied I will chime in.

On 5/25/2021 10:21 AM, Daniel Mezhiborsky 
([email protected]) wrote:
> Hello all,
> 
> We currently have a relatively small (~250 users, 2TB) AFS cell that I 
> am planning on retiring soon. 

If you are willing to explain, I'm sure the community would appreciate 
hearing the reasons behind the migration away from OpenAFS and how the 
replacement was selected.

> I'd like to get our data out of AFS-space, 
> but this is proving problematic because of performance issues. Our setup 
> is one AFS server VM with iSCSI-attached /vicepa. 

It would be helpful if you could provide more details of the server 
configuration.

1. Is the client-fileserver traffic sharing the same network adapter as 
the iSCSI attached filesystem backing the vice partition?

2. How much network bandwidth is available to the client-fileserver path?

3. How much network bandwidth is available to the fileserver-iSCSI path?

4. What is the network latency (Round Trip Time) between the client and 
fileserver?

5. What version is the OpenAFS fileserver?

6. What command line options has the fileserver been started with?

7. What AFS client version and operating system (and OS version) is the 
client?

8. What are the configuration options in use by the client?

9. What is the destination file system that the rsync client is writing to?

10. Is the destination filesystem also being accessed via the network?

11. What is the OpenAFS fileserver VM configuration?  number of cores, 
amount of RAM, clock speed, etc.

> For single large-file 
> transfers with encryption off, I can get around 30MB/s read/write, but 
> it seems like metadata operations are very slow, so copying our data 
> (mostly home directories) directly from AFS with cp/rsync is taking a 
> prohibitively long time. 

Which metadata operations are slow?

Are you referring to operations reading the metadata from /afs or 
writing it to the destination?

> I see similar slowness with vos dump. We do 
> take regular backups with vos backupsys and backup dump that take about 
> 36 hours for full backups.
> 
> Does anyone have any recommendations on problem areas to try and tune to 
> get better performance or advice on a better way copy our data out of AFS?

You will need to provide more details about the environment before 
specific recommendations can be provided.  However, I will mention a few 
things to consider about the extraction methodology.

First, each new AFS RPC behaves similarly to a new TCP connection.  It 
starts with a minimal congestion window size and grows the window via a 
slow start algorithm until either the maximum window size of 32 * 1412 
packets has been reached or packet loss occurs.   Combined with the RTT 
for the link you can compute the bandwitdh delay product for a single 
call.

In general RPCs issued by an AFS client to read from a fileserver are 
going to be one disk cache chunk size at a time.  Smaller if the average 
file size is less than the chunk size.

If authentication is required to access the directories and files, a 
separate FetchStatus RPC will be issued for most files prior to the 
first FetchData RPC.  At least in an OpenAFS client.

One of the strengths of AFS is the caching client, but in this case 
caching is not beneficial because the data will be read once and 
discarded during this workflow.   The workflow will also be painful for 
an OpenAFS cache manager because of the callback tracking.   If the 
cache recycles before the callback expires, the cache manager will 
cancel the outstanding callbacks.

Likewise, if the fileserver callback table fills then it will need to 
expend considerable effort searching for unexpired callbacks to discard 
early.  Discarding callbacks requires issuing RPCs to the cache manager. 
  So an insufficiently large cache and callback pool can result in the 
callback lifecycle dominating the workflow.

Instead of performing one rsync at a time.  You should consider 
executing multiple rsyncs in parallel potentially from multiple client 
systems.

Provide details of the environment and more specific recommendations can 
be provided.

Jeffrey Altman
AuriStor, Inc.

begin:vcard
fn:Jeffrey Altman
n:Altman;Jeffrey
org:AuriStor, Inc.
adr:;;255 W 94TH ST STE 6B;New York;NY;10025-6985;United States
email;internet:[email protected]
title:CEO
tel;work:+1-212-769-9018
url:https://www.linkedin.com/in/jeffreyaltman/
version:2.1
end:vcard