Re: [PATCH v4 4/6] xfs: generic AG set based steering

On 7/21/2026 8:50 AM, Darrick J. Wong wrote:
> On Fri, Jul 17, 2026 at 06:25:36PM +0530, Kanchan Joshi wrote:
>> Improve allocator concurrency and reduce interleaving by introducing
>> fixed sized AG set.
>> Use low bits of the inode as a hash to select AG within the AG set.
>> Overall, a file will try to use the same AG (and contiguity is maintained),
>> but multiple files will be spread across all AGs in the target AG set.
>>
>> Suggested-by: Dave Chinner<[email protected]>
>> Signed-off-by: Kanchan Joshi<[email protected]>
>> Signed-off-by: Anuj Gupta<[email protected]>
>> ---
>>   fs/xfs/libxfs/xfs_bmap.c | 38 ++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 38 insertions(+)
>>
>> diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c
>> index d64defeda645..fd1a3aa4ad3f 100644
>> --- a/fs/xfs/libxfs/xfs_bmap.c
>> +++ b/fs/xfs/libxfs/xfs_bmap.c
>> @@ -3192,6 +3192,36 @@ xfs_bmap_select_minlen(
>>   	return args->maxlen;
>>   }
>>   
>> +#define	GENERIC_AG_SET_SZ	(2)
> What does this define?

We need more than 1 AG to do inode-based fanning of files and improve 
concurrency (AGF lock). This is the minimum required to do any 
improvement, in the default allocator.
This 'minimum fan-out' is what Dave suggested to extract out of 
write-stream and apply to two other allocators- default (which is what 
you see now), and filestream (future work).

And this can be reviewed as an independent patch as it does not require 
any other work done in the series. No application input is required, 
this is automatic optimization.

The write-stream allocator (next patch), uses a different AG-set of size 
NR_AGs/NR_WRITE_STREAMs. And that is an explicit (application input 
based) allocator.

  >> +
>> +static inline xfs_agnumber_t
>> +xfs_default_ag_set_size(
>> +	struct xfs_inode	*ip)
>> +{
>> +	struct xfs_mount	*mp = ip->i_mount;
>> +
>> +	return min_t(xfs_agnumber_t, GENERIC_AG_SET_SZ, mp->m_sb.sb_agcount);
> Because I'm not sure what it means on a single-AG filesystem.

On a single-AG filesystem, this will do nothing. The base-AG input given 
by the default allocator will not get changed.

But we get improvements on multi-AG filsystem; not sure if you saw the 
numbers in the cover letter. Corresponding to the 'generic AG-set' below:

1. On regular NVMe

a. Inter-stream concurrency
---------------------------
fio: 4k write, direct IO, 16 jobs, 1 directory, 16 files * 8GiB, iodepth 32
xfs: 16 AGs, 4 write-streams

base: 41 KIOPS
generic AG-set: 93 KIOPS (+126%)
write-stream AG-set: 227 KIOPS (+453%)
here, 16 files are assigned 4 unique write-streams (4 files/stream)

b. Intra-stream concurrency
----------------------------
fio: 4k write, direct IO, 4 jobs, 1 directory, 4 files * 8GiB, iodepth 32
xfs: 16 AGs, 4 write-streams, generic AG-set size = 2, write-stream 
AG-set size = 4

base: 59 KIOPS
generic AG-set: 94 KIOPS (+59%)
write-stream AG-set: 112 KIOPS (+89%)
here, 4 files are assigned single write-stream


>> +}
>> +
>> +static xfs_agnumber_t
>> +xfs_ag_to_ag_set(
>> +	struct xfs_bmalloca	*ap,
>> +	xfs_agnumber_t		base_agno)
>> +{
>> +	struct xfs_inode	*ip = ap->ip;
>> +	struct xfs_mount	*mp = ip->i_mount;
>> +	xfs_agnumber_t		set_size;
>> +
>> +	/* Apply fanning only for regular file data */
>> +	if (!(ap->datatype & XFS_ALLOC_USERDATA))
>> +		return base_agno;
>> +
>> +	set_size = xfs_default_ag_set_size(ip);
>> +	/* Fan out within the AG set using low bits of the inode */
>> +	return (base_agno + (XFS_INO_TO_AGINO(mp, I_INO(ip)) % set_size)) %
>> +		mp->m_sb.sb_agcount;
>> +}
>> +
>>   static int
>>   xfs_bmap_btalloc_select_lengths(
>>   	struct xfs_bmalloca	*ap,
>> @@ -3587,8 +3617,16 @@ xfs_bmap_btalloc_best_length(
>>   {
>>   	xfs_extlen_t		blen = 0;
>>   	int			error;
>> +	xfs_agnumber_t		target_ag, start_ag;
>>   
>>   	ap->blkno = XFS_INODE_TO_FSB(ap->ip);
>> +
>> +	/* fan out initial AG across the generic AG set */
>> +	start_ag = XFS_FSB_TO_AGNO(args->mp, ap->blkno);
>> +	target_ag = xfs_ag_to_ag_set(ap, start_ag);
>> +	if (target_ag != start_ag)
>> +		ap->blkno = XFS_AGB_TO_FSB(args->mp, target_ag, 0);
> /me wonders, if xfs_bmap_rtalloc looked at ap->blkno for a hint the way
> that the data device allocator does, then would it be trivial to have
> write streams on the rt device too?
Trying to understand if this kind of fanning out (this patch) or the 
logical placement (patch 5 with write-stream based AG placement) help us 
solve anything for rt device?
For regular RT device, the round-robin rotor will distribute files into 
multiple RTGs. So we don't have AGF like contention.
And using write-stream to explicitly send multiple files to a single RTG 
will do more harm than good?

And for zoned RT device, the placement is happening via write-hint.

Am I missing something that you have in mind?

All the above is for logical isolation. For physical isolation, your 
comments already help to establish that write-stream can be made to 
work. For the regular RT device sitting on top of NVMe (FDP-enabled). 
That needs few changes, but in previous patch (#3).


> I guess the tricky part would be figuring out what to do if you ever
> want to switch a file between rt and data devices -- presumably you'd
> just reset the write stream id to the default, but I guess you could
> reject such a switch if the id had been set explicitly.

Yes, we do such checks in patch 3 now also.