[SPDK] Re: NVMe Multipath subsystem controller IDs

> -----Original Message-----
> From: Jonathan Teh <jonathan.teh(a)mayadata.io>
> Sent: Thursday, April 1, 2021 2:49 AM
> To: spdk(a)lists.01.org
> Subject: [SPDK] NVMe Multipath subsystem controller IDs
> 
> I'm looking at using Linux's native NVMe multipathing with SPDK's nvmf target
> where multiple SPDK instances are used.
> 
> As the subsystem issues controller IDs in the range [1, 0xFFEF], the initial
> connection to each individual target ends up with controller ID 1, leading to a
> clash for the connection to the 2nd, and subsequent, targets requiring multiple
> connections to "cycle" to the non-clashing ID.
> 
> This issue that was previously discussed in
> https://lists.01.org/hyperkitty/list/spdk(a)lists.01.org/thread/ADLA6IPHVGV5LW
> 2AXZUBWLCYQW3HGOND/
> but I don't think a resolution was agreed upon.

Exposing the same subsystem from two nodes is still certainly problematic. I think all of the
issues I listed in that thread are still valid (reservations, discovery, etc.) A better solution is likely
namespaces shared across subsystems, but that is not in the NVMe specification today.

> 
> One approach would be to limit the controller ID range per subsystem (which
> was also previously proposed) which was implemented for the Linux kernel
> NVMe target in 5.7 as
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9
> 4a39d61f80fcd679debda11e1ca02b88d90e67e.
> 
> 
> Would you consider a pull request to implement a similar feature in SPDK? An
> external orchestrator could then assign the subsystem in each SPDK instance a
> non-overlapping controller ID range.

Could we make an API in include/spdk/nvmf.h to allow the user to register a function pointer for generating
the controller ids? Maybe just stick a function pointer into spdk_nvmf_target_opts. Then you can use whatever
algorithm you want. How does that sound?

> 
> Thanks.
> 
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org