Re: Breakage in ktls-utils with nfs keyring?

On 03/05/2026 22:11, Chuck Lever wrote:
>
> On Sun, May 3, 2026, at 9:48 AM, Sagi Grimberg wrote:
>> On 02/05/2026 6:08, Chuck Lever wrote:
>>> On Fri, May 1, 2026, at 4:19 PM, Scott Mayhew wrote:
>>>> On Thu, 30 Apr 2026, Chuck Lever wrote:
>>>>
>>>>> Cc'ing the ktls-utils development list.
>>>>>
>>>>> On Thu, Apr 30, 2026, at 9:32 AM, Sagi Grimberg wrote:
>>>>>> Hey Chuck,
>>>>>>
>>>>>> Upstream ktls-utils fails passing client certificate and private key
>>>>>> using the .nfs keyring.
>>>>>> Bisecting leads commit facd084e43fc ("tlshd: Client-side dual
>>>>>> certificate support").
>>>>>>
>>>>>> I manually apply this (probably wrong) change and keyring works:
>>>>>> --
>>>>>> diff --git a/src/tlshd/client.c b/src/tlshd/client.c
>>>>>> index 2664ffb..a946797 100644
>>>>>> --- a/src/tlshd/client.c
>>>>>> +++ b/src/tlshd/client.c
>>>>>> @@ -327,7 +327,7 @@ tlshd_x509_retrieve_key_cb(gnutls_session_t session,
>>>>>>            } else {
>>>>>>                    tlshd_log_debug("%s: Selecting x509.certificate from
>>>>>> conf file", __func__);
>>>>>>                    *pcert_length = tlshd_certs_len;
>>>>>> -               *pcert = tlshd_certs + tlshd_pq_certs_len;
>>>>>> +               *pcert = tlshd_certs;
>>>>>>                    *privkey = tlshd_privkey;
>>>>>>            }
>>>>>>            return 0;
>>>>>> --
>>>>>>
>>>>>> But, I have a feeling its not the correct change...
>>>>> Scott, can you triage this?
>>>> So when I added the dual certificate support, I didn't touch any of the
>>>> keyring code.  Frankly, I'm not entirely sure what is the right way to
>>>> set it up and the docs are pretty much nonexistent.  As far as I can
>>>> tell:
>>>>
>>>> - you need to load nfs.ko first so that the .nfs keyring gets created
>>>>     via nfs_init_keyring()
>>>> - you need to restart tlshd so that it links the .nfs keyring into its
>>>>     session keyring (I tried loading nfs.ko at boot via modules-load.d,
>>>>     but tlshd still reported an error saying it couldn't find the .nfs
>>>>     keyring)
>>>> - you need to convert the cert and key to DER format
>>>> - you need to add the cert and key to the .nfs keyring, e.g.
>>>>
>>>>     keyctl padd user "nfs_cert" %:.nfs < smayhew-rawhide.crt.der
>>>>     keyctl padd user "nfs_key" %:.nfs < smayhew-rawhide.key.der
>>>>
>>>> - then you mount w/ '-o xprtsec=mtls,cert_serial=...,privkey_serial=...'
>>>>
>>>> Is that somewhat accurate?
>> It is.
>>
>>>>     Is there a better way to do it?
>> Have a script/automation SW.
> Our intention is to have mount.nfs pick up some of this work.

That would be a nice addition.

> We need a solution that can pick up a different certificate for each
> network namespace, for instance. And we want to enable certificate
> storage in the system's TPM someday.
>
> And this needs to be made reliable relative to module load order.

I support everything you said.

>>>>    It seems
>>>> like a lot more work than just using the config file.
>> Well in some cases, storing credentials on a persistent file is not a
>> viable option.
>> For nvme there is a userspace utility that helps with this to some extent.
> This is because handling an NVMe PSK in the keyring is a first-class,
> supported mechanism. Handling the x.509 certificate this way hasn't
> really been thought through.

What makes NVMe PSK more "supported" than x.509? Both rely on userspace
to create keys using a keyring (either a well-known keyring, or some 
user created keyring)
and pass it to the kernel as parameters (either comma-separate-string to 
/dev/nvme-fabrics
or as a comma-separated-string to mount.nfs). These are not different IMO.

Unlike NFS, the NVMe spec defines standard identities (host and 
subsystem qualified names) and
shared secret format as well as naming conventions. Hence nvme-cli 
provides a nice interface that
does not force the user to open and read the specification. nvme-cli TLS 
helpers primary purpose
is not really to save the user adding a key to a keyring and adding it 
as a parameter to the driver
connection string, that is just a nice by-product afaik.

>
>>> It is more work because keyring support for the NFS consumers is still
>>> aspirational/experimental.
>> Can you elaborate? I think people expect to be able to pass certs/keys to
>> tlshd the .nfs keyring. Also I expected it to work (as it used to).
> The "cert_serial" and "privkey_serial" mount options are not documented
> at all in nfs(5).

I thought this was an oversight?

>   They are intended to be a way for the mount.nfs command
> to pass key serial numbers to the kernel NFS client, not as an
> administrative interface. Because, yuck.

I agree this can be made nicer with passing a key identity perhaps (although
it can create also some annoyance with how flexible key identities can be).
In NVMe both options are supported.

The way I see it, use of a keyring most likely mean users rely on some
automation software to populate it anyways.

> This doesn't mean we don't want to support using a keyring for x.509
> certificates on NFS mounts. It means the capability isn't finished yet.

I understand. But it does exist and I know of multiple users of it.

>>> I've pushed your patch to a "fixes" branch for folks to try out. I'm not
>>> sure yet whether we want a 1.4.1 release with this fix, since keyring
>>> support for NFS is "not finished".
>> I understand that there are features that are not supported via the
>> keyring interface. But I think that users expect things that used to work to
>> continue working. My personal opinion is that releasing this fix is
>> appropriate
>> given that this is a regression.
> You are the first user I know of for this capability.

The first that you know :)
Definitely not the only one.

> Yes, technically it's a regression, but it's not really a feature that
> is supposed to be ready for users at this stage.
>
> You are also building tlshd from scratch rather than using a distro-
> packaged version of it. That's rare enough, but it also means you can
> apply the patch that fixes the issue and build it yourself.

This breakage was brought to my attention by a user working on
Ubuntu24.04 ktls-utils (1.3.0). It'd be better if we'd caught it sooner...

> I'm open to considering a dot-release, but you haven't convinced me yet.

Ultimately it's your call Chuck. But IMO we shouldn't hold out a fix
for this until we are happy with a nicer mount.nfs interface.

>> Is keyring support for NFS marked as "experimental" or "not finished"
>> anywhere?
> Where would we add such a marking? nfs(5) doesn't document either of
> those mount options.

Yea, I agree.

> Patches and architecture are welcome. As I said, we want this to work
> eventually.

 From my perspective (up until this conversation at least) it used to work.
I was not aware of any dissatisfaction with the interface.

Anyways, would be happy to contribute to this (don't know anything
about the pq stuff though)...