Re: Breakage in ktls-utils with nfs keyring?

On Sun, May 3, 2026, at 9:48 AM, Sagi Grimberg wrote:
> On 02/05/2026 6:08, Chuck Lever wrote:
>>
>> On Fri, May 1, 2026, at 4:19 PM, Scott Mayhew wrote:
>>> On Thu, 30 Apr 2026, Chuck Lever wrote:
>>>
>>>> Cc'ing the ktls-utils development list.
>>>>
>>>> On Thu, Apr 30, 2026, at 9:32 AM, Sagi Grimberg wrote:
>>>>> Hey Chuck,
>>>>>
>>>>> Upstream ktls-utils fails passing client certificate and private key
>>>>> using the .nfs keyring.
>>>>> Bisecting leads commit facd084e43fc ("tlshd: Client-side dual
>>>>> certificate support").
>>>>>
>>>>> I manually apply this (probably wrong) change and keyring works:
>>>>> --
>>>>> diff --git a/src/tlshd/client.c b/src/tlshd/client.c
>>>>> index 2664ffb..a946797 100644
>>>>> --- a/src/tlshd/client.c
>>>>> +++ b/src/tlshd/client.c
>>>>> @@ -327,7 +327,7 @@ tlshd_x509_retrieve_key_cb(gnutls_session_t session,
>>>>>           } else {
>>>>>                   tlshd_log_debug("%s: Selecting x509.certificate from
>>>>> conf file", __func__);
>>>>>                   *pcert_length = tlshd_certs_len;
>>>>> -               *pcert = tlshd_certs + tlshd_pq_certs_len;
>>>>> +               *pcert = tlshd_certs;
>>>>>                   *privkey = tlshd_privkey;
>>>>>           }
>>>>>           return 0;
>>>>> --
>>>>>
>>>>> But, I have a feeling its not the correct change...
>>>>
>>>> Scott, can you triage this?
>>> So when I added the dual certificate support, I didn't touch any of the
>>> keyring code.  Frankly, I'm not entirely sure what is the right way to
>>> set it up and the docs are pretty much nonexistent.  As far as I can
>>> tell:
>>>
>>> - you need to load nfs.ko first so that the .nfs keyring gets created
>>>    via nfs_init_keyring()
>>> - you need to restart tlshd so that it links the .nfs keyring into its
>>>    session keyring (I tried loading nfs.ko at boot via modules-load.d,
>>>    but tlshd still reported an error saying it couldn't find the .nfs
>>>    keyring)
>>> - you need to convert the cert and key to DER format
>>> - you need to add the cert and key to the .nfs keyring, e.g.
>>>
>>>    keyctl padd user "nfs_cert" %:.nfs < smayhew-rawhide.crt.der
>>>    keyctl padd user "nfs_key" %:.nfs < smayhew-rawhide.key.der
>>>
>>> - then you mount w/ '-o xprtsec=mtls,cert_serial=...,privkey_serial=...'
>>>
>>> Is that somewhat accurate?
>
> It is.
>
>>>    Is there a better way to do it?
>
> Have a script/automation SW.

Our intention is to have mount.nfs pick up some of this work.

We need a solution that can pick up a different certificate for each
network namespace, for instance. And we want to enable certificate
storage in the system's TPM someday.

And this needs to be made reliable relative to module load order.


>>>   It seems
>>> like a lot more work than just using the config file.
>
> Well in some cases, storing credentials on a persistent file is not a 
> viable option.
> For nvme there is a userspace utility that helps with this to some extent.

This is because handling an NVMe PSK in the keyring is a first-class,
supported mechanism. Handling the x.509 certificate this way hasn't
really been thought through.

>> It is more work because keyring support for the NFS consumers is still
>> aspirational/experimental.
>
> Can you elaborate? I think people expect to be able to pass certs/keys to
> tlshd the .nfs keyring. Also I expected it to work (as it used to).

The "cert_serial" and "privkey_serial" mount options are not documented
at all in nfs(5). They are intended to be a way for the mount.nfs command
to pass key serial numbers to the kernel NFS client, not as an
administrative interface. Because, yuck.

This doesn't mean we don't want to support using a keyring for x.509
certificates on NFS mounts. It means the capability isn't finished yet.


>> I've pushed your patch to a "fixes" branch for folks to try out. I'm not
>> sure yet whether we want a 1.4.1 release with this fix, since keyring
>> support for NFS is "not finished".
>
> I understand that there are features that are not supported via the
> keyring interface. But I think that users expect things that used to work to
> continue working. My personal opinion is that releasing this fix is 
> appropriate
> given that this is a regression.

You are the first user I know of for this capability.

Yes, technically it's a regression, but it's not really a feature that
is supposed to be ready for users at this stage.

You are also building tlshd from scratch rather than using a distro-
packaged version of it. That's rare enough, but it also means you can
apply the patch that fixes the issue and build it yourself.

I'm open to considering a dot-release, but you haven't convinced me yet.


> Is keyring support for NFS marked as "experimental" or "not finished" 
> anywhere?

Where would we add such a marking? nfs(5) doesn't document either of
those mount options.

Patches and architecture are welcome. As I said, we want this to work
eventually.


-- 
Chuck Lever