Re: mutual authentication fails when FIPS enabled

Hi Petr,

   I would like to clarify my original problem. The mutual 
authentication is between my application using OpenSSL and another 
server. I am using LIBP11 to with OpenSSL to allow OpenSSL to access the 
certificate and private key on my smart card. i.e. OpenSSL is using 
LIBP11 to get the private key and certificate.

When FIPS is enabled, I see the OpenSSL error:

    *error:0409909D:rsa routines:RSA_public_encrypt:non fips rsa method*


Since I am not making any calls directly to LIBP11, how can I fix this 
in my code? i.e. OpenSSL is using LIBP11, not my application code.

This problem was described in the Red Hat bug report:

https://bugzilla.redhat.com/show_bug.cgi?id=1827535  <https://bugzilla.redhat.com/show_bug.cgi?id=1827535>

The solution was to add a flag RSA_FLAG_FIPS_METHOD in the p11_rsa.c 
function:

    RSA_METHOD *PKCS11_get_rsa_method(void)
    {
         static RSA_METHOD *ops = NULL;

         if (!ops) {
             alloc_rsa_ex_index();
             ops = RSA_meth_dup(RSA_get_default_method());
             if (!ops)
                 return NULL;
             RSA_meth_set1_name(ops, "libp11 RSA method");
    *RSA_meth_set_flags(ops, RSA_FLAG_FIPS_METHOD); <-- This flag was
    added to fix the problem*
             RSA_meth_set_priv_enc(ops, pkcs11_rsa_priv_enc_method);
             RSA_meth_set_priv_dec(ops, pkcs11_rsa_priv_dec_method);
             RSA_meth_set_finish(ops, pkcs11_rsa_free_method);
         }
         return ops;
    }


Is the problem related to the actual PIN? If so, how can I fix that?

My OpenSSL dynamic engine is initialized with code to obtain the smart 
card password from the user with getPassword():

    ENGINE_ctrl_cmd_string(engine, "PIN", getPassword(), 0);



Thanks,
George


On 2021-02-01 1:35 p.m., Petr Pisar wrote:
> V Mon, Feb 01, 2021 at 12:39:30PM -0500, George napsal(a):
>> Hi,
>>
>>      I had a closer look at this problem and confirmed that I have
>> successfully enabled FIPS in OpenSSL and am using the FIPS compliant
>> cipher suite TLS_RSA_WITH_AES_128_CBC_SHA.
>> (https://wiki.openssl.org/index.php/FIPS_mode_and_TLS#TLS_1.2)
>>
> That's a cipher suite for TLS (encrypted TCP).
>
>> However, LIBP11 still fails unless I make a modification to the source
>> code in p11_rsa.c. Does this mean there is indeed a bug in LIBP11?
>>
> I think that a mutual authentication in context of libp11 means a "secure"
> authentication of a user to a cryptographic card instead of sending a PIN in
> a "plain" text between the user and the card. I saw a few hints that there is
> used a MD5-based HMAC. And that probably triggers your issue.
>
> -- Petr
>
>
> _______________________________________________
> Opensc-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/opensc-devel

_______________________________________________
Opensc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensc-devel