Re: mutual authentication fails when FIPS enabled

I tracked the problem down to the code in OpenSLL, which does the RSA 
encryption. I had a look at how the LIBP11 function 
PKCS11_get_rsa_method(void) is being used in OpenSSL and it appears that 
it will *never* work when FIPS is enabled. There seems to be a problem 
with the logic.

For example, the OpenSSL function to handle RSA encryption is 
openssl-1.0.2u\crypto\rsa\rsa_crpt.c :

    int RSA_public_encrypt(int flen, const unsigned char *from, unsigned
    char *to, RSA *rsa, int padding)
    {
    #ifdef OPENSSL_FIPS
         if (FIPS_mode() && *!(r**sa->meth->flags &
    RSA_FLAG_FIPS_METHOD**)*     <--- rsa->meth->flags = 0, and
    RSA_FLAG_FIPS_METHOD = 0x400
             && *!(rsa->flags & RSA_FLAG_NON_FIPS_ALLOW)*)
    {                 <--- rsa->flags = 6, and RSA_FLAG_NON_FIPS_ALLOW =
    0x400
             RSAerr(RSA_F_RSA_PUBLIC_ENCRYPT, RSA_R_NON_FIPS_RSA_METHOD);
             return -1;
         }
    #endif
         return (rsa->meth->rsa_pub_enc(flen, from, to, rsa, padding));
    }


The first part of the logic in the if-check is "!(rsa->meth->flags & 
RSA_FLAG_FIPS_METHOD)" and will always result in "true" because 
"rsa->meth->flags" was set to 0 in the LIBP11:

    SA_METHOD *PKCS11_get_rsa_method(void)
    {
    . . .
             RSA_meth_set_flags(ops, 0);
    . . .
    }


The second part of the if-check "!(rsa->flags & 
RSA_FLAG_NON_FIPS_ALLOW)" is also "true" because I do not allow non-FIPS 
methods. i.e. RSA_FLAG_NON_FIPS_ALLOW is not set.

Therefore, the following FIPS error is generated:

    *error:0409909D:rsa routines:RSA_public_encrypt:non fips rsa method*


There does not appear to be any success path for 
RSA_public_encrypt(...), based on the current logic. Is this the correct 
behaviour, or have I missed something?



Thanks,
George


On 2021-02-02 1:50 a.m., Petr Pisar wrote:
> V Mon, Feb 01, 2021 at 10:15:43PM -0500, George napsal(a):
>>     I would like to clarify my original problem. The mutual
>> authentication is between my application using OpenSSL and another
>> server. I am using LIBP11 to with OpenSSL to allow OpenSSL to access the
>> certificate and private key on my smart card. i.e. OpenSSL is using
>> LIBP11 to get the private key and certificate.
>>
>> When FIPS is enabled, I see the OpenSSL error:
>>
>>      *error:0409909D:rsa routines:RSA_public_encrypt:non fips rsa method*
>>
>>
>> Since I am not making any calls directly to LIBP11, how can I fix this
>> in my code? i.e. OpenSSL is using LIBP11, not my application code.
>>
> There can be many places which violate FIPS. Your (server or client) X.509
> certificate can use a weak digest algorithm. The TLS cipher suite can use
> a weak algorithm. Your PKCS11 driver can use a weak algorithm when loging into
> a smart card. And probably other places. I recommend you to split your code
> into smaller pieces: Establishing a TLS connection without the smart card.
> Logging into the smart card without making a TLS connection. Making TLS
> connection using keys on the card without libp11 library (libp11 is only an
> abstraction layer. You can use OpenSSL engines directly.) And find out where
> the problem exactly is.
>
>> Is the problem related to the actual PIN? If so, how can I fix that?
>>
> The problem is not a value of the PIN. It might be a way how the card driver
> logs into the card with the PIN. If it is so, you would have to look into the
> PKCS11 driver of your card.
>
> -- Petr
>
>
> _______________________________________________
> Opensc-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/opensc-devel

_______________________________________________
Opensc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensc-devel