Re: mutual authentication fails when FIPS enabled

I just realized that the LIBP11 function RSA_public_encrypt(...) ends up 
using a function from the OpenSSL FIPS 2.0.16 module to do the RSA 
encryption. Does that mean I can safely configure RSA_FLAG_FIPS_METHOD 
to be "true" in

    SA_METHOD *PKCS11_get_rsa_method(void)
    {
    . . .
             RSA_meth_set_flags(ops, *RSA_FLAG_FIPS_METHOD*);
    . . .
    }

?


I traced the code for

    int RSA_public_encrypt(int flen, const unsigned char *from, unsigned
    char *to, RSA *rsa, int padding)
    {
    . . .
         return (rsa->meth->rsa_pub_enc(flen, from, to, rsa, padding));
    }

down to the code in openssl-fips-2.0.16\crypto\rsa\rsa_eay.c :

    static int RSA_eay_public_encrypt(int flen, const unsigned char
    *from, unsigned char *to, RSA *rsa, int padding)

Since we know this if FIPS certified, does this imply the RSA encryption 
function configured in PKCS11_get_rsa_method(void) is also FIPS compliant?



Thanks,
George


On 2021-02-02 12:11 p.m., George wrote:
> I tracked the problem down to the code in OpenSLL, which does the RSA 
> encryption. I had a look at how the LIBP11 function 
> PKCS11_get_rsa_method(void) is being used in OpenSSL and it appears 
> that it will *never* work when FIPS is enabled. There seems to be a 
> problem with the logic.
>
> For example, the OpenSSL function to handle RSA encryption is 
> openssl-1.0.2u\crypto\rsa\rsa_crpt.c :
>
>     int RSA_public_encrypt(int flen, const unsigned char *from,
>     unsigned char *to, RSA *rsa, int padding)
>     {
>     #ifdef OPENSSL_FIPS
>         if (FIPS_mode() && *!(r**sa->meth->flags &
>     RSA_FLAG_FIPS_METHOD**)*     <--- rsa->meth->flags = 0, and
>     RSA_FLAG_FIPS_METHOD = 0x400
>             && *!(rsa->flags & RSA_FLAG_NON_FIPS_ALLOW)*)
>     {                 <--- rsa->flags = 6, and RSA_FLAG_NON_FIPS_ALLOW
>     = 0x400
>             RSAerr(RSA_F_RSA_PUBLIC_ENCRYPT, RSA_R_NON_FIPS_RSA_METHOD);
>             return -1;
>         }
>     #endif
>         return (rsa->meth->rsa_pub_enc(flen, from, to, rsa, padding));
>     }
>
>
> The first part of the logic in the if-check is "!(rsa->meth->flags & 
> RSA_FLAG_FIPS_METHOD)" and will always result in "true" because 
> "rsa->meth->flags" was set to 0 in the LIBP11:
>
>     SA_METHOD *PKCS11_get_rsa_method(void)
>     {
>     . . .
>             RSA_meth_set_flags(ops, 0);
>     . . .
>     }
>
>
> The second part of the if-check "!(rsa->flags & 
> RSA_FLAG_NON_FIPS_ALLOW)" is also "true" because I do not allow 
> non-FIPS methods. i.e. RSA_FLAG_NON_FIPS_ALLOW is not set.
>
> Therefore, the following FIPS error is generated:
>
>     *error:0409909D:rsa routines:RSA_public_encrypt:non fips rsa method*
>
>
> There does not appear to be any success path for 
> RSA_public_encrypt(...), based on the current logic. Is this the 
> correct behaviour, or have I missed something?
>
>
>
> Thanks,
> George
>
>
> On 2021-02-02 1:50 a.m., Petr Pisar wrote:
>> V Mon, Feb 01, 2021 at 10:15:43PM -0500, George napsal(a):
>>>     I would like to clarify my original problem. The mutual
>>> authentication is between my application using OpenSSL and another
>>> server. I am using LIBP11 to with OpenSSL to allow OpenSSL to access the
>>> certificate and private key on my smart card. i.e. OpenSSL is using
>>> LIBP11 to get the private key and certificate.
>>>
>>> When FIPS is enabled, I see the OpenSSL error:
>>>
>>>      *error:0409909D:rsa routines:RSA_public_encrypt:non fips rsa method*
>>>
>>>
>>> Since I am not making any calls directly to LIBP11, how can I fix this
>>> in my code? i.e. OpenSSL is using LIBP11, not my application code.
>>>
>> There can be many places which violate FIPS. Your (server or client) X.509
>> certificate can use a weak digest algorithm. The TLS cipher suite can use
>> a weak algorithm. Your PKCS11 driver can use a weak algorithm when loging into
>> a smart card. And probably other places. I recommend you to split your code
>> into smaller pieces: Establishing a TLS connection without the smart card.
>> Logging into the smart card without making a TLS connection. Making TLS
>> connection using keys on the card without libp11 library (libp11 is only an
>> abstraction layer. You can use OpenSSL engines directly.) And find out where
>> the problem exactly is.
>>
>>> Is the problem related to the actual PIN? If so, how can I fix that?
>>>
>> The problem is not a value of the PIN. It might be a way how the card driver
>> logs into the card with the PIN. If it is so, you would have to look into the
>> PKCS11 driver of your card.
>>
>> -- Petr
>>
>>
>> _______________________________________________
>> Opensc-devel mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/opensc-devel
>

_______________________________________________
Opensc-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensc-devel