Bug in pkey_rsa_encrypt() and _decrypt()

Working on pkcs11 engine, I discovered a bug in crypto/rsa/rsa_pmeth.c in pkey_rsa_encrypt() and pkey_rsa_decrypt().

 

They cause a crash when called with out==NULL. Normally it should not happen – but when an engine is called, and it cannot process the padding – it reverts to the original OpenSSL-provided pkey_rsa_encrypt() or pkey_rsa_decrypt() (as appropriate). OpenSSL pkeyutl makes two calls when the key is not directly available (aka not presented in a disk file), and the first call with out==NULL crashes when RSA_private_decrypt() or RSA_public_encrypt() tries to copy the result to out.

 

The fix should be adding something like

 

  if (out == NULL) {

       int klen = RSA_size(ctx->pkey->pkey.rsa);

       *outlen = klen;

       return 1;

  }

 

right before the call to RSA_public_encrypt().

 

P.S. It’s more critical in pkey_rsa_encrypt(), because it’s more likely that the engine would handle the decryption operation completely by itself.

--

Regards,

Uri Blumenthal

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev