Why do I get the following error `wrong signature length` when I try to validate a signed file using the c++ OpenSSL 3.1 library?

"Christian F. Gonzalez Di Antonio" <[email protected]>
Newsgroups gmane.comp.encryption.openssl.user
Message-ID <CAEoszoazdP_ST+5P0yZupuuD6XzguHYJJbxAY-E_CW=Lb87dRg@mail.gmail.com>
I posted this on
https://stackoverflow.com/questions/78604338/why-do-i-get-the-following-error-wrong-signature-length-when-i-try-to-validate

I'm writing an c++ program LicenseValidator ->
https://github.com/christiangda/LicenseValidator
<https://github.com/christiangda/LicenseValidator> to validate a
hypothetical program license using OpenSSL 3.1 Library
<https://wiki.openssl.org/index.php/OpenSSL_3.0>, and when I tried to
validate the licensed content I got the following error:

Failed to verify license
008C1AF901000000:error:02000077:rsa routines:ossl_rsa_verify:wrong
signature length:crypto/rsa/rsa_sign.c:338:
008C1AF901000000:error:1C880004:Provider routines:rsa_verify:RSA
lib:providers/implementations/signature/rsa_sig.c:785:

I would appreciate any help or guidance on what I am doing wrong.

I am not at all an expert in the c/c++ programming language and this is the
first time I have tried to use the OpenSSL library.

Of course, I've used GitHub Copilot, gemini, and chatgpt to write and
understand the repository code. The chalenge is about the examples I found
on internet, the majority of them are about OpenSSL v1 and the v3 is very
different, so was hard to understand the migration.

The README.md
<https://github.com/christiangda/LicenseValidator/blob/main/README.md> file
has the instructions to create all the necessary keys, etc, references I
used and the instructions to compile it using cmake.

The core function is LicenseValidator/src/License.cpp
<https://github.com/christiangda/LicenseValidator/blob/82f5501ab2e5bf2d91dc4298245b36fde2efd66b/src/License.cpp#L79>
:

bool verifyLicense(const unsigned char *licenseContent, const unsigned
char *licenseSignature, const std::string pubkey){
    EVP_PKEY *pkey = loadRsaPemPubKey(pubkey);
    if (pkey == NULL)
    {
        std::cerr << "Failed to load public key" << std::endl;
        ERR_print_errors_fp(stdout);
        return false;
    }

    EVP_PKEY_CTX *ctx = EVP_PKEY_CTX_new(pkey, NULL);
    if (ctx == NULL)
    {
        std::cerr << "Failed to create EVP_PKEY_CTX" << std::endl;
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    if (EVP_PKEY_verify_init(ctx) <= 0)
    {
        std::cerr << "Failed to initialize EVP_PKEY_CTX" << std::endl;
        EVP_PKEY_CTX_free(ctx);
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    // PKCS1 padding scheme
    if (EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_PADDING) <= 0)
    {
        std::cerr << "Failed to set RSA padding" << std::endl;
        EVP_PKEY_CTX_free(ctx);
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    // SHA256 digest
    if (EVP_PKEY_CTX_set_signature_md(ctx, EVP_sha256()) <= 0)
    {
        std::cerr << "Failed to set signature MD" << std::endl;
        EVP_PKEY_CTX_free(ctx);
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    if (EVP_PKEY_verify(ctx, licenseSignature,
sizeof(licenseSignature), licenseContent, sizeof(licenseContent)) <=
0)
    {
        std::cerr << "Failed to verify license" << std::endl;
        EVP_PKEY_CTX_free(ctx);
        EVP_PKEY_free(pkey);
        ERR_print_errors_fp(stdout);
        return false;
    }

    EVP_PKEY_CTX_free(ctx);
    EVP_PKEY_free(pkey);

    return true;
}

Some guidance about how to solve the error I got.

-- 
Saludos,
Christian
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.