Re: Cryptlib OpenPGP enveloping occasionally produces unreadable data

Lahiru <[email protected]> writes:

>I notice that when we perform openpgp enveloping, cryptlib occasionally
>produces data that cannot be de-enveloped by itself. I've attached the code
>for a test I wrote for this. When I run it, cryptlib gives
>CRYPT_ERROR_BADDATA about 5-10 times out of a 1000.

The problem occurs because some of the lower-level code used by cryptlib
performs leading-zero truncation and some doesn't (specifically, PKCS #11
drivers are erratic as to whether they do it or not, see the code comment in
device/pkcs11_pkc.c).  cryptlib tries to compensate for the differences, but
in this case it over-compensated (the fact that about one in 256 operations
failed was the giveaway because 1/256 results will have a leading zero, what
puzzled me was that I knew I'd addressed this issue, but it was being
cancelled out in a second location that only occurred with PGP data).  Anyway,
the fix is, in mechs/mech_pkwrap.c, line 297, to change the code block to:

		{
		status = krnlSendMessage( mechanismInfo->wrapContext,
								  IMESSAGE_CTX_ENCRYPT, wrappedData, 
								  wrappedDataLength );
		if( cryptStatusOK( status ) )
			{
			const BYTE *dataPtr = mechanismInfo->wrappedData;
			int dataLength = wrappedDataLength;

			/* The PKC wrap functions take a fixed-length input and produce 
			   a fixed-length output but some of this can be leading-zero 
			   padding, so we strip the padding if there's any present */
			if( *dataPtr == 0 )
				{
				while( *dataPtr == 0 && dataLength > 16 )
					{
					dataPtr++;
					dataLength--;
					}
				ENSURES( dataLength >= 16 );
				memmove( mechanismInfo->wrappedData, dataPtr, 
						 dataLength );
				memset( ( BYTE * ) mechanismInfo->wrappedData + dataLength, 
						0, wrappedDataLength - dataLength );
				}
			mechanismInfo->wrappedDataLength = dataLength;
			}
		}

Peter.

_______________________________________________
Cryptlib mailing list
[email protected] via Mail: [email protected]
Archive: ftp://ftp.franken.de/pub/crypt/cryptlib/archives/
http://news.gmane.org/gmane.comp.encryption.cryptlib
Posts from non-subscribed addresses are blocked to prevent spam, please
subscribe in order to post messages.