Re: PyCrypto rant thread

On Thu, Oct 16, 2008 at 07:38:08AM -0600, zooko wrote:
> The first thing that I didn't like was that CTR mode made a Python 
> callback for each block, which was too slow and CPU intensive, so I wrote 
> a patch to do CTR mode all in C code, and only with a simple incrementing 
> counter, and I submitted that patch to AMK, who was then the ostensible 
> maintainer of PyCrypto.

The next version of PyCrypto will add the Crypto.Util.Counter module, which 
lets you do something like this:

     >>> from binascii import b2a_hex
     >>> from Crypto.Util import Counter
     >>> ctr = Counter.new(nbits=64,
                   prefix="\xDE\xAD\xBE\xEF",
                   suffix="\xAA\xAA\xAA\xAA",
                   initial_value=1,
                   little_endian=False)
     >>> b2a_hex(ctr())
     'deadbeef0000000000000001aaaaaaaa'
     >>> b2a_hex(ctr())
     'deadbeef0000000000000002aaaaaaaa'
     >>> ctr.next_value()
     3L
     >>> b2a_hex(ctr())
     'deadbeef0000000000000003aaaaaaaa'

As usual, you will then be able to feed the counter object to a block 
cipher:

     >>> from Crypto.Cipher import AES
     >>> key = "\0" * 32
     >>> a = AES.new(key, AES.MODE_CTR, counter=ctr)
     >>> a.encrypt("\0"*16)
     '\xfc{\xae\x9d\xe5ch\xf0R\x02\\P6\x04\xc5\xab'

The neat part is that (as of today) PyCrypto implements a shortcut that 
bypasses the Python method-call interface when you use these counter 
objects with PyCrypto's block ciphers.

> The fourth thing that I didn't like was that my secure distributed 
> filesystem (Tahoe) had a mysterious unit test failure one day.  It turned 
> out that the new version of PyCrypto's SHA-256 implementation, which was 
> another, newer, copy from LibTomCrypt, had a *different* bug, one that 
> caused incorrect output whenever the input was a certain length in bytes, 
> modulo 64.

Yeah...

The SHA256 "length = 55 (mod 64)" bug will be fixed in the next release of 
PyCrypto.  It's also tested for in the new test suite (see below).

> So the fifth and most important thing that I didn't like about PyCrypto 
> was: insufficiently thorough tests and insufficiently careful quality 
> control, e.g. copying code from LibTomCrypt and then not watching the 
> LibTomCrypt project to see if that code was subsequently discovered to be 
> buggy.

I've written a brand new self-test suite: It's located in the 
Crypto.SelfTest package.  I'm not completely happy with it, but it has a 
few key features that were missing from the old test code:

     - It can run outside the source tree (so distro users will be able to 
       run the tests on their own systems)

     - I have cited sources for most of the test vectors.  Where I couldn't 
       find any test vectors elsewhere, I generated them by hand using 
       another implementation (usually the openssl command).  So you no 
       longer need to wonder whether the test vectors are just a dump of 
       PyCrypto's previous output.

Also, before I publish my changes, I usually run the entire test suite 
against Python versions 2.1, 2.2, 2.3, 2.4, 2.5, and 2.6 on my machine 
(Linux/i386).  I also occasionally test on Win32 and Linux/amd64.

> Note that Crypto++ itself has a pretty good track record of code quality.  
> It was also the first open source crypto library to be certified as FIPS 
> 140-2 level 1 compliant (the second was OpenSSL).

I don't know if PyCrypto will ever get official certification.  My main 
concerns regarding certification are as follows:

     1. The standards are often many years behind the state of the art.  For 
        example, FIPS 186-3, which finally allows DSA keys to be larger than 
        1024 bits, still hasn't been published in non-draft form.

     2. Different standards tend to specify different algorithms for the 
        same thing.  For example, FIPS 186-2 (DSS) specifies a particular 
        algorithm for generating primes using SHA-1, but it's specific to 
        DSA and not very useful elsewhere.  I don't want to add multiple 
        prime-generation functions (thus increasing the risk of 
        implementation bugs) just to pass certification.

     3. From what I understand (and the situation is not at all clear to me) 
        some ECC patents have been licensed for use by the U.S. Federal 
        Government, but not for others to use.  It sounds like the U.S.  
        government is moving toward ECC, but I will not be adding 
        patent-encumbered features that nobody can actually use just to pass 
        certification.

     4. Barely anyone uses "certifiable" code anyway.  IIRC, either Mozilla 
        Seamonkey or Firefox has a FIPS mode, but it's not enabled by 
        default, and "turn on FIPS mode" has never been suggested as a 
        workaround in any security advisory I have seen.  OpenSSL's FIPS 
        certification didn't prevent the Debian PRNG bug.

Rather than worry about an unlikely certification that won't matter anyway, 
I would rather write a simpler, secure, state-of-the-art implementation and 
let others worry about maintaining certification-specific branches.

As for code quality, I would like to see PyCrypto do a _lot_ better than it 
currently does.  I also want to see improvements in the quality of 
*downstream* code, and I am taking steps in that direction.  For example, 
the next version of PyCrypto will deprecate 
Crypto.Util.randpool.RandomPool.  There's nothing wrong with the RandomPool 
implementation by itself.  Sure, almost everyone misuses it, but that's 
just "user error", right?  No, it's an error-prone interface, so I'm 
replacing it.  (See "RandomPool encourages insecure code" 
https://bugs.launchpad.net/pycrypto/+bug/249765.)

The Crypto.PublicKey package has similar problems.  The RSA module doesn't 
do any padding, and the DSA module doesn't perform the appropriate hash 
before signing.  I don't know about the other PublicKey modules, but I 
suspect they have similar problems.  Crypto.PublicKey is another 
error-prone API, so it will eventually be replaced, too.  (That won't 
happen until after the next release, though.)

> Thanks for asking!  I feel much better having had a Rant with my morning 
> coffee, and now I look forward to a happy day of trying to debug that 
> damned seg fault.  :-)  You can see if I succeeded by watching the 
> pycryptopp buildbot waterfall:
>
> http://allmydata.org/buildbot-pycryptopp/waterfall?reload=60

I think I need something like that. :-)

Cheers,
  - Dwayne

-- 
Dwayne C. Litzenberger <[email protected]>
  Key-signing key   - 19E1 1FE8 B3CF F273 ED17  4A24 928C EC13 39C2 5CF7
  Annual key (2008) - 4B2A FD82 FC7D 9E38 38D9  179F 1C11 B877 E780 4B45

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iEYEARECAAYFAkj7wkYACgkQHBG4d+eAS0VYTACgjk466Tk6fkqeWqVZ/yP34RBc
f8gAn1IHsb9+DvQmPVBjioXskvwM5oaG
=sLIQ
-----END PGP SIGNATURE-----