Re: Twisted Web Agent configurable Elliptic Curves settings
Glyph Lefkowitz <[email protected]> Thu, 27 Apr 2017 01:30:36 -0700
| Newsgroups | gmane.comp.python.twisted.web |
|---|---|
| Message-ID | <[email protected]> |
--===============0363610105629322102== Content-Type: multipart/alternative; boundary="Apple-Mail=_E5254928-222E-4479-A73F-C159F59B8A85" --Apple-Mail=_E5254928-222E-4479-A73F-C159F59B8A85 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Apr 26, 2017, at 10:32 AM, Paul Tremberth = <[email protected]> wrote: >=20 > Hello, >=20 > The other day, we had a Scrapy user report an issue connecting to = https://www.skelbiu.lt/ <https://www.skelbiu.lt/> with OpenSSL 1.1 [1] Thanks for passing this issue on! Always nice to have users engaging = directly with us rather than trying janky workarounds :). > To not mix scrapy's things with Twisted Web, I used this (adapted from = official docs): That's a good example to use, yes. > And I did get a Handshake failure too: >=20 > $ python twistedtest.py=20 > [Failure instance: Traceback (failure with no frames): <class = 'twisted.web._newclient.ResponseNeverReceived'>: = [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', = 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>] > ] >=20 > It seems this happens (at least) with OpenSSL 1.1.0e (currently in = Debian 9 sid [2]) > It does not happen (for me) with OpenSSL 1.0.2g for example. What platform are you on? How do you know what version of OpenSSL = you're using? (It can sometimes be quite tricky to suss out what = OpenSSL twisted is using unless you know the internals fairly well, = unfortunately; 'twist --version' really ought to print it out.) > I dug into this this afternoon and narrowed it down to the use of=20 > _defaultCurveName =3D u"prime256v1" > in twisted.internet._sslverify.py >=20 > I tried patching the current trunk with _defaultCurveName =3D = u"secp384r1" (the EC that ssllabs.com <http://ssllabs.com/> reports) > and it did work. >=20 > Looking at ClientHello messages for openssl 1.0.2 and 1.1 [4]: > with 1.1, only 1 Elliptic Curve is sent by Twisted Web Agent, = secp256r1 What about with 1.0.2? > openssl v1.1 client uses 4 by default: ecdh_x25519, secp256r1, = secp521r1, secp384r1 >=20 > I was wondering what is the proper way to configure requested Elliptic = Curves. > I haven't seen any interface for this, contrary to ciphers with = acceptableCiphers. It's possible that there should be an interface for this, but, your = issue should not be fixed with an API to work around this bug. It = should be fixed with a fix for this bug. My understanding is that Twisted just configures one curve, whereas = OpenSSL configures 4. Twisted should just configure all 4 (unless there's some security reason = not to match OpenSSL's behavior, which we should probably check on). Just guessing based on what I see here, I imagine that would mean = getting rid of the _ecCurve attribute, and instead having an _ecCurves =3D= something; then, getting rid of the call to SSL_CTX_set_tmp_ecdh and = instead using SSL_CTX_set1_curves, as = https://github.com/openssl/openssl/commit/6977e8ee4a718a76351ba5275a9f0be4= e530eab5 = <https://github.com/openssl/openssl/commit/6977e8ee4a718a76351ba5275a9f0be= 4e530eab5> seems to indicate that's how you have to request multiple = curves at once. Twisted should just fix this, not make it configurable. Later, for = security testing purposes, we may want to make it a fine-grained = configurable thing, but for right now the priority should be getting = correct behavior into a release. > Thank you for your input. Thanks for using Twisted!= --Apple-Mail=_E5254928-222E-4479-A73F-C159F59B8A85 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Apr 26, 2017, at 10:32 AM, Paul Tremberth <<a = href=3D"mailto:[email protected]" = class=3D"">[email protected]</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = class=3D"">Hello,<br class=3D""><br class=3D"">The other day, we had a = Scrapy user report an issue connecting to <a = href=3D"https://www.skelbiu.lt/" class=3D"">https://www.skelbiu.lt/</a> = with OpenSSL 1.1 [1]<br class=3D""></div></div></blockquote><div><br = class=3D""></div><div>Thanks for passing this issue on! Always = nice to have users engaging directly with us rather than trying janky = workarounds :).</div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div dir=3D"ltr" class=3D"">To not mix = scrapy's things with Twisted Web, I used this (adapted from official = docs):<br class=3D""></div></div></blockquote><div><br = class=3D""></div>That's a good example to use, yes.<br class=3D""><div><br= class=3D""></div><blockquote type=3D"cite" class=3D""><div = class=3D""><div dir=3D"ltr" class=3D"">And I did get a Handshake failure = too:<br class=3D""><br class=3D""> $ python twistedtest.py = <br class=3D""> [Failure instance: Traceback (failure with = no frames): <class = 'twisted.web._newclient.ResponseNeverReceived'>: = [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', = 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>]<br = class=3D""> ]<br class=3D""><br class=3D"">It seems this = happens (at least) with OpenSSL 1.1.0e (currently in Debian 9 sid = [2])<br class=3D"">It does not happen (for me) with OpenSSL 1.0.2g for = example.<br class=3D""></div></div></blockquote><div><br = class=3D""></div><div>What platform are you on? How do you know = what version of OpenSSL you're using? (It can sometimes be quite = tricky to suss out what OpenSSL twisted is using unless you know the = internals fairly well, unfortunately; 'twist --version' really ought to = print it out.)</div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div dir=3D"ltr" class=3D"">I dug into this = this afternoon and narrowed it down to the use of <div = class=3D"">_defaultCurveName =3D u"prime256v1"</div><div class=3D"">in = twisted.internet._sslverify.py<br class=3D""><br class=3D""></div><div = class=3D"">I tried patching the current trunk with _defaultCurveName =3D = u"secp384r1" (the EC that <a href=3D"http://ssllabs.com/" = class=3D"">ssllabs.com</a> reports)<br class=3D"">and it did work.<br = class=3D""><br class=3D"">Looking at ClientHello messages for openssl = 1.0.2 and 1.1 [4]:<br class=3D"">with 1.1, only 1 Elliptic Curve is sent = by Twisted Web Agent, secp256r1<br = class=3D""></div></div></div></blockquote><div><br = class=3D""></div><div>What about with 1.0.2?</div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D"">openssl v1.1 client uses 4 by = default: ecdh_x25519, secp256r1, secp521r1, secp384r1<br class=3D""><br = class=3D"">I was wondering what is the proper way to configure requested = Elliptic Curves.<br class=3D"">I haven't seen any interface for this, = contrary to ciphers with acceptableCiphers.<br = class=3D""></div></div></div></blockquote><div><br = class=3D""></div><div>It's possible that there should be an interface = for this, but, <i class=3D"">your</i> issue should not be fixed = with an API to work around this bug. It should be fixed with a <i = class=3D"">fix</i> for this bug.</div><div><br = class=3D""></div><div>My understanding is that Twisted just configures = one curve, whereas OpenSSL configures 4.</div><div><br = class=3D""></div><div>Twisted should just configure all 4 (unless = there's some security reason not to match OpenSSL's behavior, which we = should probably check on).</div><div><br class=3D""></div><div>Just = guessing based on what I see here, I imagine that would mean getting rid = of the _ecCurve attribute, and instead having an _ecCurves =3D = something; then, getting rid of the call to SSL_CTX_set_tmp_ecdh = and instead using SSL_CTX_set1_curves, as <a = href=3D"https://github.com/openssl/openssl/commit/6977e8ee4a718a76351ba527= 5a9f0be4e530eab5" = class=3D"">https://github.com/openssl/openssl/commit/6977e8ee4a718a76351ba= 5275a9f0be4e530eab5</a> seems to indicate that's how you have to = request multiple curves at once.</div><div><br = class=3D""></div><div>Twisted should just fix this, not make it = configurable. Later, for security testing purposes, we may want to = make it a fine-grained configurable thing, but for right now the = priority should be getting correct behavior into a release.</div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D"">Thank you for your = input.</div></div></div></blockquote><br class=3D""></div><div>Thanks = for using Twisted!</div></body></html>= --Apple-Mail=_E5254928-222E-4479-A73F-C159F59B8A85-- --===============0363610105629322102== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KVHdpc3RlZC13 ZWIgbWFpbGluZyBsaXN0ClR3aXN0ZWQtd2ViQHR3aXN0ZWRtYXRyaXguY29tCmh0dHA6Ly90d2lz dGVkbWF0cml4LmNvbS9jZ2ktYmluL21haWxtYW4vbGlzdGluZm8vdHdpc3RlZC13ZWIK --===============0363610105629322102==--