r46774 - Merge tls-endpoint-wrapper-5642-5: "tls:" endpoint
glyph-TA+aISz0psMTMxyoc4vAAJOcrHinNvQL0E9HWUfgJXw@public.gmane.org
| Newsgroups | gmane.comp.python.twisted.commits |
|---|---|
| Message-ID | <[email protected]> |
Author: glyph
Date: Thu Feb 11 02:46:49 2016
New Revision: 46774
Added:
trunk/twisted/topfiles/5642.feature
Modified:
trunk/docs/core/howto/endpoints.rst
trunk/twisted/internet/endpoints.py
trunk/twisted/internet/test/test_endpoints.py
trunk/twisted/plugins/twisted_core.py
trunk/twisted/python/compat.py
trunk/twisted/test/iosim.py
Log:
Merge tls-endpoint-wrapper-5642-5: "tls:" endpoint
Author: habnabit, glyph
Reviewer: glyph, hawkowl
Fixes: #5642
Add a new, better "tls:" endpoint, and numerous test utilities to facilitate covering it.
Modified: trunk/docs/core/howto/endpoints.rst
==============================================================================
--- trunk/docs/core/howto/endpoints.rst (original)
+++ trunk/docs/core/howto/endpoints.rst Thu Feb 11 02:46:49 2016
@@ -148,13 +148,39 @@
For example, ``tcp:host=twistedmatrix.com:port=80:timeout=15``.
-SSL
- All TCP arguments are supported, plus: ``certKey``, ``privateKey``, ``caCertsDir``.
- ``certKey`` (optional) gives a filesystem path to a certificate (PEM format).
- ``privateKey`` (optional) gives a filesystem path to a private key (PEM format).
- ``caCertsDir`` (optional) gives a filesystem path to a directory containing trusted CA certificates to use to verify the server certificate.
+TLS
+ Required arguments: ``host``, ``port``.
+
+ Optional arguments: ``timeout``, ``bindAddress``, ``certificate``, ``privateKey``, ``trustRoots``, ``endpoint``.
+
+ - ``host`` is a (UTF-8 encoded) hostname to connect to, as well as the host name to verify against.
+ - ``port`` is a numeric port number to connect to.
+ - ``timeout`` and ``bindAddress`` have the same meaning as the ``timeout`` and ``bindAddress`` for TCP clients.
+ - ``certificate`` is the certificate to use for the client; it should be the path name of a PEM file containing a certificate for which ``privateKey`` is the private key.
+ - ``privateKey`` is the client's private key, matching the certificate specified by ``certificate``.
+ It should be the path name of a PEM file containing an X.509 client certificate.
+ If ``certificate`` is specified but ``privateKey`` is unspecified, Twisted will look for the certificate in the same file as specified by ``certificate``.
+ - ``trustRoots`` specifies a path to a directory of PEM-encoded certificate files. If you leave this unspecified, Twisted will do its best to use the platform default set of trust roots, which should be the default WebTrust set.
+ - the optional ``endpoint`` parameter changes the meaning of the ``tls:`` endpoint slightly.
+ Rather than the default of connecting over TCP with the same hostname used for verification, you can connect over *any* endpoint type.
+ If you specify the endpoint here, ``host`` and ``port`` are used for certificate verification purposes only.
+ Bear in mind you will need to backslash-escape the colons in the endpoint description here.
+
+ This client connects to the supplied hostname, validates the server's hostname against the supplied hostname, and then upgrades to TLS immediately after validation succeeds.
+
+ The simplest example of this would be: ``tls:example.com:443``.
+
+ You can use the ``endpoint:`` feature with TCP if you want to connect to a host name; for example, if your DNS is not working, but you know that the IP address 7.6.5.4 points to ``awesome.site.example.com``, you could specify: ``tls:awesome.site.example.com:443:endpoint=tcp\:7.6.5.4\:443``.
+
+ You can use it with any other endpoint type as well, though; for example, if you had a local UNIX socket that established a tunnel to ``awesome.site.example.com`` in ``/var/run/awesome.sock``, you could instead do ``tls:awesome.site.example.com:443:endpoint=unix\:/var/run/awesome.sock``.
+
+ Or, from python code::
+
+ wrapped = HostnameEndpoint('example.com', 443)
+ contextFactory = optionsForClientTLS(hostname=u'example.com')
+ endpoint = wrapClientTLS(contextFactory, wrapped)
+ conn = endpoint.connect(Factory.forProtocol(Protocol))
- For example, ``ssl:host=twistedmatrix.com:port=443:caCertsDir=/etc/ssl/certs`` .
UNIX
Supported arguments: ``path``, ``timeout``, ``checkPID``.
``path`` gives a filesystem path to a listening UNIX domain socket server.
@@ -176,6 +202,24 @@
endpoint = HostnameEndpoint(reactor, "twistedmatrix.com", 80)
conn = endpoint.connect(Factory.forProtocol(Protocol))
+SSL (Deprecated)
+
+ .. note::
+
+ You should generally prefer the "TLS" client endpoint, above, unless you need to work with versions of Twisted older than 16.0.
+ Among other things:
+
+ - the ``ssl:`` client endpoint requires that you pass ''both'' ``hostname=`` (for hostname verification) as well as ``host=`` (for a TCP connection address) in order to get hostname verification, which is required for security, whereas ``tls:`` does the correct thing by default by using the same hostname for both.
+
+ - the ``ssl:`` client endpoint doesn't work with IPv6, and the ``tls:`` endpoint does.
+
+ All TCP arguments are supported, plus: ``certKey``, ``privateKey``, ``caCertsDir``.
+ ``certKey`` (optional) gives a filesystem path to a certificate (PEM format).
+ ``privateKey`` (optional) gives a filesystem path to a private key (PEM format).
+ ``caCertsDir`` (optional) gives a filesystem path to a directory containing trusted CA certificates to use to verify the server certificate.
+
+ For example, ``ssl:host=twistedmatrix.com:port=443:caCertsDir=/etc/ssl/certs``.
+
Servers
~~~~~~~
Modified: trunk/twisted/internet/endpoints.py
==============================================================================
--- trunk/twisted/internet/endpoints.py (original)
+++ trunk/twisted/internet/endpoints.py Thu Feb 11 02:46:49 2016
@@ -35,14 +35,26 @@
from twisted.internet.task import LoopingCall
from twisted.plugin import IPlugin, getPlugins
from twisted.python import log
-from twisted.python.compat import nativeString
+from twisted.python.compat import nativeString, unicode, _matchingString
from twisted.python.components import proxyForInterface
from twisted.python.constants import NamedConstant, Names
from twisted.python.failure import Failure
from twisted.python.filepath import FilePath
+from twisted.python.compat import iterbytes
from twisted.python.systemd import ListenFDs
+try:
+ from twisted.protocols.tls import TLSMemoryBIOFactory
+ from twisted.internet._sslverify import _idnaBytes, _idnaText
+ from twisted.internet.ssl import (
+ optionsForClientTLS, PrivateCertificate, Certificate, KeyPair,
+ CertificateOptions, trustRootFromCertificates
+ )
+ from OpenSSL.SSL import Error as SSLError
+except ImportError:
+ TLSMemoryBIOFactory = None
+
__all__ = ["clientFromString", "serverFromString",
"TCP4ServerEndpoint", "TCP6ServerEndpoint",
"TCP4ClientEndpoint", "TCP6ClientEndpoint",
@@ -50,7 +62,8 @@
"SSL4ServerEndpoint", "SSL4ClientEndpoint",
"AdoptedStreamServerEndpoint", "StandardIOEndpoint",
"ProcessEndpoint", "HostnameEndpoint",
- "StandardErrorBehavior", "connectProtocol"]
+ "StandardErrorBehavior", "connectProtocol",
+ "wrapClientTLS"]
@@ -1264,12 +1277,13 @@
@param description: a string as described by L{serverFromString} or
L{clientFromString}.
+ @type description: L{str} or L{bytes}
@return: an iterable of 2-tuples of (L{_OP} or L{_STRING}, string). Tuples
starting with L{_OP} will contain a second element of either ':' (i.e.
'next parameter') or '=' (i.e. 'assign parameter value'). For example,
- the string 'hello:greet\=ing=world' would result in a generator
- yielding these values::
+ the string 'hello:greeting=world' would result in a generator yielding
+ these values::
_STRING, 'hello'
_OP, ':'
@@ -1277,18 +1291,23 @@
_OP, '='
_STRING, 'world'
"""
- current = ''
- ops = ':='
- nextOps = {':': ':=', '=': ':'}
- description = iter(description)
- for n in description:
- if n in ops:
+ empty = _matchingString(u'', description)
+ colon = _matchingString(u':', description)
+ equals = _matchingString(u'=', description)
+ backslash = _matchingString(u'\x5c', description)
+ current = empty
+
+ ops = colon + equals
+ nextOps = {colon: colon + equals, equals: colon}
+ iterdesc = iter(iterbytes(description))
+ for n in iterdesc:
+ if n in iterbytes(ops):
yield _STRING, current
yield _OP, n
- current = ''
+ current = empty
ops = nextOps[n]
- elif n == '\\':
- current += next(description)
+ elif n == backslash:
+ current += next(iterdesc)
else:
current += n
yield _STRING, current
@@ -1309,16 +1328,17 @@
C{_parse('a:b:d=1:c')} would be C{(['a', 'b', 'c'], {'d': '1'})}.
"""
args, kw = [], {}
+ colon = _matchingString(u':', description)
def add(sofar):
if len(sofar) == 1:
args.append(sofar[0])
else:
- kw[sofar[0]] = sofar[1]
+ kw[nativeString(sofar[0])] = sofar[1]
sofar = ()
for (type, value) in _tokenize(description):
if type is _STRING:
sofar += (value,)
- elif value == ':':
+ elif value == colon:
add(sofar)
sofar = ()
add(sofar)
@@ -1381,10 +1401,10 @@
if parser is None:
# If the required parser is not found in _server, check if
# a plugin exists for the endpointType
- for plugin in getPlugins(IStreamServerEndpointStringParser):
- if plugin.prefix == endpointType:
- return (plugin, args[1:], kw)
- raise ValueError("Unknown endpoint type: '%s'" % (endpointType,))
+ plugin = _matchPluginToPrefix(
+ getPlugins(IStreamServerEndpointStringParser), endpointType
+ )
+ return (plugin, args[1:], kw)
return (endpointType.upper(),) + parser(factory, *args[1:], **kw)
@@ -1406,6 +1426,19 @@
+def _matchPluginToPrefix(plugins, endpointType):
+ """
+ Match plugin to prefix.
+ """
+ endpointType = endpointType.lower()
+ for plugin in plugins:
+ if (_matchingString(plugin.prefix.lower(),
+ endpointType) == endpointType):
+ return plugin
+ raise ValueError("Unknown endpoint type: '%s'" % (endpointType,))
+
+
+
def serverFromString(reactor, description):
"""
Construct a stream server endpoint from an endpoint description string.
@@ -1501,7 +1534,10 @@
@rtype: C{str}
"""
- return argument.replace('\\', '\\\\').replace(':', '\\:')
+ backslash, colon = '\\:'
+ for c in backslash, colon:
+ argument = argument.replace(c, backslash + c)
+ return argument
@@ -1549,16 +1585,14 @@
"""
Load certificate-authority certificate objects in a given directory.
- @param directoryPath: a L{FilePath} pointing at a directory to load .pem
- files from.
+ @param directoryPath: a L{unicode} or L{bytes} pointing at a directory to
+ load .pem files from, or L{None}.
- @return: a C{list} of L{OpenSSL.crypto.X509} objects.
+ @return: an L{IOpenSSLTrustRoot} provider.
"""
- from twisted.internet import ssl
-
caCerts = {}
for child in directoryPath.children():
- if not child.basename().split('.')[-1].lower() == 'pem':
+ if not child.asTextMode().basename().split(u'.')[-1].lower() == u'pem':
continue
try:
data = child.getContent()
@@ -1566,13 +1600,94 @@
# Permission denied, corrupt disk, we don't care.
continue
try:
- theCert = ssl.Certificate.loadPEM(data)
- except ssl.SSL.Error:
+ theCert = Certificate.loadPEM(data)
+ except SSLError:
# Duplicate certificate, invalid certificate, etc. We don't care.
pass
else:
- caCerts[theCert.digest()] = theCert.original
- return caCerts.values()
+ caCerts[theCert.digest()] = theCert
+ return trustRootFromCertificates(caCerts.values())
+
+
+
+def _parseTrustRootPath(pathName):
+ """
+ Parse a string referring to a directory full of certificate authorities
+ into a trust root.
+
+ @param pathName: path name
+ @type pathName: L{unicode} or L{bytes} or L{None}
+
+ @return: L{None} or L{IOpenSSLTrustRoot}
+ """
+ if pathName is None:
+ return None
+ return _loadCAsFromDir(FilePath(pathName))
+
+
+
+def _privateCertFromPaths(certificatePath, keyPath):
+ """
+ Parse a certificate path and key path, either or both of which might be
+ C{None}, into a certificate object.
+
+ @param certificatePath: the certificate path
+ @type certificatePath: L{bytes} or L{unicode} or L{None}
+
+ @param keyPath: the private key path
+ @type keyPath: L{bytes} or L{unicode} or L{None}
+
+ @return: a L{PrivateCertificate} or L{None}
+ """
+ if certificatePath is None:
+ return None
+ certBytes = FilePath(certificatePath).getContent()
+ if keyPath is None:
+ return PrivateCertificate.loadPEM(certBytes)
+ else:
+ return PrivateCertificate.fromCertificateAndKeyPair(
+ Certificate.loadPEM(certBytes),
+ KeyPair.load(FilePath(keyPath).getContent(), 1)
+ )
+
+
+
+def _parseClientSSLOptions(kwargs):
+ """
+ Parse common arguments for SSL endpoints, creating an L{CertificateOptions}
+ instance.
+
+ @param kwargs: A dict of keyword arguments to be parsed, potentially
+ containing keys C{certKey}, C{privateKey}, C{caCertsDir}, and
+ C{hostname}. See L{_parseClientSSL}.
+ @type kwargs: L{dict}
+
+ @return: The remaining arguments, including a new key C{sslContextFactory}.
+ """
+ hostname = kwargs.pop('hostname', None)
+ clientCertificate = _privateCertFromPaths(kwargs.pop('certKey', None),
+ kwargs.pop('privateKey', None))
+ trustRoot = _parseTrustRootPath(kwargs.pop('caCertsDir', None))
+ if hostname is not None:
+ configuration = optionsForClientTLS(
+ _idnaText(hostname), trustRoot=trustRoot,
+ clientCertificate=clientCertificate
+ )
+ else:
+ # _really_ though, you should specify a hostname.
+ if clientCertificate is not None:
+ privateKeyOpenSSL = clientCertificate.privateKey.original
+ certificateOpenSSL = clientCertificate.original
+ else:
+ privateKeyOpenSSL = None
+ certificateOpenSSL = None
+ configuration = CertificateOptions(
+ trustRoot=trustRoot,
+ privateKey=privateKeyOpenSSL,
+ certificate=certificateOpenSSL,
+ )
+ kwargs['sslContextFactory'] = configuration
+ return kwargs
@@ -1594,38 +1709,16 @@
will be scanned for files ending in C{.pem}, all of which will be
considered valid certificate authorities for this connection.
- @type caCertsDir: C{str}
+ @type caCertsDir: L{str}
- @return: The coerced values as a C{dict}.
+ @param hostname: The hostname to use for validating the server's
+ certificate.
+ @type hostname: L{unicode}
+
+ @return: The coerced values as a L{dict}.
"""
- from twisted.internet import ssl
kwargs = _parseClientTCP(*args, **kwargs)
- certKey = kwargs.pop('certKey', None)
- privateKey = kwargs.pop('privateKey', None)
- caCertsDir = kwargs.pop('caCertsDir', None)
- if certKey is not None:
- certx509 = ssl.Certificate.loadPEM(
- FilePath(certKey).getContent()).original
- else:
- certx509 = None
- if privateKey is not None:
- privateKey = ssl.PrivateCertificate.loadPEM(
- FilePath(privateKey).getContent()).privateKey.original
- else:
- privateKey = None
- if caCertsDir is not None:
- verify = True
- caCerts = _loadCAsFromDir(FilePath(caCertsDir))
- else:
- verify = False
- caCerts = None
- kwargs['sslContextFactory'] = ssl.CertificateOptions(
- certificate=certx509,
- privateKey=privateKey,
- verify=verify,
- caCerts=caCerts
- )
- return kwargs
+ return _parseClientSSLOptions(kwargs)
@@ -1737,11 +1830,11 @@
args, kwargs = _parse(description)
aname = args.pop(0)
name = aname.upper()
- for plugin in getPlugins(IStreamClientEndpointStringParserWithReactor):
- if plugin.prefix.upper() == name:
- return plugin.parseStreamClient(reactor, *args, **kwargs)
if name not in _clientParsers:
- raise ValueError("Unknown endpoint type: %r" % (aname,))
+ plugin = _matchPluginToPrefix(
+ getPlugins(IStreamClientEndpointStringParserWithReactor), name
+ )
+ return plugin.parseStreamClient(reactor, *args, **kwargs)
kwargs = _clientParsers[name](*args, **kwargs)
return _endpointClientFactories[name](reactor, **kwargs)
@@ -1767,3 +1860,154 @@
def buildProtocol(self, addr):
return protocol
return endpoint.connect(OneShotFactory())
+
+
+
+@implementer(interfaces.IStreamClientEndpoint)
+class _WrapperEndpoint(object):
+ """
+ An endpoint that wraps another endpoint.
+ """
+
+ def __init__(self, wrappedEndpoint, wrapperFactory):
+ """
+ Construct a L{_WrapperEndpoint}.
+ """
+ self._wrappedEndpoint = wrappedEndpoint
+ self._wrapperFactory = wrapperFactory
+
+
+ def connect(self, protocolFactory):
+ """
+ Connect the given protocol factory and unwrap its result.
+ """
+ return self._wrappedEndpoint.connect(
+ self._wrapperFactory(protocolFactory)
+ ).addCallback(lambda protocol: protocol.wrappedProtocol)
+
+
+
+def wrapClientTLS(connectionCreator, wrappedEndpoint):
+ """
+ Wrap an endpoint which upgrades to TLS as soon as the connection is
+ established.
+
+ @since: 16.0
+
+ @param connectionCreator: The TLS options to use when connecting; see
+ L{twisted.internet.ssl.optionsForClientTLS} for how to construct this.
+ @type connectionCreator:
+ L{twisted.internet.interfaces.IOpenSSLClientConnectionCreator}
+
+ @param wrappedEndpoint: The endpoint to wrap.
+ @type wrappedEndpoint: An L{IStreamClientEndpoint} provider.
+
+ @return: an endpoint that provides transport level encryption layered on
+ top of C{wrappedEndpoint}
+ @rtype: L{twisted.internet.interfaces.IStreamClientEndpoint}
+ """
+ if TLSMemoryBIOFactory is None:
+ raise NotImplementedError(
+ "OpenSSL not available. Try `pip install twisted[tls]`."
+ )
+ return _WrapperEndpoint(
+ wrappedEndpoint,
+ lambda protocolFactory:
+ TLSMemoryBIOFactory(connectionCreator, True, protocolFactory)
+ )
+
+
+
+def _parseClientTLS(reactor, host, port, timeout=b'30', bindAddress=None,
+ certificate=None, privateKey=None, trustRoots=None,
+ endpoint=None, **kwargs):
+ """
+ Internal method to construct an endpoint from string parameters.
+
+ @param reactor: The reactor passed to L{clientFromString}.
+
+ @param host: The hostname to connect to.
+ @type host: L{bytes} or L{unicode}
+
+ @param port: The port to connect to.
+ @type port: L{bytes} or L{unicode}
+
+ @param timeout: For each individual connection attempt, the number of
+ seconds to wait before assuming the connection has failed.
+ @type timeout: L{bytes} or L{unicode}
+
+ @param bindAddress: The address to which to bind outgoing connections.
+ @type bindAddress: L{bytes} or L{unicode}
+
+ @param certificate: a string representing a filesystem path to a
+ PEM-encoded certificate.
+ @type certificate: L{bytes} or L{unicode}
+
+ @param privateKey: a string representing a filesystem path to a PEM-encoded
+ certificate.
+ @type privateKey: L{bytes} or L{unicode}
+
+ @param endpoint: an optional string endpoint description of an endpoint to
+ wrap; if this is passed then C{host} is used only for certificate
+ verification.
+ @type endpoint: L{bytes} or L{unicode}
+
+ @return: a client TLS endpoint
+ @rtype: L{IStreamClientEndpoint}
+ """
+ if kwargs:
+ raise TypeError('unrecognized keyword arguments present',
+ list(kwargs.keys()))
+ host = host if isinstance(host, unicode) else host.decode("utf-8")
+ bindAddress = (bindAddress
+ if isinstance(bindAddress, unicode) or bindAddress is None
+ else bindAddress.decode("utf-8"))
+ port = int(port)
+ timeout = int(timeout)
+ return wrapClientTLS(
+ optionsForClientTLS(
+ host, trustRoot=_parseTrustRootPath(trustRoots),
+ clientCertificate=_privateCertFromPaths(certificate,
+ privateKey)),
+ clientFromString(reactor, endpoint) if endpoint is not None
+ else HostnameEndpoint(reactor, _idnaBytes(host), port, timeout,
+ bindAddress)
+ )
+
+
+
+@implementer(IPlugin, IStreamClientEndpointStringParserWithReactor)
+class _TLSClientEndpointParser(object):
+ """
+ Stream client endpoint string parser for L{wrapClientTLS} with
+ L{HostnameEndpoint}.
+
+ @ivar prefix: See
+ L{IStreamClientEndpointStringParserWithReactor.prefix}.
+ """
+ prefix = 'tls'
+
+ @staticmethod
+ def parseStreamClient(reactor, *args, **kwargs):
+ """
+ Redirects to another function L{_parseClientTLS}; tricks zope.interface
+ into believing the interface is correctly implemented, since the
+ signature is (C{reactor}, C{*args}, C{**kwargs}). See
+ L{_parseClientTLS} for an the specific signature description for this
+ endpoint parser.
+
+ @param reactor: The reactor passed to L{clientFromString}.
+
+ @param args: The positional arguments in the endpoint description.
+ @type args: L{tuple}
+
+ @param kwargs: The named arguments in the endpoint description.
+ @type kwargs: L{dict}
+
+ @return: a client TLS endpoint
+ @rtype: L{IStreamClientEndpoint}
+ """
+ return _parseClientTLS(reactor, *args, **kwargs)
+
+
+
Modified: trunk/twisted/internet/test/test_endpoints.py
==============================================================================
--- trunk/twisted/internet/test/test_endpoints.py (original)
+++ trunk/twisted/internet/test/test_endpoints.py Thu Feb 11 02:46:49 2016
@@ -14,9 +14,9 @@
from socket import AF_INET, AF_INET6, SOCK_STREAM, IPPROTO_TCP
from zope.interface import implementer
from zope.interface.verify import verifyObject, verifyClass
+from types import FunctionType
from twisted.trial import unittest
-from twisted.test import __file__ as testInitPath
from twisted.test.proto_helpers import MemoryReactorClock as MemoryReactor
from twisted.test.proto_helpers import RaisingMemoryReactor, StringTransport
from twisted.test.proto_helpers import StringTransportWithDisconnection
@@ -37,9 +37,12 @@
from twisted.python.filepath import FilePath
from twisted.python.modules import getModule
from twisted.python.systemd import ListenFDs
+from twisted.protocols import basic, policies
+from twisted.test.iosim import connectedServerAndClient, connectableEndpoint
+from twisted.internet.error import ConnectingCancelledError
+from twisted.python.compat import nativeString
-
-pemPath = FilePath(testInitPath).sibling("server.pem")
+pemPath = getModule("twisted.test").filePath.sibling("server.pem")
casPath = getModule(__name__).filePath.sibling("fake_CAs")
chainPath = casPath.child("chain.pem")
escapedPEMPathName = endpoints.quoteStringArgument(pemPath.path)
@@ -49,9 +52,11 @@
try:
from twisted.test.test_sslverify import makeCertificate
- from twisted.internet.ssl import PrivateCertificate, Certificate
- from twisted.internet.ssl import CertificateOptions, KeyPair
- from twisted.internet.ssl import DiffieHellmanParameters
+ from twisted.internet.ssl import (
+ PrivateCertificate, Certificate, CertificateOptions, KeyPair,
+ DiffieHellmanParameters
+ )
+ from twisted.protocols.tls import TLSMemoryBIOFactory
from OpenSSL.SSL import (
ContextType, SSLv23_METHOD, TLSv1_METHOD, OP_NO_SSLv3
)
@@ -2397,8 +2402,8 @@
descriptions.
"""
self.assertEqual(
- self.parse(r'unix:foo\:bar\=baz\:qux\\', self.f),
- ('UNIX', ('foo:bar=baz:qux\\', self.f),
+ self.parse('unix:foo\x5c:bar\x5c=baz\x5c:qux\x5c\x5c', self.f),
+ ('UNIX', ('foo:bar=baz:qux\x5c', self.f),
{'mode': 0o666, 'backlog': 50, 'wantPID': True}))
@@ -2408,8 +2413,8 @@
for interpolation into L{endpoints.serverFromString} and
L{endpoints.clientFactory} arguments.
"""
- self.assertEqual(endpoints.quoteStringArgument("some : stuff \\"),
- "some \\: stuff \\\\")
+ self.assertEqual(endpoints.quoteStringArgument("some : stuff \x5c"),
+ "some \x5c: stuff \x5c\x5c")
def test_impliedEscape(self):
@@ -2893,10 +2898,20 @@
[casPath.child("thing1.pem"), casPath.child("thing2.pem")]
if x.basename().lower().endswith('.pem')
]
- self.assertEqual(sorted((Certificate(x) for x in certOptions.caCerts),
- key=lambda cert: cert.digest()),
- sorted(expectedCerts,
- key=lambda cert: cert.digest()))
+ addedCerts = []
+ class ListCtx(object):
+ def get_cert_store(self):
+ class Store(object):
+ def add_cert(self, cert):
+ addedCerts.append(cert)
+ return Store()
+ certOptions.trustRoot._addCACertsToContext(ListCtx())
+ self.assertEqual(
+ sorted((Certificate(x) for x in addedCerts),
+ key=lambda cert: cert.digest()),
+ sorted(expectedCerts,
+ key=lambda cert: cert.digest())
+ )
def test_sslPositionalArgs(self):
@@ -2954,7 +2969,8 @@
casPathClone = casPath.child("ignored").parent()
casPathClone.clonePath = UnreadableFilePath
self.assertEqual(
- [Certificate(x) for x in endpoints._loadCAsFromDir(casPathClone)],
+ [Certificate(x) for x in
+ endpoints._loadCAsFromDir(casPathClone)._caCerts],
[Certificate.loadPEM(casPath.child("thing1.pem").getContent())])
@@ -3314,3 +3330,375 @@
endpoint = Endpoint()
self.assertIs(result, endpoints.connectProtocol(endpoint, object()))
+
+
+
+class UppercaseWrapperProtocol(policies.ProtocolWrapper, object):
+ """
+ A wrapper protocol which uppercases all strings passed through it.
+ """
+
+ def dataReceived(self, data):
+ """
+ Uppercase a string passed in from the transport.
+
+ @param data: The string to uppercase.
+ @type data: L{bytes}
+ """
+ super(UppercaseWrapperProtocol, self).dataReceived(data.upper())
+
+
+ def write(self, data):
+ """
+ Uppercase a string passed out to the transport.
+
+ @param data: The string to uppercase.
+ @type data: L{bytes}
+ """
+ super(UppercaseWrapperProtocol, self).write(data.upper())
+
+
+ def writeSequence(self, seq):
+ """
+ Uppercase a series of strings passed out to the transport.
+
+ @param seq: An iterable of strings.
+ """
+ for data in seq:
+ self.write(data)
+
+
+
+class UppercaseWrapperFactory(policies.WrappingFactory, object):
+ """
+ A wrapper factory which uppercases all strings passed through it.
+ """
+ protocol = UppercaseWrapperProtocol
+
+
+
+class NetstringTracker(basic.NetstringReceiver, object):
+ """
+ A netstring receiver which keeps track of the strings received.
+
+ @ivar strings: A L{list} of received strings, in order.
+ """
+
+ def __init__(self):
+ self.strings = []
+
+
+ def stringReceived(self, string):
+ """
+ Receive a string and append it to C{self.strings}.
+
+ @param string: The string to be appended to C{self.strings}.
+ """
+ self.strings.append(string)
+
+
+
+class FakeError(Exception):
+ """
+ An error which isn't really an error.
+
+ This is raised in the L{wrapClientTLS} tests in place of a
+ 'real' exception.
+ """
+
+
+
+class WrapperClientEndpointTests(unittest.TestCase):
+ """
+ Tests for L{_WrapperClientEndpoint}.
+ """
+
+ def setUp(self):
+ self.endpoint, self.completer = connectableEndpoint()
+ self.context = object()
+ self.wrapper = endpoints._WrapperEndpoint(self.endpoint,
+ UppercaseWrapperFactory)
+ self.factory = Factory.forProtocol(NetstringTracker)
+
+
+ def test_wrappingBehavior(self):
+ """
+ Any modifications performed by the underlying L{ProtocolWrapper}
+ propagate through to the wrapped L{Protocol}.
+ """
+ connecting = self.wrapper.connect(self.factory)
+ pump = self.completer.succeedOnce()
+ proto = self.successResultOf(connecting)
+ pump.server.transport.write(b'5:hello,')
+ pump.flush()
+ self.assertEqual(proto.strings, [b'HELLO'])
+
+
+ def test_methodsAvailable(self):
+ """
+ Methods defined on the wrapped L{Protocol} are accessible from the
+ L{Protocol} returned from C{connect}'s L{Deferred}.
+ """
+ connecting = self.wrapper.connect(self.factory)
+ pump = self.completer.succeedOnce()
+ proto = self.successResultOf(connecting)
+ proto.sendString(b'spam')
+ self.assertEqual(pump.clientIO.getOutBuffer(), b'4:SPAM,')
+
+
+ def test_connectionFailure(self):
+ """
+ Connection failures propagate upward to C{connect}'s L{Deferred}.
+ """
+ d = self.wrapper.connect(self.factory)
+ self.assertNoResult(d)
+ self.completer.failOnce(FakeError())
+ self.failureResultOf(d, FakeError)
+
+
+ def test_connectionCancellation(self):
+ """
+ Cancellation propagates upward to C{connect}'s L{Deferred}.
+ """
+ d = self.wrapper.connect(self.factory)
+ self.assertNoResult(d)
+ d.cancel()
+ self.failureResultOf(d, ConnectingCancelledError)
+
+
+ def test_transportOfTransportOfWrappedProtocol(self):
+ """
+ The transport of the wrapped L{Protocol}'s transport is the transport
+ passed to C{makeConnection}.
+ """
+ connecting = self.wrapper.connect(self.factory)
+ pump = self.completer.succeedOnce()
+ proto = self.successResultOf(connecting)
+ self.assertIdentical(
+ proto.transport.transport, pump.clientIO)
+
+
+
+def connectionCreatorFromEndpoint(memoryReactor, tlsEndpoint):
+ """
+ Given a L{MemoryReactor} and the result of calling L{wrapClientTLS},
+ extract the L{IOpenSSLClientConnectionCreator} associated with it.
+
+ Implementation presently uses private attributes but could (and should) be
+ refactored to just call C{.connect()} on the endpoint, when
+ L{HostnameEndpoint} starts directing its C{getaddrinfo} call through the
+ reactor it is passed somehow rather than via the global threadpool.
+
+ @param memoryReactor: the reactor attached to the given endpoint.
+ (Presently unused, but included so tests won't need to be modified to
+ honor it.)
+
+ @param tlsEndpoint: The result of calling L{wrapClientTLS}.
+
+ @return: the client connection creator associated with the endpoint
+ wrapper.
+ @rtype: L{IOpenSSLClientConnectionCreator}
+ """
+ return tlsEndpoint._wrapperFactory(None)._connectionCreator
+
+
+
+def makeHostnameEndpointSynchronous(hostnameEndpoint):
+ """
+ Make the given L{HostnameEndpoint} fire its L{defer.Deferred} from
+ C{connect} synchronously by patching its C{_deferToThread} implementation
+ to return an already-succeeded Deferred.
+
+ @param hostnameEndpoint: The hostname endpoint to patch.
+ """
+ family = AF_INET
+ socktype = SOCK_STREAM
+ proto = IPPROTO_TCP
+ canonname = b''
+ sockaddr = ('127.0.0.1', 4321)
+ gaiResult = family, socktype, proto, canonname, sockaddr
+ def synchronousDeferToThreadForGAI(*args):
+ return defer.succeed([gaiResult])
+ hostnameEndpoint._deferToThread = synchronousDeferToThreadForGAI
+
+
+
+class WrapClientTLSParserTests(unittest.TestCase):
+ """
+ Tests for L{_TLSClientEndpointParser}.
+ """
+
+ if skipSSL:
+ skip = skipSSL
+
+ def test_hostnameEndpointConstruction(self):
+ """
+ A L{HostnameEndpoint} is constructed from parameters passed to
+ L{clientFromString}.
+ """
+ reactor = object()
+ endpoint = endpoints.clientFromString(
+ reactor,
+ nativeString(
+ 'tls:example.com:443:timeout=10:bindAddress=127.0.0.1'))
+ hostnameEndpoint = endpoint._wrappedEndpoint
+ self.assertIs(hostnameEndpoint._reactor, reactor)
+ self.assertEqual(hostnameEndpoint._host, b'example.com')
+ self.assertEqual(hostnameEndpoint._port, 443)
+ self.assertEqual(hostnameEndpoint._timeout, 10)
+ self.assertEqual(hostnameEndpoint._bindAddress,
+ nativeString('127.0.0.1'))
+
+
+ def test_utf8Encoding(self):
+ """
+ The hostname passed to L{clientFromString} is treated as utf-8 bytes;
+ it is then encoded as IDNA when it is passed along to
+ L{HostnameEndpoint}, and passed as unicode to L{optionsForClientTLS}.
+ """
+ reactor = object()
+ endpoint = endpoints.clientFromString(
+ reactor, b'tls:\xc3\xa9xample.example.com:443'
+ )
+ self.assertEqual(
+ endpoint._wrappedEndpoint._host, b'xn--xample-9ua.example.com')
+ connectionCreator = connectionCreatorFromEndpoint(reactor, endpoint)
+ self.assertEqual(connectionCreator._hostname,
+ u'\xe9xample.example.com')
+
+
+ def test_tls(self):
+ """
+ When passed a string endpoint description beginning with C{tls:},
+ L{clientFromString} returns a client endpoint initialized with the
+ values from the string.
+ """
+ # We can't peer into the unknowable chaos of the heart of OpenSSL
+ # (there's no public API to extract from a Context what its trust roots
+ # or certificate is); instead, we have to somehow extract information
+ # about this stuff from how the context behaves. So this test is an
+ # integration test.
+
+ # There are good examples of how to construct relevant test-fixture
+ # data in
+ # twisted.test.test_sslverify.certificatesForAuthorityAndServer; that
+ # more directly tests the nuances of this code. Remember that this
+ # should test both positive and negative cases.
+
+ reactor = MemoryReactor()
+
+ # The certificate in question here is a self-signed certificate for
+ # 'localhost', so use 'localhost' as a hostname and the directory
+ # containing the cert itself for the CAs list.
+ endpoint = endpoints.clientFromString(
+ reactor,
+ 'tls:localhost:4321:privateKey={}:certificate={}:trustRoots={}'
+ .format(
+ escapedPEMPathName, escapedPEMPathName,
+ endpoints.quoteStringArgument(pemPath.parent().path)
+ ).encode('ascii')
+ )
+ makeHostnameEndpointSynchronous(endpoint._wrappedEndpoint)
+ d = endpoint.connect(Factory.forProtocol(Protocol))
+ host, port, factory, timeout, bindAddress = reactor.tcpClients.pop()
+ clientProtocol = factory.buildProtocol(None)
+ self.assertNoResult(d)
+ assert clientProtocol is not None
+ serverCert = PrivateCertificate.loadPEM(pemPath.getContent())
+ serverOptions = CertificateOptions(
+ privateKey=serverCert.privateKey.original,
+ certificate=serverCert.original,
+ extraCertChain=[
+ Certificate.loadPEM(chainPath.getContent()).original],
+ trustRoot=serverCert,
+ )
+ plainServer = Protocol()
+ serverProtocol = TLSMemoryBIOFactory(
+ serverOptions, isClient=False,
+ wrappedFactory=Factory.forProtocol(lambda: plainServer)
+ ).buildProtocol(None)
+ sProto, cProto, pump = connectedServerAndClient(
+ lambda: serverProtocol,
+ lambda: clientProtocol,
+ )
+ # verify privateKey
+ plainServer.transport.write(b"hello\r\n")
+ plainClient = self.successResultOf(d)
+ plainClient.transport.write(b"hi you too\r\n")
+ pump.flush()
+ self.assertFalse(plainServer.transport.disconnecting)
+ self.assertFalse(plainClient.transport.disconnecting)
+ self.assertFalse(plainServer.transport.disconnected)
+ self.assertFalse(plainClient.transport.disconnected)
+ peerCertificate = Certificate.peerFromTransport(plainServer.transport)
+ self.assertEqual(peerCertificate,
+ Certificate.loadPEM(pemPath.getContent()))
+
+
+ def test_tlsWithDefaults(self):
+ """
+ When passed a C{tls:} strports description without extra arguments,
+ L{clientFromString} returns a client endpoint whose context factory is
+ initialized with default values.
+ """
+ reactor = object()
+ endpoint = endpoints.clientFromString(reactor, b'tls:example.com:443')
+ creator = connectionCreatorFromEndpoint(reactor, endpoint)
+ self.assertEqual(creator._hostname, u'example.com')
+ self.assertEqual(endpoint._wrappedEndpoint._host, b'example.com')
+
+
+
+def replacingGlobals(function, **newGlobals):
+ """
+ Create a copy of the given function with the given globals substituted.
+
+ The globals must already exist in the function's existing global scope.
+
+ @param function: any function object.
+ @type function: L{types.FunctionType}
+
+ @param newGlobals: each keyword argument should be a global to set in the
+ new function's returned scope.
+ @type newGlobals: L{dict}
+
+ @return: a new function, like C{function}, but with new global scope.
+ """
+ try:
+ codeObject = function.func_code
+ funcGlobals = function.func_globals
+ except AttributeError:
+ codeObject = function.__code__
+ funcGlobals = function.__globals__
+ for key in newGlobals:
+ if key not in funcGlobals:
+ raise TypeError(
+ "Name bound by replacingGlobals but not present in module: {}"
+ .format(key)
+ )
+ mergedGlobals = {}
+ mergedGlobals.update(funcGlobals)
+ mergedGlobals.update(newGlobals)
+ newFunction = FunctionType(codeObject, mergedGlobals)
+ mergedGlobals[function.__name__] = newFunction
+ return newFunction
+
+
+
+class WrapClientTLSTests(unittest.TestCase):
+ """
+ Tests for the error-reporting behavior of L{wrapClientTLS} when
+ C{pyOpenSSL} is unavailable.
+ """
+
+ def test_noOpenSSL(self):
+ """
+ If SSL is not supported, L{TLSMemoryBIOFactory} will be L{None}, which
+ causes C{_wrapper} to also be L{None}. If C{_wrapper} is L{None}, then
+ an exception is raised.
+ """
+ replaced = replacingGlobals(endpoints.wrapClientTLS,
+ TLSMemoryBIOFactory=None)
+ notImplemented = self.assertRaises(NotImplementedError, replaced,
+ None, None)
+ self.assertIn("OpenSSL not available", str(notImplemented))
Modified: trunk/twisted/plugins/twisted_core.py
==============================================================================
--- trunk/twisted/plugins/twisted_core.py (original)
+++ trunk/twisted/plugins/twisted_core.py Thu Feb 11 02:46:49 2016
@@ -2,8 +2,12 @@
# See LICENSE for details.
-from twisted.internet.endpoints import _SystemdParser, _TCP6ServerParser, _StandardIOParser
+from twisted.internet.endpoints import (
+ _SystemdParser, _TCP6ServerParser, _StandardIOParser,
+ _TLSClientEndpointParser)
systemdEndpointParser = _SystemdParser()
tcp6ServerEndpointParser = _TCP6ServerParser()
stdioEndpointParser = _StandardIOParser()
+tlsClientEndpointParser = _TLSClientEndpointParser()
+
Modified: trunk/twisted/python/compat.py
==============================================================================
--- trunk/twisted/python/compat.py (original)
+++ trunk/twisted/python/compat.py Thu Feb 11 02:46:49 2016
@@ -375,6 +375,38 @@
+def _matchingString(constantString, inputString):
+ """
+ Some functions, such as C{os.path.join}, operate on string arguments which
+ may be bytes or text, and wish to return a value of the same type. In
+ those cases you may wish to have a string constant (in the case of
+ C{os.path.join}, that constant would be C{os.path.sep}) involved in the
+ parsing or processing, that must be of a matching type in order to use
+ string operations on it. L{_matchingString} will take a constant string
+ (either L{bytes} or L{unicode}) and convert it to the same type as the
+ input string. C{constantString} should contain only characters from ASCII;
+ to ensure this, it will be encoded or decoded regardless.
+
+ @param constantString: A string literal used in processing.
+ @type constantString: L{unicode} or L{bytes}
+
+ @param inputString: A byte string or text string provided by the user.
+ @type inputString: L{unicode} or L{bytes}
+
+ @return: C{constantString} converted into the same type as C{inputString}
+ @rtype: the type of C{inputString}
+ """
+ if isinstance(constantString, bytes):
+ otherType = constantString.decode("ascii")
+ else:
+ otherType = constantString.encode("ascii")
+ if type(constantString) == type(inputString):
+ return constantString
+ else:
+ return otherType
+
+
+
if _PY3:
def reraise(exception, traceback):
raise exception.with_traceback(traceback)
Modified: trunk/twisted/test/iosim.py
==============================================================================
--- trunk/twisted/test/iosim.py (original)
+++ trunk/twisted/test/iosim.py Thu Feb 11 02:46:49 2016
@@ -1,4 +1,4 @@
-# -*- test-case-name: twisted.test.test_amp.TLSTests,twisted.test.test_iosim -*-
+# -*- test-case-name: twisted.test.test_amp,twisted.test.test_iosim -*-
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.
@@ -16,11 +16,16 @@
pass
from zope.interface import implementer, directlyProvides
+from twisted.internet.endpoints import TCP4ClientEndpoint, TCP4ServerEndpoint
+from twisted.internet.protocol import Factory, Protocol
+from twisted.internet.error import ConnectionRefusedError
from twisted.python.failure import Failure
from twisted.internet import error
from twisted.internet import interfaces
+from .proto_helpers import MemoryReactorClock
+
class TLSNegotiation:
def __init__(self, obj, connectState):
@@ -427,3 +432,111 @@
cio = clientTransportFactory(c)
sio = serverTransportFactory(s)
return c, s, connect(s, sio, c, cio, debug)
+
+
+
+def _factoriesShouldConnect(clientInfo, serverInfo):
+ """
+ Should the client and server described by the arguments be connected to
+ each other, i.e. do their port numbers match?
+
+ @param clientInfo: the args for connectTCP
+ @type clientInfo: L{tuple}
+
+ @param serverInfo: the args for listenTCP
+ @type serverInfo: L{tuple}
+
+ @return: If they do match, return factories for the client and server that
+ should connect; otherwise return L{None}, indicating they shouldn't be
+ connected.
+ @rtype: L{types.NoneType} or 2-L{tuple} of (L{ClientFactory},
+ L{IProtocolFactory})
+ """
+ (clientHost, clientPort, clientFactory, clientTimeout,
+ clientBindAddress) = clientInfo
+ (serverPort, serverFactory, serverBacklog,
+ serverInterface) = serverInfo
+ if serverPort == clientPort:
+ return clientFactory, serverFactory
+ else:
+ return None
+
+
+
+class ConnectionCompleter(object):
+ """
+ A L{ConnectionCompleter} can cause synthetic TCP connections established by
+ L{MemoryReactor.connectTCP} and L{MemoryReactor.listenTCP} to succeed or
+ fail.
+ """
+ def __init__(self, memoryReactor):
+ """
+ Create a L{ConnectionCompleter} from a L{MemoryReactor}.
+
+ @param memoryReactor: The reactor to attach to.
+ @type memoryReactor: L{MemoryReactor}
+ """
+ self._reactor = memoryReactor
+
+
+ def succeedOnce(self, debug=False):
+ """
+ Complete a single TCP connection established on this
+ L{ConnectionCompleter}'s L{MemoryReactor}.
+
+ @param debug: A flag; whether to dump output from the established
+ connection to stdout.
+ @type debug: L{bool}
+
+ @return: a pump for the connection, or L{None} if no connection could
+ be established.
+ @rtype: L{IOPump} or L{None}
+ """
+ memoryReactor = self._reactor
+ for clientIdx, clientInfo in enumerate(memoryReactor.tcpClients):
+ for serverInfo in memoryReactor.tcpServers:
+ factories = _factoriesShouldConnect(clientInfo, serverInfo)
+ if factories:
+ memoryReactor.tcpClients.remove(clientInfo)
+ memoryReactor.connectors.pop(clientIdx)
+ clientFactory, serverFactory = factories
+ clientProtocol = clientFactory.buildProtocol(None)
+ serverProtocol = serverFactory.buildProtocol(None)
+ serverTransport = makeFakeServer(serverProtocol)
+ clientTransport = makeFakeClient(clientProtocol)
+ return connect(serverProtocol, serverTransport,
+ clientProtocol, clientTransport,
+ debug)
+
+
+ def failOnce(self, reason=Failure(ConnectionRefusedError())):
+ """
+ Fail a single TCP connection established on this
+ L{ConnectionCompleter}'s L{MemoryReactor}.
+
+ @param reason: the reason to provide that the connection failed.
+ @type reason: L{Failure}
+ """
+ self._reactor.tcpClients.pop(0)[2].clientConnectionFailed(
+ self._reactor.connectors.pop(0), reason
+ )
+
+
+
+def connectableEndpoint(debug=False):
+ """
+ Create an endpoint that can be fired on demand.
+
+ @param debug: A flag; whether to dump output from the established
+ connection to stdout.
+ @type debug: L{bool}
+
+ @return: A client endpoint, and an object that will cause one of the
+ L{Deferred}s returned by that client endpoint.
+ @rtype: 2-L{tuple} of (L{IStreamClientEndpoint}, L{ConnectionCompleter})
+ """
+ reactor = MemoryReactorClock()
+ clientEndpoint = TCP4ClientEndpoint(reactor, "0.0.0.0", 4321)
+ serverEndpoint = TCP4ServerEndpoint(reactor, 4321)
+ serverEndpoint.listen(Factory.forProtocol(Protocol))
+ return clientEndpoint, ConnectionCompleter(reactor)