Re: Weird epoller bug
Robert DiFalco <[email protected]> Tue, 17 Dec 2019 13:21:49 -0800
| Newsgroups | gmane.comp.python.twisted |
|---|---|
| Message-ID | <CAAXGW-yMByxY5tcd7a5LZ57UXMRktfqe0vdX+i6wm+EXMG=xiA@mail.gmail.com> |
--===============6862763626060936198==
Content-Type: multipart/alternative; boundary="0000000000002932560599eceb41"
--0000000000002932560599eceb41
Content-Type: text/plain; charset="UTF-8"
One more thing, there seems to be an error that can occur in _newclient.py
where _finishedRequest will be type None and it's errback will be attempted
to be called. Consider this in _newclient.py. If an exception occurs in in
maybeDeferred before _finishedRequest is assigned, then it will be None
when it is chained elsewhere.
def request(self, request):
"""
Issue C{request} over C{self.transport} and return a L{Deferred} which
will fire with a L{Response} instance or an error.
@param request: The object defining the parameters of the request to
issue.
@type request: L{Request}
@rtype: L{Deferred}
@return: The deferred may errback with L{RequestGenerationFailed} if
the request was not fully written to the transport due to a local
error. It may errback with L{RequestTransmissionFailed} if it was
not fully written to the transport due to a network error. It may
errback with L{ResponseFailed} if the request was sent (not
necessarily received) but some or all of the response was lost. It
may errback with L{RequestNotSent} if it is not possible to send
any more requests using this L{HTTP11ClientProtocol}.
"""
if self._state != 'QUIESCENT':
return fail(RequestNotSent())
self._state = 'TRANSMITTING'
_requestDeferred = maybeDeferred(request.writeTo, self.transport)
def cancelRequest(ign):
# Explicitly cancel the request's deferred if it's still trying to
# write when this request is cancelled.
if self._state in (
'TRANSMITTING', 'TRANSMITTING_AFTER_RECEIVING_RESPONSE'):
_requestDeferred.cancel()
else:
self.transport.abortConnection()
self._disconnectParser(Failure(CancelledError()))
self._finishedRequest = Deferred(cancelRequest)
On Tue, Dec 17, 2019 at 11:19 AM Robert DiFalco <[email protected]>
wrote:
> I have a theory that since this runs every 20 seconds that it is falling
> behind somehow. Ulimits is too low, twisted is maybe eating the too many
> file handles exception and then trying to close a handle that doesn't
> actually exist. Dunno.
>
> On Tue, Dec 17, 2019 at 10:18 AM Robert DiFalco <[email protected]>
> wrote:
>
>> We recently switched from these versions and recently our TLS health
>> check using treq seems to be using more and more file handles and getting
>> this exception. Additionally our health check gets response time outs at a
>> much higher frequency. Any tips on how I would go about debugging this? Or
>> is this perhaps a known issue in one of the versions below and I need to
>> either bump up or down a release version? Thanks!
>>
>> twisted 16.6.0 -> 19.19.0
>> treq 15.1.0 -> 18.6.0
>>
>> File
>> "/home/foo/foo/local/lib/python2.7/site-packages/twisted/internet/epollreactor.py",
>> line 183, in removeWriter
>> EPOLLOUT, EPOLLIN)
>> File
>> "/home/foo/foo/local/lib/python2.7/site-packages/twisted/internet/epollreactor.py",
>> line 160, in _remove
>> self._poller.unregister(fd)
>> exceptions.IOError: [Errno 2] No such file or directory
>>
>>
>>
>>
>>
--0000000000002932560599eceb41
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr">One more thing, there seems to be an erro=
r that can occur in _newclient.py where _finishedRequest will be type None =
and it's errback will be attempted to be called. Consider this in _newc=
lient.py. If an exception occurs in in maybeDeferred before _finishedReques=
t is assigned, then it will be None when it is chained elsewhere.=C2=A0<br>=
<pre style=3D"background-color:rgb(43,43,43);color:rgb(169,183,198);font-fa=
mily:Menlo;font-size:9.8pt"><span style=3D"color:rgb(204,120,50)">def </spa=
n><span style=3D"color:rgb(255,198,109)">request</span>(<span style=3D"colo=
r:rgb(148,85,141)">self</span><span style=3D"color:rgb(204,120,50)">, </spa=
n>request):<br> <span style=3D"color:rgb(98,151,85);font-style:italic">&=
quot;""<br></span><span style=3D"color:rgb(98,151,85);font-style:=
italic"> Issue C{request} over C{self.transport} and return a L{Deferred=
} which<br></span><span style=3D"color:rgb(98,151,85);font-style:italic"> =
will fire with a L{Response} instance or an error.<br></span><span style=
=3D"color:rgb(98,151,85);font-style:italic"><br></span><span style=3D"color=
:rgb(98,151,85);font-style:italic"> @param request: The object defining =
the parameters of the request to<br></span><span style=3D"color:rgb(98,151,=
85);font-style:italic"> issue.<br></span><span style=3D"color:rgb(98,=
151,85);font-style:italic"> @type request: L{Request}<br></span><span st=
yle=3D"color:rgb(98,151,85);font-style:italic"><br></span><span style=3D"co=
lor:rgb(98,151,85);font-style:italic"> @rtype: L{Deferred}<br></span><sp=
an style=3D"color:rgb(98,151,85);font-style:italic"> @return: The deferr=
ed may errback with L{RequestGenerationFailed} if<br></span><span style=3D"=
color:rgb(98,151,85);font-style:italic"> the request was not fully w=
ritten to the transport due to a local<br></span><span style=3D"color:rgb(9=
8,151,85);font-style:italic"> error. It may errback with L{RequestT=
ransmissionFailed} if it was<br></span><span style=3D"color:rgb(98,151,85);=
font-style:italic"> not fully written to the transport due to a netw=
ork error. It may<br></span><span style=3D"color:rgb(98,151,85);font-style=
:italic"> errback with L{ResponseFailed} if the request was sent (no=
t<br></span><span style=3D"color:rgb(98,151,85);font-style:italic"> =
necessarily received) but some or all of the response was lost. It<br></sp=
an><span style=3D"color:rgb(98,151,85);font-style:italic"> may errba=
ck with L{RequestNotSent} if it is not possible to send<br></span><span sty=
le=3D"color:rgb(98,151,85);font-style:italic"> any more requests usi=
ng this L{HTTP11ClientProtocol}.<br></span><span style=3D"color:rgb(98,151,=
85);font-style:italic"> """<br></span><span style=3D"colo=
r:rgb(98,151,85);font-style:italic"> </span><span style=3D"color:rgb(204=
,120,50)">if </span><span style=3D"color:rgb(148,85,141)">self</span>._stat=
e !=3D <span style=3D"color:rgb(165,194,97)">'QUIESCENT'</span>:<br=
> <span style=3D"color:rgb(204,120,50)">return </span>fail(RequestNo=
tSent())<br><br> <span style=3D"color:rgb(148,85,141)">self</span>._stat=
e =3D <span style=3D"color:rgb(165,194,97)">'TRANSMITTING'<br></spa=
n><span style=3D"color:rgb(165,194,97)"> </span>_requestDeferred =3D may=
beDeferred(request.writeTo<span style=3D"color:rgb(204,120,50)">, </span><s=
pan style=3D"color:rgb(148,85,141)">self</span>.transport)<br><br> <span=
style=3D"color:rgb(204,120,50)">def </span><span style=3D"color:rgb(255,19=
8,109)">cancelRequest</span>(ign):<br> <span style=3D"color:rgb(128,=
128,128)"># Explicitly cancel the request's deferred if it's still =
trying to<br></span><span style=3D"color:rgb(128,128,128)"> # write =
when this request is cancelled.<br></span><span style=3D"color:rgb(128,128,=
128)"> </span><span style=3D"color:rgb(204,120,50)">if </span>self._=
state <span style=3D"color:rgb(204,120,50)">in </span>(<br> =
<span style=3D"color:rgb(165,194,97)">'TRANSMITTING'</span><span st=
yle=3D"color:rgb(204,120,50)">, </span><span style=3D"color:rgb(165,194,97)=
">'TRANSMITTING_AFTER_RECEIVING_RESPONSE'</span>):<br> _=
requestDeferred.cancel()<br> <span style=3D"color:rgb(204,120,50)">e=
lse</span>:<br> self.transport.abortConnection()<br> =
self._disconnectParser(Failure(CancelledError()))<br> <span style=3D"col=
or:rgb(148,85,141)">self</span>._finishedRequest =3D Deferred(cancelRequest=
)<br></pre></div><div><br></div><div><br></div><br><div class=3D"gmail_quot=
e"><div dir=3D"ltr" class=3D"gmail_attr">On Tue, Dec 17, 2019 at 11:19 AM R=
obert DiFalco <<a href=3D"mailto:[email protected]">robert.difalc=
[email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=
=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding=
-left:1ex"><div dir=3D"ltr">I have a theory that since this runs every 20 s=
econds that it is falling behind somehow. Ulimits is too low, twisted is ma=
ybe eating the too many file handles exception and then trying to close a h=
andle that doesn't actually exist. Dunno.</div><br><div class=3D"gmail_=
quote"><div dir=3D"ltr" class=3D"gmail_attr">On Tue, Dec 17, 2019 at 10:18 =
AM Robert DiFalco <<a href=3D"mailto:[email protected]" target=3D=
"_blank">[email protected]</a>> wrote:<br></div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex"><div dir=3D"ltr">We recently switched from=
these versions and recently our TLS health check using treq seems to be us=
ing more and more file handles and getting this exception. Additionally our=
health check gets response time outs at a much higher frequency. Any tips =
on how I would go about debugging this? Or is this perhaps a known issue in=
one of the versions below and I need to either bump up or down a release v=
ersion? Thanks!<div><br></div><div>twisted 16.6.0 -> 19.19.0</div><div>t=
req 15.1.0 -> 18.6.0<div><div><br></div><div>=C2=A0 File "/home/foo=
/foo/local/lib/python2.7/site-packages/twisted/internet/epollreactor.py&quo=
t;, line 183, in removeWriter<br>=C2=A0 =C2=A0 EPOLLOUT, EPOLLIN)<br>=C2=A0=
File "/home/foo/foo/local/lib/python2.7/site-packages/twisted/interne=
t/epollreactor.py", line 160, in _remove<br>=C2=A0 =C2=A0 self._poller=
.unregister(fd)<br>exceptions.IOError: [Errno 2] No such file or directory<=
br><br><br><table cellpadding=3D"0" style=3D"border-spacing:0px;line-height=
:20px;color:rgb(0,0,0);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-=
serif;font-size:14px"><tbody><tr><td colspan=3D"2" style=3D"vertical-align:=
top;width:auto;padding:2px 0px"><span style=3D"vertical-align:top"><br><br>=
</span></td></tr></tbody></table></div></div></div></div>
</blockquote></div>
</blockquote></div></div>
--0000000000002932560599eceb41--
--===============6862763626060936198==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KVHdpc3RlZC1Q
eXRob24gbWFpbGluZyBsaXN0ClR3aXN0ZWQtUHl0aG9uQHR3aXN0ZWRtYXRyaXguY29tCmh0dHBz
Oi8vdHdpc3RlZG1hdHJpeC5jb20vY2dpLWJpbi9tYWlsbWFuL2xpc3RpbmZvL3R3aXN0ZWQtcHl0
aG9uCg==
--===============6862763626060936198==--