TimeoutError happening when Pyro4 under heavy load. How do I handle this?
"Vernon D. Cole" <[email protected]>
| Newsgroups | gmane.comp.python.pyro |
|---|---|
| Message-ID | <CAH-ZgAf6Xm7ZzSbAgOHHTiUF_ON4ouxPSHndYYjoDwtuizOqpg@mail.gmail.com> |
Help me, Obi-Wan...! Please be patient while I tell my story. (This is intended to build up a pity factor.) It also explains the complex tangle of software involved in my problem. I have been for many years the maintainer of adodbapi. It is an open source, pure Python, PEP 249 compliant, data access module. Because ADO is a Microsoft proprietary beast, it has always been limited to Windows use, which, as a Linux user by preference, bothers me. ADO is capable of connecting to a wide variety of tabular data sources, but is most frequently seen as a tool for accessing Microsoft SQL Server. Last December, I went to work for eHealth Africa to help them with their ongoing project of wiping out polio. Not only can I help bring about world peace (medically speaking), but I got the chance to really learn and use django, which has been a long time goal. I flew to Africa and started the project of putting their data into a django-operated PostgreSQL database. A few weeks ago, I became aware that the Center for Disease Control needs this data to be on an SQL Server, because of the limitations of their C#/Windows data analysis tools. It happens that there is a third-party add-on for django (called django-mssql) which talks to SQL Server. I learned about this by finding my name in the documentation -- because the data access method it uses is a fork of an old version of adodbapi. I long ago rolled the new features of that fork back into the main fork of adodbapi so that they could use the release version (which has, for example, Python3 and IronPython support) in place of their customized version. They had not made the transition, yet. About this same time, a discussion popped up on the django developers list to the effect that they would welcome django-mssql into supported status if only they could test the blasted thing using Linux. I replied with the suggestion that a remote-proceedure-call interface using a Windows server should be able to make that happen, and I just happen to know the guy who maintains the key module. I was invited to give it a try. So I pulled the development repositories of all three projects into my machine and started work. The first task was to select an RPC tool. Pyro4 won the nod. I went to work. I can now run the full adodbapi test suite on Ubuntu with no errors, using a Pyro4 connection to a Windows 2008 Server. Then I pulled the forked backend out of django-mssql and dropped mine in. After a bit of code-cleaning, it runs the django-mssql test suite, and does a fair job on (huge) django test suite, when running on the 2008 Server. Three days ago, I tried the last step: run the django test suite on Ubuntu using my new remote driver. A couple of quick fixes later and it almost works. But this "almost" seems to be deep in the bowels of Pyro4. A few minutes into the test run my server collects enough TimeoutErrors that it (it seems to me) runs out of worker threads and things come to a screeching halt. I have no hope of reducing this to a simple test case -- the simple cases work fine. My guess is that the four timing loops -- django and Pyro4 on the client, and Pyro4 and pywin32/COM on the server -- are somehow stepping on each other. I am hoping that somebody who understands the timing better than I can look over my code and see what I did wrong. My code for timeout handling in the server is a shot in the dark -- obviously wrong. The code can most easily be viewed on the repositories at: http://sourceforge.net/p/adodbapi/code/ci/default/tree/ see "remote.py" and "server.py" The other toys involved are: https://bitbucket.org/vernondcole/django-mssql-ado-merge and https://github.com/django/django I am testing on 1.5.x/stable The printouts from the two consoles are concatenated in the attached file. They separate at a line of "-----------------------------------" I _know_ this is a terrible thing to ask, but I am desperate. -- Vernon Cole ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. This 200-page book is written by three acclaimed leaders in the field. The early access version is available now. Download your free book today! http://p.sf.net/sfu/neotech_d2d_may _______________________________________________ Pyro-core mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/pyro-core
Pyro_stall_console_dump.txt
(text/plain, 25.5 KB)
Client configuration: Ubuntu 13.04 64 bit.
v v v v v on the client v v v v v v v v
vernon@VPad:~/PycharmProjects/ado/adodbapi$ python is64bit.py
('is64bit.Python() =', True, 'is64bit.os() =', True)
vernon@VPad:~/PycharmProjects/ado/adodbapi$ python
Python 2.7.4 (default, Apr 19 2013, 18:28:01)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
--------------------------------------------------------------------------------------------
vernon@VPad:~/PycharmProjects/django-mssql-ado-merge/tests/test_main$ python manage.py test
Creating test database for alias 'default'...
.......................................[2013-05-11 10:21:55.721,Pyro4.core,ERROR] invoke: reply sequence out of sync, got 0 expected 9
E.F^C^CTraceback (most recent call last):
File "manage.py", line 11, in <module>
execute_manager(settings)
File "/home/vernon/PycharmProjects/django/django/core/management/__init__.py", line 469, in execute_manager
utility.execute()
File "/home/vernon/PycharmProjects/django/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/vernon/PycharmProjects/django/django/core/management/commands/test.py", line 51, in run_from_argv
super(Command, self).run_from_argv(argv)
File "/home/vernon/PycharmProjects/django/django/core/management/base.py", line 222, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/vernon/PycharmProjects/django/django/core/management/commands/test.py", line 72, in execute
super(Command, self).execute(*args, **options)
File "/home/vernon/PycharmProjects/django/django/core/management/base.py", line 255, in execute
output = self.handle(*args, **options)
File "/home/vernon/PycharmProjects/django/django/core/management/commands/test.py", line 89, in handle
failures = test_runner.run_tests(test_labels)
File "/home/vernon/PycharmProjects/django/django/test/simple.py", line 368, in run_tests
result = self.run_suite(suite)
File "/home/vernon/PycharmProjects/django/django/test/simple.py", line 328, in run_suite
verbosity=self.verbosity, failfast=self.failfast).run(suite)
File "/usr/lib/python2.7/unittest/runner.py", line 151, in run
test(result)
File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
test(result)
File "/home/vernon/PycharmProjects/django/django/test/testcases.py", line 265, in __call__
super(SimpleTestCase, self).__call__(result)
File "/usr/lib/python2.7/unittest/case.py", line 396, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python2.7/unittest/case.py", line 332, in run
testMethod()
File "/home/vernon/PycharmProjects/django-mssql-ado-merge/tests/test_main/regressiontests/models.py", line 278, in testDistinctRelated
q = list(Bug58TableRecipe.objects.filter(item__ingredient__name__in=['bread','butter']).distinct())
File "/home/vernon/PycharmProjects/django/django/db/models/query.py", line 92, in __len__
self._result_cache.extend(self._iter)
File "/home/vernon/PycharmProjects/django/django/db/models/query.py", line 301, in iterator
for row in compiler.results_iter():
File "/home/vernon/PycharmProjects/django/django/db/models/sql/compiler.py", line 775, in results_iter
for rows in self.execute_sql(MULTI):
File "/home/vernon/PycharmProjects/django/django/db/models/sql/compiler.py", line 854, in <lambda>
result = iter((lambda: cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)),
File "/usr/local/lib/python2.7/dist-packages/adodbapi/remote.py", line 445, in fetchmany
return api.SQLrows(self.rs, len(self.rs), self)
File "/usr/local/lib/python2.7/dist-packages/adodbapi/apibase.py", line 529, in __init__
self.converters = cursor.converters
File "/usr/local/lib/python2.7/dist-packages/adodbapi/remote.py", line 378, in __getattr__
return self.remote_cursor.get_attribute_for_remote(key)
File "/usr/local/lib/python2.7/dist-packages/Pyro4/core.py", line 157, in __call__
return self.__send(self.__name, args, kwargs)
File "/usr/local/lib/python2.7/dist-packages/Pyro4/core.py", line 296, in _pyroInvoke
msgType, flags, seq, data = MessageFactory.getMessage(self._pyroConnection, MessageFactory.MSG_RESULT)
File "/usr/local/lib/python2.7/dist-packages/Pyro4/core.py", line 574, in getMessage
headerdata = connection.recv(cls.HEADERSIZE)
File "/usr/local/lib/python2.7/dist-packages/Pyro4/socketutil.py", line 440, in recv
return receiveData(self.sock, size)
File "/usr/local/lib/python2.7/dist-packages/Pyro4/socketutil.py", line 153, in receiveData
data=sock.recv(size, socket.MSG_WAITALL)
File "/usr/lib/python2.7/unittest/signals.py", line 36, in __call__
self.default_handler(signum, frame)
KeyboardInterrupt
vernon@VPad:~/PycharmProjects/django-mssql-ado-merge/tests/test_main$
-----------------------------------------------------------------------------
On the server: Windows 2008
v v v v v v v v v v v v v v
C:\Users\vernon\Desktop\ado\adodbapi>py is64bit.py
('is64bit.Python() =', True, 'is64bit.os() =', True)
C:\Users\vernon\Desktop\ado\adodbapi>py
Python 2.7.4 (default, Apr 6 2013, 19:55:15) [MSC v.1500 64 bit (AMD64)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>>
--------------------------------------------------------------------------
C:\Users\vernon\Desktop\django-mssql-ado-merge\tests\test_main>py -m adodbapi.se
rver host=0.0.0.0 --verbose=4
[2013-05-11 10:20:05.796,Pyro4,INFO] Pyro log configured using built-in defaults
, level=DEBUG
adodbapi v2.5.0.d9d
adodbapi.server v2.5.0.d9
[2013-05-11 10:20:05.858,Pyro4.socketserver.threadpool,INFO] starting thread poo
l socketserver
[2013-05-11 10:20:05.858,Pyro4.tpjobqueue,DEBUG] spawned new idle worker: Pyro-W
orker-46325208
[2013-05-11 10:20:05.874,Pyro4.tpjobqueue,DEBUG] spawned new idle worker: Pyro-W
orker-46325544
[2013-05-11 10:20:05.874,Pyro4.tpjobqueue,DEBUG] spawned new idle worker: Pyro-W
orker-46350576
[2013-05-11 10:20:05.874,Pyro4.tpjobqueue,DEBUG] spawned new idle worker: Pyro-W
orker-46350912
[2013-05-11 10:20:05.890,Pyro4.socketserver.threadpool,INFO] 4 workers started
[2013-05-11 10:20:05.890,Pyro4.core,DEBUG] created daemon on 0.0.0.0:9099
ado.connection server running on uri=PYRO:[email protected]:9099
use HOST=nnn and PORT=nnn to change interface addresses
(use ^C or <Ctrl-Break> to interrupt...)
[2013-05-11 10:20:05.904,Pyro4.core,INFO] daemon 0.0.0.0:9099 entering requestlo
op
[2013-05-11 10:20:05.904,Pyro4.socketserver.threadpool,DEBUG] threadpool server
requestloop
[2013-05-11 10:21:20.551,Pyro4.socketserver.threadpool,DEBUG] connected ('192.16
8.25.44', 50178)
[2013-05-11 10:21:20.614,Pyro4.socketserver.threadpool,DEBUG] connected ('192.16
8.25.44', 50179)
adodbapi v2.5.0.d9d attempting: "PROVIDER=sqlncli10;DATA SOURCE=EHA-WINServ1\SQL
Express;Initial Catalog=master;Integrated Security=SSPI;"
adodbapi New connection at 2C345C0
[2013-05-11 10:21:20.816,Pyro4.socketserver.threadpool,DEBUG] disconnected ('192
.168.25.44', 50178)
adodbapi New cursor at 2C34C88 on conn 2C345C0
[2013-05-11 10:21:20.862,Pyro4.socketserver.threadpool,DEBUG] connected ('192.16
8.25.44', 50180)
Params= []
Executing command="CREATE DATABASE [test_django_remote_test] "
[2013-05-11 10:21:23.842,Pyro4.socketserver.threadpool,DEBUG] disconnected ('192
.168.25.44', 50180)
adodbapi Closed connection at 2C345C0
[2013-05-11 10:21:23.858,Pyro4.socketserver.threadpool,DEBUG] disconnected ('192
.168.25.44', 50179)
[2013-05-11 10:21:23.874,Pyro4.socketserver.threadpool,DEBUG] connected ('192.16
8.25.44', 50181)
[2013-05-11 10:21:23.951,Pyro4.socketserver.threadpool,DEBUG] connected ('192.16
8.25.44', 50182)
adodbapi v2.5.0.d9d attempting: "PROVIDER=sqlncli10;DATA SOURCE=EHA-WINServ1\SQL
Express;Initial Catalog=test_django_remote_test;Integrated Security=SSPI;"
adodbapi New connection at 2C34940
[2013-05-11 10:21:24.996,Pyro4.socketserver.threadpool,DEBUG] disconnected ('192
.168.25.44', 50181)
adodbapi New cursor at 2C34BE0 on conn 2C34940
adodbapi New cursor at 2C3D668 on conn 2C34940
[2013-05-11 10:21:25.434,Pyro4.socketserver.threadpool,DEBUG] connected ('192.16
8.25.44', 50183)
Params= []
Executing command="CREATE TABLE [regressiontests_bug19table] (
[id] int IDENTITY (1, 1) NOT NULL PRIMARY KEY,
[choice] nvarchar(max) NOT NULL
)
;"
Params= []
Executing command="CREATE TABLE [regressiontests_bug21table] (
[id] int IDENTITY (1, 1) NOT NULL PRIMARY KEY,
[a] nvarchar(50) NOT NULL,
[d] decimal(5, 2) NOT NULL
)
;"
Params= []
Executing command="CREATE TABLE [regressiontests_bug27table] (
[id] int IDENTITY (1, 1) NOT NULL PRIMARY KEY,
[a] int NOT NULL
)
;"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[[[[[... hundreds of lines clipped out here .... resuming at 1st timeout ]]]]]]]
the "drop table" Exception is expected -- it is a cleanup step
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
[2013-05-11 10:22:14.105,Pyro4.socketserver.threadpool,DEBUG] connected ('192.16
8.25.44', 50458)
adodbapi v2.5.0.d9d attempting: "PROVIDER=sqlncli10;DATA SOURCE=EHA-WINServ1\SQL
Express;Initial Catalog=test_django_remote_test;Integrated Security=SSPI;"
adodbapi New connection at 2C49080
adodbapi New cursor at 2D5E320 on conn 2C49080[
2013-05-11 10:22:14.184,Pyro4.socketserver.threadpool,DEBUG] disconnected ('192.
168.25.44', 50457)
[2013-05-11 10:22:14.200,Pyro4.socketserver.threadpool,DEBUG] connected ('192.16
8.25.44', 50459)
Params= []
Executing command="drop table dbapi20test_booze"
[2013-05-11 10:22:14.230,Pyro4.core,DEBUG] Exception occurred while handling req
uest: DatabaseError(u'(-2147352567, \'Exception occurred.\', (0, u\'Microsoft SQ
L Server Native Client 10.0\', u"Cannot drop the table \'dbapi20test_booze\', be
cause it does not exist or you do not have permission.", None, 0, -2147217865),
None)\nCommand:\ndrop table dbapi20test_booze\nParameters:\n[]',)
Params= []
Executing command="drop table dbapi20test_barflys"
[2013-05-11 10:22:14.278,Pyro4.core,DEBUG] Exception occurred while handling req
uest: DatabaseError(u'(-2147352567, \'Exception occurred.\', (0, u\'Microsoft SQ
L Server Native Client 10.0\', u"Cannot drop the table \'dbapi20test_barflys\',
because it does not exist or you do not have permission.", None, 0, -2147217865)
, None)\nCommand:\ndrop table dbapi20test_barflys\nParameters:\n[]',)
[2013-05-11 10:22:14.292,Pyro4.core,DEBUG] Exception occurred while handling req
uest: TimeoutError('receiving: timeout',)
[2013-05-11 10:22:14.292,Pyro4.tpjobqueue,DEBUG] worker halted: Pyro-Worker-4635
0576
Exception in thread Pyro-Worker-46350576 :
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\site-packages\Pyro4\tpjobqueue.py", line 62, in run
self.job()
File "C:\Python27\lib\site-packages\Pyro4\socketserver\threadpoolserver.py", l
ine 32, in __call__
self.daemon.handleRequest(self.csock)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 769, in handleRequest
msgType, flags, seq, data = MessageFactory.getMessage(conn, MessageFactory.M
SG_INVOKE)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 574, in getMessage
headerdata = connection.recv(cls.HEADERSIZE)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 440, in recv
return receiveData(self.sock, size)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 187, in receive
Data
raise TimeoutError("receiving: timeout")
TimeoutError: receiving: timeout
[2013-05-11 10:22:14.309,Pyro4.socketserver.threadpool,DEBUG] disconnected ('192
.168.25.44', 50459)
adodbapi Closed connection at 2C49080
[2013-05-11 10:22:14.325,Pyro4.socketserver.threadpool,DEBUG] disconnected ('192
.168.25.44', 50458)
[2013-05-11 10:22:14.339,Pyro4.socketserver.threadpool,DEBUG] connected ('192.16
8.25.44', 50460)
[2013-05-11 10:22:14.403,Pyro4.socketserver.threadpool,DEBUG] connected ('192.16
8.25.44', 50461)
adodbapi v2.5.0.d9d attempting: "PROVIDER=sqlncli10;DATA SOURCE=EHA-WINServ1\SQL
Express;Initial Catalog=test_django_remote_test;Integrated Security=SSPI;"
adodbapi New connection at 2C3DCF8
[[[[[[[[[[[[[[[[[[[[[ another clip ]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
[2013-05-11 10:22:16.882,Pyro4.socketserver.threadpool,DEBUG] connected ('192.16
8.25.44', 50482)
Params= [Name: p0, Dir.: Input, Type: adInteger, Size: 0, Value: "1", Precision:
0, NumericScale: 0
Name: p1, Dir.: Input, Type: adInteger, Size: 0, Value: "2", Precision: 0, Numer
icScale: 0
Name: p2, Dir.: Input, Type: adBSTR, Size: 6, Value: "1 Tbsp", Precision: 0, Num
ericScale: 0]
Executing command="SET NOCOUNT ON;DECLARE @sqlserver_ado_return_id table ([id] i
nt);INSERT INTO [regressiontests_bug58tableitem] ([recipe_id], [ingredient_id],
[amount]) OUTPUT INSERTED.[id] INTO @sqlserver_ado_return_id VALUES (%s, %s, %s)
;SELECT * FROM @sqlserver_ado_return_id"
adodbapi New cursor at 2E4CE80 on conn 2E4C860
[2013-05-11 10:22:17.023,Pyro4.socketserver.threadpool,DEBUG] connected ('192.16
8.25.44', 50483)
Params= [Name: p0, Dir.: Input, Type: adBSTR, Size: 5, Value: "bread", Precision
: 0, NumericScale: 0
Name: p1, Dir.: Input, Type: adBSTR, Size: 6, Value: "butter", Precision: 0, Num
ericScale: 0]
Executing command="SELECT DISTINCT [regressiontests_bug58tablerecipe].[id], [reg
ressiontests_bug58tablerecipe].[name] FROM [regressiontests_bug58tablerecipe] IN
NER JOIN [regressiontests_bug58tableitem] ON ([regressiontests_bug58tablerecipe]
.[id] = [regressiontests_bug58tableitem].[recipe_id]) INNER JOIN [regressiontest
s_bug58tableingredient] ON ([regressiontests_bug58tableitem].[ingredient_id] = [
regressiontests_bug58tableingredient].[id]) WHERE [regressiontests_bug58tableing
redient].[name] IN (%s, %s)"
[2013-05-11 10:22:24.276,Pyro4.core,DEBUG] Exception occurred while handling req
uest: TimeoutError('receiving: timeout',)
[2013-05-11 10:22:24.276,Pyro4.tpjobqueue,DEBUG] worker halted: Pyro-Worker-4632
5208
Exception in thread Pyro-Worker-46325208 :
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\site-packages\Pyro4\tpjobqueue.py", line 62, in run
self.job()
File "C:\Python27\lib\site-packages\Pyro4\socketserver\threadpoolserver.py", l
ine 32, in __call__
self.daemon.handleRequest(self.csock)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 769, in handleRequest
msgType, flags, seq, data = MessageFactory.getMessage(conn, MessageFactory.M
SG_INVOKE)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 574, in getMessage
headerdata = connection.recv(cls.HEADERSIZE)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 440, in recv
return receiveData(self.sock, size)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 187, in receive
Data
raise TimeoutError("receiving: timeout")
TimeoutError: receiving: timeout
[2013-05-11 10:22:24.309,Pyro4.core,DEBUG] Exception occurred while handling req
uest: TimeoutError('receiving: timeout',)
[2013-05-11 10:22:24.309,Pyro4.tpjobqueue,DEBUG] worker halted: Pyro-Worker-4649
5728
Exception in thread Pyro-Worker-46495728 :
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\site-packages\Pyro4\tpjobqueue.py", line 62, in run
self.job()
File "C:\Python27\lib\site-packages\Pyro4\socketserver\threadpoolserver.py", l
ine 32, in __call__
self.daemon.handleRequest(self.csock)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 769, in handleRequest
msgType, flags, seq, data = MessageFactory.getMessage(conn, MessageFactory.M
SG_INVOKE)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 574, in getMessage
headerdata = connection.recv(cls.HEADERSIZE)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 440, in recv
return receiveData(self.sock, size)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 187, in receive
Data
raise TimeoutError("receiving: timeout")
TimeoutError: receiving: timeout
[2013-05-11 10:22:25.244,Pyro4.core,DEBUG] Exception occurred while handling req
uest: TimeoutError('receiving: timeout',)
[2013-05-11 10:22:25.244,Pyro4.tpjobqueue,DEBUG] worker halted: Pyro-Worker-4643
9000
Exception in thread Pyro-Worker-46439000 :
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\site-packages\Pyro4\tpjobqueue.py", line 62, in run
self.job()
File "C:\Python27\lib\site-packages\Pyro4\socketserver\threadpoolserver.py", l
ine 32, in __call__
self.daemon.handleRequest(self.csock)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 769, in handleRequest
msgType, flags, seq, data = MessageFactory.getMessage(conn, MessageFactory.M
SG_INVOKE)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 574, in getMessage
headerdata = connection.recv(cls.HEADERSIZE)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 440, in recv
return receiveData(self.sock, size)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 187, in receive
Data
raise TimeoutError("receiving: timeout")
TimeoutError: receiving: timeout
[2013-05-11 10:22:25.276,Pyro4.core,DEBUG] Exception occurred while handling req
uest: TimeoutError('receiving: timeout',)
[2013-05-11 10:22:25.276,Pyro4.tpjobqueue,DEBUG] worker halted: Pyro-Worker-4635
0912
Exception in thread Pyro-Worker-46350912 :
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\site-packages\Pyro4\tpjobqueue.py", line 62, in run
self.job()
File "C:\Python27\lib\site-packages\Pyro4\socketserver\threadpoolserver.py", l
ine 32, in __call__
self.daemon.handleRequest(self.csock)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 769, in handleRequest
msgType, flags, seq, data = MessageFactory.getMessage(conn, MessageFactory.M
SG_INVOKE)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 574, in getMessage
headerdata = connection.recv(cls.HEADERSIZE)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 440, in recv
return receiveData(self.sock, size)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 187, in receive
Data
raise TimeoutError("receiving: timeout")
TimeoutError: receiving: timeout
[2013-05-11 10:22:46.460,Pyro4.core,DEBUG] Exception occurred while handling req
uest: TimeoutError('receiving: timeout',)
[2013-05-11 10:22:46.460,Pyro4.tpjobqueue,DEBUG] worker halted: Pyro-Worker-4649
3992
Exception in thread Pyro-Worker-46493992 :
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\site-packages\Pyro4\tpjobqueue.py", line 62, in run
self.job()
File "C:\Python27\lib\site-packages\Pyro4\socketserver\threadpoolserver.py", l
ine 32, in __call__
self.daemon.handleRequest(self.csock)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 769, in handleRequest
msgType, flags, seq, data = MessageFactory.getMessage(conn, MessageFactory.M
SG_INVOKE)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 574, in getMessage
headerdata = connection.recv(cls.HEADERSIZE)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 440, in recv
return receiveData(self.sock, size)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 187, in receive
Data
raise TimeoutError("receiving: timeout")
TimeoutError: receiving: timeout
[2013-05-11 10:22:46.617,Pyro4.core,DEBUG] Exception occurred while handling req
uest: TimeoutError('receiving: timeout',)
[2013-05-11 10:22:46.617,Pyro4.tpjobqueue,DEBUG] worker halted: Pyro-Worker-4860
0568
Exception in thread Pyro-Worker-48600568 :
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\site-packages\Pyro4\tpjobqueue.py", line 62, in run
self.job()
File "C:\Python27\lib\site-packages\Pyro4\socketserver\threadpoolserver.py", l
ine 32, in __call__
self.daemon.handleRequest(self.csock)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 769, in handleRequest
msgType, flags, seq, data = MessageFactory.getMessage(conn, MessageFactory.M
SG_INVOKE)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 574, in getMessage
headerdata = connection.recv(cls.HEADERSIZE)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 440, in recv
return receiveData(self.sock, size)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 187, in receive
Data
raise TimeoutError("receiving: timeout")
TimeoutError: receiving: timeout
[2013-05-11 10:22:46.726,Pyro4.core,DEBUG] Exception occurred while handling req
uest: TimeoutError('receiving: timeout',)
[2013-05-11 10:22:46.726,Pyro4.tpjobqueue,DEBUG] worker halted: Pyro-Worker-4757
2752
Exception in thread Pyro-Worker-47572752 :
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\site-packages\Pyro4\tpjobqueue.py", line 62, in run
self.job()
File "C:\Python27\lib\site-packages\Pyro4\socketserver\threadpoolserver.py", l
ine 32, in __call__
self.daemon.handleRequest(self.csock)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 769, in handleRequest
msgType, flags, seq, data = MessageFactory.getMessage(conn, MessageFactory.M
SG_INVOKE)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 574, in getMessage
headerdata = connection.recv(cls.HEADERSIZE)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 440, in recv
return receiveData(self.sock, size)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 187, in receive
Data
raise TimeoutError("receiving: timeout")
TimeoutError: receiving: timeout
[2013-05-11 10:22:46.865,Pyro4.core,DEBUG] Exception occurred while handling req
uest: TimeoutError('receiving: timeout',)
[2013-05-11 10:22:46.865,Pyro4.tpjobqueue,DEBUG] worker halted: Pyro-Worker-4635
0632
Exception in thread Pyro-Worker-46350632 :
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\site-packages\Pyro4\tpjobqueue.py", line 62, in run
self.job()
File "C:\Python27\lib\site-packages\Pyro4\socketserver\threadpoolserver.py", l
ine 32, in __call__
self.daemon.handleRequest(self.csock)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 769, in handleRequest
msgType, flags, seq, data = MessageFactory.getMessage(conn, MessageFactory.M
SG_INVOKE)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 574, in getMessage
headerdata = connection.recv(cls.HEADERSIZE)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 440, in recv
return receiveData(self.sock, size)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 187, in receive
Data
raise TimeoutError("receiving: timeout")
TimeoutError: receiving: timeout
[2013-05-11 10:22:47.022,Pyro4.core,DEBUG] Exception occurred while handling req
uest: TimeoutError('receiving: timeout',)
[2013-05-11 10:22:47.022,Pyro4.tpjobqueue,DEBUG] worker halted: Pyro-Worker-4632
5544
Exception in thread Pyro-Worker-46325544 :
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\site-packages\Pyro4\tpjobqueue.py", line 62, in run
self.job()
File "C:\Python27\lib\site-packages\Pyro4\socketserver\threadpoolserver.py", l
ine 32, in __call__
self.daemon.handleRequest(self.csock)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 769, in handleRequest
msgType, flags, seq, data = MessageFactory.getMessage(conn, MessageFactory.M
SG_INVOKE)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 574, in getMessage
headerdata = connection.recv(cls.HEADERSIZE)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 440, in recv
return receiveData(self.sock, size)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 187, in receive
Data
raise TimeoutError("receiving: timeout")
TimeoutError: receiving: timeout
[2013-05-11 10:22:47.038,Pyro4.core,DEBUG] Exception occurred while handling req
uest: TimeoutError('receiving: timeout',)
[2013-05-11 10:22:47.038,Pyro4.tpjobqueue,DEBUG] worker halted: Pyro-Worker-4854
9560
Exception in thread Pyro-Worker-48549560 :
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\site-packages\Pyro4\tpjobqueue.py", line 62, in run
self.job()
File "C:\Python27\lib\site-packages\Pyro4\socketserver\threadpoolserver.py", l
ine 32, in __call__
self.daemon.handleRequest(self.csock)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 769, in handleRequest
msgType, flags, seq, data = MessageFactory.getMessage(conn, MessageFactory.M
SG_INVOKE)
File "C:\Python27\lib\site-packages\Pyro4\core.py", line 574, in getMessage
headerdata = connection.recv(cls.HEADERSIZE)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 440, in recv
return receiveData(self.sock, size)
File "C:\Python27\lib\site-packages\Pyro4\socketutil.py", line 187, in receive
Data
raise TimeoutError("receiving: timeout")
TimeoutError: receiving: timeout
[[[[[[[[[[[[ the console holds here until the ^C is hit on the remote ]]]]]]]]]]
[2013-05-11 10:25:14.316,Pyro4.socketserver.threadpool,DEBUG] disconnected ('192
.168.25.44', 50483)