Re: [gensim:2397] Re: Not able to run Distributed LDA

Hello Ritesh (cross-posting to the Pyro list, where people may have more 
qualified experience with these issues),

I restarted the system, ran the server again but still the same message.
>>
>> I ran "python -c 'import Pyro4; print(Pyro4.config.SERIALIZERS_ACCEPTED)' 
>> in the Console. It prints "set(['pickle'])"; so I guess it is set correctly 
>> there.
>>
>> However when I run it in the Python Shell, it gives a different result 
>> - set(['json', 'marshal', 'serpent']).
>> In WIndows, however, CMD and Python Shell both prints the same 
>> "set(['pickle'])". 
>>
>> So I guess that is the problem. Is there a way to fix that in Python 
>> Shell?
>>
>
Which Python shell are you using, and how exactly do you launch it?


Ok I set the configuration values in Python Shell and then it works! But I 
> have to do it every time I restart shell. Is there some persistent method 
> of doing it in the shell?
>

Well, we could set PYRO_SERIALIZERS_ACCEPTED and PYRO_SERIALIZER directly 
from Python, instead of depending on ENV vars. But I was under the 
impression it's the same thing. Irmen?

 

Also I ran the distributed LDA on one Windows and one Linux system. There 
> are two workers and the dispatcher on my Ubuntu. And there is one worker on 
> Windows. Everything seems to go fine. The worker registers with the name 
> server running on Ubuntu. The dispatcher registers the worker on Windows. 
> But when I run it, the dispatcher assigns jobs only to the workers on 
> Ubuntu and not on Windows. And the worker on Windows tells me that "No 
> connection could be made because the target machine actively refused it". 
> It looked like a Firewall issue. So I disabled Firewall on Windows and as 
> far as I know the firewall on Ubuntu is set to 'allow' all connections. But 
> still it does not work. What could be possible reason?
>

Sounds like a firewall issue to me, but again, the Pyro folks will have 
more experience with such "heterogeneous" communication, what could go 
wrong and how to debug it.

Best,
Radim
 

>
> Regards,
> Ritesh
>
> On Monday, 6 January 2014 11:40:28 UTC+5:30, Ritesh Kumar wrote:
>>
>> Hi Radim,
>>
>>
 

>
>>
>> On Sunday, 5 January 2014 22:52:00 UTC+5:30, Radim Řehůřek wrote:
>>>
>>> I can't see how this could have anything to do with the OS (and 
>>> certainly not CLASSPATH). It must be something obvious we're missing.
>>>
>>> One more idea: was the Pyro nameserver run with these env vars as well? 
>>> This nameserver is a process that runs in the background.
>>> If not, terminate it and re-launch with PYRO_SERIALIZERS_ACCEPTED 
>>> and PYRO_SERIALIZER set.
>>>
>>> Another idea: run `python -c 'import Pyro4; 
>>> print(Pyro4.config.SERIALIZERS_ACCEPTED)'` to see whether pyro picks up the 
>>> env values correctly.
>>>
>>> -rr
>>>
>>>
>>> On Sunday, January 5, 2014 1:59:45 PM UTC+1, Ritesh Kumar wrote:
>>>>
>>>> I checked in each shell. The variables are set correctly. What is 
>>>> surprising is that the same version works well in Windows XP (I used the 
>>>> Enthought installation there and then installed gensim and Pyro4 from the 
>>>> tar.gz file - the same ones I used in Ubuntu). So it seems that the problem 
>>>> is specific to Ubuntu (and probably not related to Pyro). It does not seem 
>>>> to be very likely but could there be a possibility that other variables in 
>>>> bashrc (specially CLASSPATH) is causing it to fail? Or maybe should I use 
>>>> the Enthought distribution in Ubuntu also (I saw that they have one for 
>>>> Linux also)?
>>>>
>>>> On Sunday, 5 January 2014 17:42:52 UTC+5:30, Radim Řehůřek wrote:
>>>>>
>>>>> Hmm, that looks fine to me.
>>>>>
>>>>> Can you explicitly check, in each worker/dispatcher shell, that they 
>>>>> are indeed set? => `echo $PYRO_SERIALIZERS_ACCEPTED` and `echo 
>>>>> $PYRO_SERIALIZER`
>>>>>
>>>>> Other than that, I'm out of ideas... may be some new change to Pyro 
>>>>> again.
>>>>>
>>>>> Re. mixing OS's (windows + linux heterogeneous cluster): I never 
>>>>> tried, but it should be possible, yes. Please let me know it went if you do 
>>>>> try.
>>>>>
>>>>> Cheers,
>>>>> Radim
>>>>>
>>>>>
>>>>> On Sunday, January 5, 2014 12:55:44 PM UTC+1, Ritesh Kumar wrote:
>>>>>>
>>>>>> Hello Radim,
>>>>>>
>>>>>> I put it in bashrc. Here is what I put
>>>>>>
>>>>>> export PYRO_SERIALIZERS_ACCEPTED="pickle"
>>>>>> export PYRO_SERIALIZER="pickle"
>>>>>>
>>>>>> Regards,
>>>>>>  
>>>>>>
>>>>>> On 5 January 2014 17:10, Radim Řehůřek <[email protected]> wrote:
>>>>>>
>>>>>>> Hello Ritesh, how exactly did you set the env vars? Did you do it in 
>>>>>>> every shell? Or put it in your bashrc?
>>>>>>>
>>>>>>> -rr
>>>>>>>
>>>>>>>
>>>>>>> On Sunday, January 5, 2014 6:12:41 AM UTC+1, Ritesh Kumar wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am trying to run the distributed LDA. I have set up Pyro4 
>>>>>>>> correctly(I guess). I have followed the tutorials verbatim. I have set the 
>>>>>>>> two environment variables -PYRO_SERIALIZERS_ACCEPTED and PYRO_SERIALIZER. I 
>>>>>>>> started the Pyro name server and ran two instances of worker script and one 
>>>>>>>> instance 
>>>>>>>> of dispatcher script on the same machine (I have a system with 4 
>>>>>>>> cores and 4 GB of memory).
>>>>>>>>
>>>>>>>> Here is my code
>>>>>>>>
>>>>>>>>
>>>>>>>> from gensim import corpora, models, utils
>>>>>>>>
>>>>>>>> import logging
>>>>>>>>
>>>>>>>> logging.basicConfig(format='%(asctime)s : %(levelname)s : 
>>>>>>>> %(message)s', level=logging.INFO)
>>>>>>>>
>>>>>>>> file_read = "/home/ritesh/Documents/svmInput_trial"
>>>>>>>> file_save = "/home/ritesh/Documents/svmInput_trial_lda"
>>>>>>>>
>>>>>>>> corpus = corpora.SvmLightCorpus(file_read)
>>>>>>>> corpus1m = utils.RepeatCorpus(corpus, 1000000)
>>>>>>>>
>>>>>>>> model_lda = models.LdaModel(corpus1m, num_topics=50, distributed=
>>>>>>>> True)
>>>>>>>>
>>>>>>>> corpus_lda = model_lda[corpus]
>>>>>>>> corpora.SvmLightCorpus.serialize(file_save, corpus_lda)
>>>>>>>>
>>>>>>>>
>>>>>>>> Here is the log information in the Python Shell
>>>>>>>>
>>>>>>>> 2014-01-05 10:06:38,792 : INFO : loading corpus from /home/ritesh/
>>>>>>>> Documents/data/svmInput_trial
>>>>>>>> 2014-01-05 10:06:38,818 : WARNING : no word id mapping provided;initializing 
>>>>>>>> from corpus, assuming identity
>>>>>>>> 2014-01-05 10:06:43,481 : INFO : using symmetric alpha at 
>>>>>>>> 0.333333333333
>>>>>>>> 2014-01-05 10:06:43,488 : ERROR : failed to initialize distributed 
>>>>>>>> LDA (message used serializer that is not accepted: 1)
>>>>>>>> Traceback (most recent call last):
>>>>>>>>   File "/home/ritesh/Documents/trial.py", line 13, in <module>
>>>>>>>>     model_lda = models.LdaModel(corpus1m, num_topics=50,distributed
>>>>>>>> =True)
>>>>>>>>   File "/home/ritesh/Documents/gensim/gensim/models/ldamodel.py",line 
>>>>>>>> 294, in __init__
>>>>>>>>     raise RuntimeError("failed to initialize distributed LDA (%s)" 
>>>>>>>> % err)
>>>>>>>> RuntimeError: failed to initialize distributed LDA (message used 
>>>>>>>> serializer that is not accepted: 1)
>>>>>>>>
>>>>>>>>
>>>>>>>> And this is the log in the Console where Pyro serve is running
>>>>>>>>
>>>>>>>> Broadcast server running on 0.0.0.0:9091
>>>>>>>> NS running on 0.0.0.0:9090 (0.0.0.0)
>>>>>>>> URI = PYRO:[email protected]:9090
>>>>>>>> Exception in thread Pyro-Worker-140380205365968 :
>>>>>>>> Traceback (most recent call last):
>>>>>>>>   File "/usr/lib/python2.7/threading.py", line 551, in__bootstrap_inner
>>>>>>>>     self.run()
>>>>>>>>   File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4-
>>>>>>>> 4.22-py2.7.egg/Pyro4/tpjobqueue.py", line 62, in run
>>>>>>>>     self.job()
>>>>>>>>   File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4-
>>>>>>>> 4.22-py2.7.egg/Pyro4/socketserver/threadpoolserver.py", line 34, in__call__
>>>>>>>>     self.daemon.handleRequest(self.csock)
>>>>>>>>   File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4-
>>>>>>>> 4.22-py2.7.egg/Pyro4/core.py", line 710, in handleRequest
>>>>>>>>     raise errors.ProtocolError("message used serializer that is 
>>>>>>>> not accepted: %d" % msg.serializer_id)
>>>>>>>> ProtocolError: message used serializer that is not accepted: 1
>>>>>>>>
>>>>>>>>
>>>>>>>> Exception in thread Pyro-Worker-140380205223632 :
>>>>>>>> Traceback (most recent call last):
>>>>>>>>   File "/usr/lib/python2.7/threading.py", line 551, in__bootstrap_inner
>>>>>>>>     self.run()
>>>>>>>>   File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4-
>>>>>>>> 4.22-py2.7.egg/Pyro4/tpjobqueue.py", line 62, in run
>>>>>>>>     self.job()
>>>>>>>>   File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4-
>>>>>>>> 4.22-py2.7.egg/Pyro4/socketserver/threadpoolserver.py", line 34, in__call__
>>>>>>>>     self.daemon.handleRequest(self.csock)
>>>>>>>>   File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4-
>>>>>>>> 4.22-py2.7.egg/Pyro4/core.py", line 710, in handleRequest
>>>>>>>>     raise errors.ProtocolError("message used serializer that is 
>>>>>>>> not accepted: %d" % msg.serializer_id)
>>>>>>>> ProtocolError: message used serializer that is not accepted: 1
>>>>>>>>
>>>>>>>>
>>>>>>>> The log of the worker scripts and dispatcher scripts are as below
>>>>>>>>
>>>>>>>> INFO : running /home/ritesh/Documents/gensim/gensim/models/
>>>>>>>> lda_worker.py
>>>>>>>> 2014-01-05 09:40:38,690 : INFO : gensim.lda_worker.fac0d0 
>>>>>>>> registered with nameserver (URI 'PYRO:gensim.lda_worker.
>>>>>>>> [email protected]:55349')
>>>>>>>>
>>>>>>>>
>>>>>>>> INFO : running /home/ritesh/Documents/gensim/gensim/models/
>>>>>>>> lda_worker.py
>>>>>>>> 2014-01-05 09:40:52,546 : INFO : gensim.lda_worker.a88a4d 
>>>>>>>> registered with nameserver (URI 'PYRO:gensim.lda_worker.
>>>>>>>> [email protected]:34068')
>>>>>>>>
>>>>>>>>
>>>>>>>> INFO : running /home/ritesh/Documents/gensim/gensim/models/
>>>>>>>> lda_dispatcher.py
>>>>>>>> 2014-01-05 09:42:39,969 : INFO : gensim.lda_dispatcher registered 
>>>>>>>> with nameserver (URI 'PYRO:[email protected]:40266')
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> It looks like a problem of Pyro (and its new methods of 
>>>>>>>> serialization) to me. However I guess setting the two environment variables 
>>>>>>>> was a workaround for this issue.
>>>>>>>>
>>>>>>>> I also tried using 4.18 version of Pyro but that does not work 
>>>>>>>> either. The Python shell freezes and there is no response (I left it for 
>>>>>>>> around 5 hours but it didnt come back to life!).
>>>>>>>>
>>>>>>>> I have also attached the corpus file that I am using with this (had 
>>>>>>>> to add .txt extension to upload it here; I used the file without that 
>>>>>>>> extension).
>>>>>>>>
>>>>>>>> Am I missing something here? Is there some problem in my 
>>>>>>>> installations and setting things up? Or is it something else?
>>>>>>>>
>>>>>>>> And I was also wondering if it is possible to use systems with 
>>>>>>>> different Operating Systems in one cluster (say, I use 2 computers with 
>>>>>>>> Windows and 2 Ubuntu, each running the worker script and one of them the 
>>>>>>>> dispatcher script)?
>>>>>>>>
>>>>>>>>  -- 
>>>>>>> You received this message because you are subscribed to a topic in 
>>>>>>> the Google Groups "gensim" group.
>>>>>>> To unsubscribe from this topic, visit 
>>>>>>> https://groups.google.com/d/topic/gensim/gA9d3HaQA24/unsubscribe.
>>>>>>> To unsubscribe from this group and all its topics, send an email to 
>>>>>>> gensim+un...-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
>>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Ritesh Kumar
>>>>>> Asst. Prof., Dept. of Linguistics, Dr. B.R. Ambedkar University, Agra
>>>>>> Doctoral Candidate, Centre for Linguistics, Jawaharlal Nehru 
>>>>>> University, New Delhi
>>>>>>  
>>>>>

-- 
You received this message because you are subscribed to the Google Groups "gensim" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gensim+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
For more options, visit https://groups.google.com/groups/opt_out.