Re: [gensim:2397] Re: Not able to run Distributed LDA

Hello Radim

​Thanks a lot for all the help! Almost everything seems to be sorted out
now. See below for the updates!​


On 6 January 2014 18:14, Radim Řehůřek <me-yqFObnq8frArm/[email protected]> wrote:

> Hello Ritesh (cross-posting to the Pyro list, where people may have more
> qualified experience with these issues),
>
> I restarted the system, ran the server again but still the same message.
>>>
>>> I ran "python -c 'import Pyro4; print(Pyro4.config.SERIALIZERS_ACCEPTED)'
>>> in the Console. It prints "set(['pickle'])"; so I guess it is set correctly
>>> there.
>>>
>>> However when I run it in the Python Shell, it gives a different result
>>> - set(['json', 'marshal', 'serpent']).
>>> In WIndows, however, CMD and Python Shell both prints the same
>>> "set(['pickle'])".
>>>
>>> So I guess that is the problem. Is there a way to fix that in Python
>>> Shell?
>>>
>>
> Which Python shell are you using, and how exactly do you launch it?
>
I am using the default Idle shell. I was launching it from the Application
Launcher. However when I launch Idle​​ from the Console, the variables are
set in the shell also. Then I learnt that variables in ~/.bashrc is only
set for the Bash Shell; if we need to use some variables in an X session
then those need to be stored in ~/.xsessionrc (generally it is not created
by default). I did that and now everything works beautifully! :)

>
>
>
> Ok I set the configuration values in Python Shell and then it works! But I
>> have to do it every time I restart shell. Is there some persistent method
>> of doing it in the shell?
>>
>
> Well, we could set PYRO_SERIALIZERS_ACCEPTED and PYRO_SERIALIZER directly
> from Python, instead of depending on ENV vars. But I was under the
> impression it's the same thing. Irmen?
>
>
>
> Also I ran the distributed LDA on one Windows and one Linux system. There
>> are two workers and the dispatcher on my Ubuntu. And there is one worker on
>> Windows. Everything seems to go fine. The worker registers with the name
>> server running on Ubuntu. The dispatcher registers the worker on Windows.
>> But when I run it, the dispatcher assigns jobs only to the workers on
>> Ubuntu and not on Windows. And the worker on Windows tells me that "No
>> connection could be made because the target machine actively refused it".
>> It looked like a Firewall issue. So I disabled Firewall on Windows and as
>> far as I know the firewall on Ubuntu is set to 'allow' all connections. But
>> still it does not work. What could be possible reason?
>>
>
> Sounds like a firewall issue to me, but again, the Pyro folks will have
> more experience with such "heterogeneous" communication, what could go
> wrong and how to debug it.
>

There is some update on this front also. I ran the dispatcher​​ on Ubuntu
(and no worker) and 9 workers on 9 different computer (all the workers were
Windows system ;)). It worked perfectly without any problem. I shall try
different combinations and keep you updated on what worked and what not.

>
> Best,
> Radim
>
>
>>
>> Regards,
>> Ritesh
>>
>> On Monday, 6 January 2014 11:40:28 UTC+5:30, Ritesh Kumar wrote:
>>>
>>> Hi Radim,
>>>
>>>
>
>
>>
>>>
>>> On Sunday, 5 January 2014 22:52:00 UTC+5:30, Radim Řehůřek wrote:
>>>>
>>>> I can't see how this could have anything to do with the OS (and
>>>> certainly not CLASSPATH). It must be something obvious we're missing.
>>>>
>>>> One more idea: was the Pyro nameserver run with these env vars as well?
>>>> This nameserver is a process that runs in the background.
>>>> If not, terminate it and re-launch with PYRO_SERIALIZERS_ACCEPTED
>>>> and PYRO_SERIALIZER set.
>>>>
>>>> Another idea: run `python -c 'import Pyro4; print(Pyro4.config.SERIALIZERS_ACCEPTED)'`
>>>> to see whether pyro picks up the env values correctly.
>>>>
>>>> -rr
>>>>
>>>>
>>>> On Sunday, January 5, 2014 1:59:45 PM UTC+1, Ritesh Kumar wrote:
>>>>>
>>>>> I checked in each shell. The variables are set correctly. What is
>>>>> surprising is that the same version works well in Windows XP (I used the
>>>>> Enthought installation there and then installed gensim and Pyro4 from the
>>>>> tar.gz file - the same ones I used in Ubuntu). So it seems that the problem
>>>>> is specific to Ubuntu (and probably not related to Pyro). It does not seem
>>>>> to be very likely but could there be a possibility that other variables in
>>>>> bashrc (specially CLASSPATH) is causing it to fail? Or maybe should I use
>>>>> the Enthought distribution in Ubuntu also (I saw that they have one for
>>>>> Linux also)?
>>>>>
>>>>> On Sunday, 5 January 2014 17:42:52 UTC+5:30, Radim Řehůřek wrote:
>>>>>>
>>>>>> Hmm, that looks fine to me.
>>>>>>
>>>>>> Can you explicitly check, in each worker/dispatcher shell, that they
>>>>>> are indeed set? => `echo $PYRO_SERIALIZERS_ACCEPTED` and `echo
>>>>>> $PYRO_SERIALIZER`
>>>>>>
>>>>>> Other than that, I'm out of ideas... may be some new change to Pyro
>>>>>> again.
>>>>>>
>>>>>> Re. mixing OS's (windows + linux heterogeneous cluster): I never
>>>>>> tried, but it should be possible, yes. Please let me know it went if you do
>>>>>> try.
>>>>>>
>>>>>> Cheers,
>>>>>> Radim
>>>>>>
>>>>>>
>>>>>> On Sunday, January 5, 2014 12:55:44 PM UTC+1, Ritesh Kumar wrote:
>>>>>>>
>>>>>>> Hello Radim,
>>>>>>>
>>>>>>> I put it in bashrc. Here is what I put
>>>>>>>
>>>>>>> export PYRO_SERIALIZERS_ACCEPTED="pickle"
>>>>>>> export PYRO_SERIALIZER="pickle"
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>>
>>>>>>> On 5 January 2014 17:10, Radim Řehůřek <[email protected]>wrote:
>>>>>>>
>>>>>>>> Hello Ritesh, how exactly did you set the env vars? Did you do it
>>>>>>>> in every shell? Or put it in your bashrc?
>>>>>>>>
>>>>>>>> -rr
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sunday, January 5, 2014 6:12:41 AM UTC+1, Ritesh Kumar wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am trying to run the distributed LDA. I have set up Pyro4
>>>>>>>>> correctly(I guess). I have followed the tutorials verbatim. I have set the
>>>>>>>>> two environment variables -PYRO_SERIALIZERS_ACCEPTED and PYRO_SERIALIZER. I
>>>>>>>>> started the Pyro name server and ran two instances of worker script and one
>>>>>>>>> instance
>>>>>>>>> of dispatcher script on the same machine (I have a system with 4
>>>>>>>>> cores and 4 GB of memory).
>>>>>>>>>
>>>>>>>>> Here is my code
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> from gensim import corpora, models, utils
>>>>>>>>>
>>>>>>>>> import logging
>>>>>>>>>
>>>>>>>>> logging.basicConfig(format='%(asctime)s : %(levelname)s :
>>>>>>>>> %(message)s', level=logging.INFO)
>>>>>>>>>
>>>>>>>>> file_read = "/home/ritesh/Documents/svmInput_trial"
>>>>>>>>> file_save = "/home/ritesh/Documents/svmInput_trial_lda"
>>>>>>>>>
>>>>>>>>> corpus = corpora.SvmLightCorpus(file_read)
>>>>>>>>> corpus1m = utils.RepeatCorpus(corpus, 1000000)
>>>>>>>>>
>>>>>>>>> model_lda = models.LdaModel(corpus1m, num_topics=50, distributed=
>>>>>>>>> True)
>>>>>>>>>
>>>>>>>>> corpus_lda = model_lda[corpus]
>>>>>>>>> corpora.SvmLightCorpus.serialize(file_save, corpus_lda)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Here is the log information in the Python Shell
>>>>>>>>>
>>>>>>>>> 2014-01-05 10:06:38,792 : INFO : loading corpus from /home/ritesh/
>>>>>>>>> Documents/data/svmInput_trial
>>>>>>>>> 2014-01-05 10:06:38,818 : WARNING : no word id mapping provided;initializing
>>>>>>>>> from corpus, assuming identity
>>>>>>>>> 2014-01-05 10:06:43,481 : INFO : using symmetric alpha at
>>>>>>>>> 0.333333333333
>>>>>>>>> 2014-01-05 10:06:43,488 : ERROR : failed to initialize
>>>>>>>>> distributed LDA (message used serializer that is not accepted: 1)
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>   File "/home/ritesh/Documents/trial.py", line 13, in <module>
>>>>>>>>>     model_lda = models.LdaModel(corpus1m, num_topics=50,distributed
>>>>>>>>> =True)
>>>>>>>>>   File "/home/ritesh/Documents/gensim/gensim/models/ldamodel.py",line
>>>>>>>>> 294, in __init__
>>>>>>>>>     raise RuntimeError("failed to initialize distributed LDA (%s)"
>>>>>>>>> % err)
>>>>>>>>> RuntimeError: failed to initialize distributed LDA (message used
>>>>>>>>> serializer that is not accepted: 1)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> And this is the log in the Console where Pyro serve is running
>>>>>>>>>
>>>>>>>>> Broadcast server running on 0.0.0.0:9091
>>>>>>>>> NS running on 0.0.0.0:9090 (0.0.0.0)
>>>>>>>>> URI = PYRO:[email protected]:9090
>>>>>>>>> Exception in thread Pyro-Worker-140380205365968 :
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>   File "/usr/lib/python2.7/threading.py", line 551, in__bootstrap_inner
>>>>>>>>>     self.run()
>>>>>>>>>   File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4-
>>>>>>>>> 4.22-py2.7.egg/Pyro4/tpjobqueue.py", line 62, in run
>>>>>>>>>     self.job()
>>>>>>>>>   File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4-
>>>>>>>>> 4.22-py2.7.egg/Pyro4/socketserver/threadpoolserver.py", line 34,
>>>>>>>>> in __call__
>>>>>>>>>     self.daemon.handleRequest(self.csock)
>>>>>>>>>   File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4-
>>>>>>>>> 4.22-py2.7.egg/Pyro4/core.py", line 710, in handleRequest
>>>>>>>>>     raise errors.ProtocolError("message used serializer that is
>>>>>>>>> not accepted: %d" % msg.serializer_id)
>>>>>>>>> ProtocolError: message used serializer that is not accepted: 1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Exception in thread Pyro-Worker-140380205223632 :
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>   File "/usr/lib/python2.7/threading.py", line 551, in__bootstrap_inner
>>>>>>>>>     self.run()
>>>>>>>>>   File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4-
>>>>>>>>> 4.22-py2.7.egg/Pyro4/tpjobqueue.py", line 62, in run
>>>>>>>>>     self.job()
>>>>>>>>>   File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4-
>>>>>>>>> 4.22-py2.7.egg/Pyro4/socketserver/threadpoolserver.py", line 34,
>>>>>>>>> in __call__
>>>>>>>>>     self.daemon.handleRequest(self.csock)
>>>>>>>>>   File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4-
>>>>>>>>> 4.22-py2.7.egg/Pyro4/core.py", line 710, in handleRequest
>>>>>>>>>     raise errors.ProtocolError("message used serializer that is
>>>>>>>>> not accepted: %d" % msg.serializer_id)
>>>>>>>>> ProtocolError: message used serializer that is not accepted: 1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The log of the worker scripts and dispatcher scripts are as below
>>>>>>>>>
>>>>>>>>> INFO : running /home/ritesh/Documents/gensim/gensim/models/
>>>>>>>>> lda_worker.py
>>>>>>>>> 2014-01-05 09:40:38,690 : INFO : gensim.lda_worker.fac0d0
>>>>>>>>> registered with nameserver (URI 'PYRO:gensim.lda_worker.fac0d0
>>>>>>>>> @127.0.0.1:55349')
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> INFO : running /home/ritesh/Documents/gensim/gensim/models/
>>>>>>>>> lda_worker.py
>>>>>>>>> 2014-01-05 09:40:52,546 : INFO : gensim.lda_worker.a88a4d
>>>>>>>>> registered with nameserver (URI 'PYRO:gensim.lda_worker.a88a4d
>>>>>>>>> @127.0.0.1:34068')
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> INFO : running /home/ritesh/Documents/gensim/gensim/models/
>>>>>>>>> lda_dispatcher.py
>>>>>>>>> 2014-01-05 09:42:39,969 : INFO : gensim.lda_dispatcher registered
>>>>>>>>> with nameserver (URI 'PYRO:[email protected]:40266')
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It looks like a problem of Pyro (and its new methods of
>>>>>>>>> serialization) to me. However I guess setting the two environment variables
>>>>>>>>> was a workaround for this issue.
>>>>>>>>>
>>>>>>>>> I also tried using 4.18 version of Pyro but that does not work
>>>>>>>>> either. The Python shell freezes and there is no response (I left it for
>>>>>>>>> around 5 hours but it didnt come back to life!).
>>>>>>>>>
>>>>>>>>> I have also attached the corpus file that I am using with this
>>>>>>>>> (had to add .txt extension to upload it here; I used the file without that
>>>>>>>>> extension).
>>>>>>>>>
>>>>>>>>> Am I missing something here? Is there some problem in my
>>>>>>>>> installations and setting things up? Or is it something else?
>>>>>>>>>
>>>>>>>>> And I was also wondering if it is possible to use systems with
>>>>>>>>> different Operating Systems in one cluster (say, I use 2 computers with
>>>>>>>>> Windows and 2 Ubuntu, each running the worker script and one of them the
>>>>>>>>> dispatcher script)?
>>>>>>>>>
>>>>>>>>>  --
>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>> the Google Groups "gensim" group.
>>>>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/
>>>>>>>> topic/gensim/gA9d3HaQA24/unsubscribe.
>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>> gensim+un...-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ritesh Kumar
>>>>>>> Asst. Prof., Dept. of Linguistics, Dr. B.R. Ambedkar University, Agra
>>>>>>> Doctoral Candidate, Centre for Linguistics, Jawaharlal Nehru
>>>>>>> University, New Delhi
>>>>>>>
>>>>>>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "gensim" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/gensim/gA9d3HaQA24/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> gensim+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
> For more options, visit https://groups.google.com/groups/opt_out.
>



-- 
Ritesh Kumar
Asst. Prof., Dept. of Linguistics, Dr. B.R. Ambedkar University, Agra
Doctoral Candidate, Centre for Linguistics, Jawaharlal Nehru University,
New Delhi

-- 
You received this message because you are subscribed to the Google Groups "gensim" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gensim+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
For more options, visit https://groups.google.com/groups/opt_out.