Re: [gensim:2405] Re: Not able to run Distributed LDA
Ritesh Kumar <[email protected]> Sun, 12 Jan 2014 04:07:12 -0800 (PST)
| Newsgroups | gmane.comp.ai.gensim,gmane.comp.python.pyro |
|---|---|
| Message-ID | <[email protected]> |
Hello Radim, I tested on all kinds of combination. It worked fine everywhere except in one case - the system in the cluster which have Pyro server running on it is not able to communicate with other systems. So if I run the dispatcher and Pyro server on the same system then it does not work at all since the system with dispatcher refuses to accept any connection; however if I run name server on one of the systems running the worker script then the dispatcher is not able to communicate with that system; but communicates well with other workers and so everything works fine. Is this the expected behaviour or is there some workaround this? The second issue is with regard to running the distributed LDA. While LDA is running, if one of the registered workers shut down in the middle or become unresponsive (and other workers are still running) then the program terminates with the following error Pyro4.errors.ConnectionClosedError: receiving: not enough data Is this expected and is there some way to tackle it without terminating the complete program? Regards, On Wednesday, January 8, 2014 12:17:12 AM UTC+5:30, Radim Řehůřek wrote: > > Ok, good to hear that, Ritesh. > > Irmen also commented on the Pyro list with good advice (tricky getting > these replies synchronized!), so check out > http://sourceforge.net/mailarchive/message.php?msg_id=31815482 > > Seems there is really no problem with pyro nor gensim; just takes some > effort to work around your local settings/configs/idiosyncrasies. > > Good luck, > Radim > > > On Monday, January 6, 2014 4:00:10 PM UTC+1, Ritesh Kumar wrote: >> >> Hello Radim >> >> Thanks a lot for all the help! Almost everything seems to be sorted out >> now. See below for the updates! >> >> >> On 6 January 2014 18:14, Radim Řehůřek <[email protected]> wrote: >> >>> Hello Ritesh (cross-posting to the Pyro list, where people may have more >>> qualified experience with these issues), >>> >>> I restarted the system, ran the server again but still the same message. >>>>> >>>>> I ran "python -c 'import Pyro4; print(Pyro4.config.SERIALIZERS_ACCEPTED)' >>>>> in the Console. It prints "set(['pickle'])"; so I guess it is set correctly >>>>> there. >>>>> >>>>> However when I run it in the Python Shell, it gives a different result >>>>> - set(['json', 'marshal', 'serpent']). >>>>> In WIndows, however, CMD and Python Shell both prints the same >>>>> "set(['pickle'])". >>>>> >>>>> So I guess that is the problem. Is there a way to fix that in Python >>>>> Shell? >>>>> >>>> >>> Which Python shell are you using, and how exactly do you launch it? >>> >> I am using the default Idle shell. I was launching it from the >> Application Launcher. However when I launch Idle from the Console, the >> variables are set in the shell also. Then I learnt that variables in >> ~/.bashrc is only set for the Bash Shell; if we need to use some variables >> in an X session then those need to be stored in ~/.xsessionrc (generally it >> is not created by default). I did that and now everything works >> beautifully! :) >> >>> >>> >>> >>> Ok I set the configuration values in Python Shell and then it works! But >>>> I have to do it every time I restart shell. Is there some persistent method >>>> of doing it in the shell? >>>> >>> >>> Well, we could set PYRO_SERIALIZERS_ACCEPTED and PYRO_SERIALIZER >>> directly from Python, instead of depending on ENV vars. But I was under the >>> impression it's the same thing. Irmen? >>> >>> >>> >>> Also I ran the distributed LDA on one Windows and one Linux system. >>>> There are two workers and the dispatcher on my Ubuntu. And there is one >>>> worker on Windows. Everything seems to go fine. The worker registers with >>>> the name server running on Ubuntu. The dispatcher registers the worker on >>>> Windows. But when I run it, the dispatcher assigns jobs only to the workers >>>> on Ubuntu and not on Windows. And the worker on Windows tells me that "No >>>> connection could be made because the target machine actively refused it". >>>> It looked like a Firewall issue. So I disabled Firewall on Windows and as >>>> far as I know the firewall on Ubuntu is set to 'allow' all connections. But >>>> still it does not work. What could be possible reason? >>>> >>> >>> Sounds like a firewall issue to me, but again, the Pyro folks will have >>> more experience with such "heterogeneous" communication, what could go >>> wrong and how to debug it. >>> >> >> There is some update on this front also. I ran the dispatcher on Ubuntu >> (and no worker) and 9 workers on 9 different computer (all the workers were >> Windows system ;)). It worked perfectly without any problem. I shall try >> different combinations and keep you updated on what worked and what not. >> >>> >>> Best, >>> Radim >>> >>> >>>> >>>> Regards, >>>> Ritesh >>>> >>>> On Monday, 6 January 2014 11:40:28 UTC+5:30, Ritesh Kumar wrote: >>>>> >>>>> Hi Radim, >>>>> >>>>> >>> >>> >>>> >>>>> >>>>> On Sunday, 5 January 2014 22:52:00 UTC+5:30, Radim Řehůřek wrote: >>>>>> >>>>>> I can't see how this could have anything to do with the OS (and >>>>>> certainly not CLASSPATH). It must be something obvious we're missing. >>>>>> >>>>>> One more idea: was the Pyro nameserver run with these env vars as >>>>>> well? This nameserver is a process that runs in the background. >>>>>> If not, terminate it and re-launch with PYRO_SERIALIZERS_ACCEPTED >>>>>> and PYRO_SERIALIZER set. >>>>>> >>>>>> Another idea: run `python -c 'import Pyro4; print(Pyro4.config.SERIALIZERS_ACCEPTED)'` >>>>>> to see whether pyro picks up the env values correctly. >>>>>> >>>>>> -rr >>>>>> >>>>>> >>>>>> On Sunday, January 5, 2014 1:59:45 PM UTC+1, Ritesh Kumar wrote: >>>>>>> >>>>>>> I checked in each shell. The variables are set correctly. What is >>>>>>> surprising is that the same version works well in Windows XP (I used the >>>>>>> Enthought installation there and then installed gensim and Pyro4 from the >>>>>>> tar.gz file - the same ones I used in Ubuntu). So it seems that the problem >>>>>>> is specific to Ubuntu (and probably not related to Pyro). It does not seem >>>>>>> to be very likely but could there be a possibility that other variables in >>>>>>> bashrc (specially CLASSPATH) is causing it to fail? Or maybe should I use >>>>>>> the Enthought distribution in Ubuntu also (I saw that they have one for >>>>>>> Linux also)? >>>>>>> >>>>>>> On Sunday, 5 January 2014 17:42:52 UTC+5:30, Radim Řehůřek wrote: >>>>>>>> >>>>>>>> Hmm, that looks fine to me. >>>>>>>> >>>>>>>> Can you explicitly check, in each worker/dispatcher shell, that >>>>>>>> they are indeed set? => `echo $PYRO_SERIALIZERS_ACCEPTED` and >>>>>>>> `echo $PYRO_SERIALIZER` >>>>>>>> >>>>>>>> Other than that, I'm out of ideas... may be some new change to Pyro >>>>>>>> again. >>>>>>>> >>>>>>>> Re. mixing OS's (windows + linux heterogeneous cluster): I never >>>>>>>> tried, but it should be possible, yes. Please let me know it went if you do >>>>>>>> try. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Radim >>>>>>>> >>>>>>>> >>>>>>>> On Sunday, January 5, 2014 12:55:44 PM UTC+1, Ritesh Kumar wrote: >>>>>>>>> >>>>>>>>> Hello Radim, >>>>>>>>> >>>>>>>>> I put it in bashrc. Here is what I put >>>>>>>>> >>>>>>>>> export PYRO_SERIALIZERS_ACCEPTED="pickle" >>>>>>>>> export PYRO_SERIALIZER="pickle" >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> >>>>>>>>> On 5 January 2014 17:10, Radim Řehůřek <[email protected]>wrote: >>>>>>>>> >>>>>>>>>> Hello Ritesh, how exactly did you set the env vars? Did you do it >>>>>>>>>> in every shell? Or put it in your bashrc? >>>>>>>>>> >>>>>>>>>> -rr >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sunday, January 5, 2014 6:12:41 AM UTC+1, Ritesh Kumar wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I am trying to run the distributed LDA. I have set up Pyro4 >>>>>>>>>>> correctly(I guess). I have followed the tutorials verbatim. I have set the >>>>>>>>>>> two environment variables -PYRO_SERIALIZERS_ACCEPTED and PYRO_SERIALIZER. I >>>>>>>>>>> started the Pyro name server and ran two instances of worker script and one >>>>>>>>>>> instance >>>>>>>>>>> of dispatcher script on the same machine (I have a system with 4 >>>>>>>>>>> cores and 4 GB of memory). >>>>>>>>>>> >>>>>>>>>>> Here is my code >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> from gensim import corpora, models, utils >>>>>>>>>>> >>>>>>>>>>> import logging >>>>>>>>>>> >>>>>>>>>>> logging.basicConfig(format='%(asctime)s : %(levelname)s : >>>>>>>>>>> %(message)s', level=logging.INFO) >>>>>>>>>>> >>>>>>>>>>> file_read = "/home/ritesh/Documents/svmInput_trial" >>>>>>>>>>> file_save = "/home/ritesh/Documents/svmInput_trial_lda" >>>>>>>>>>> >>>>>>>>>>> corpus = corpora.SvmLightCorpus(file_read) >>>>>>>>>>> corpus1m = utils.RepeatCorpus(corpus, 1000000) >>>>>>>>>>> >>>>>>>>>>> model_lda = models.LdaModel(corpus1m, num_topics=50, distributed >>>>>>>>>>> =True) >>>>>>>>>>> >>>>>>>>>>> corpus_lda = model_lda[corpus] >>>>>>>>>>> corpora.SvmLightCorpus.serialize(file_save, corpus_lda) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Here is the log information in the Python Shell >>>>>>>>>>> >>>>>>>>>>> 2014-01-05 10:06:38,792 : INFO : loading corpus from /home/ >>>>>>>>>>> ritesh/Documents/data/svmInput_trial >>>>>>>>>>> 2014-01-05 10:06:38,818 : WARNING : no word id mapping provided;initializing >>>>>>>>>>> from corpus, assuming identity >>>>>>>>>>> 2014-01-05 10:06:43,481 : INFO : using symmetric alpha at >>>>>>>>>>> 0.333333333333 >>>>>>>>>>> 2014-01-05 10:06:43,488 : ERROR : failed to initialize >>>>>>>>>>> distributed LDA (message used serializer that is not accepted: 1 >>>>>>>>>>> ) >>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>> File "/home/ritesh/Documents/trial.py", line 13, in <module> >>>>>>>>>>> model_lda = models.LdaModel(corpus1m, num_topics=50,distributed >>>>>>>>>>> =True) >>>>>>>>>>> File "/home/ritesh/Documents/gensim/gensim/models/ldamodel.py" >>>>>>>>>>> , line 294, in __init__ >>>>>>>>>>> raise RuntimeError("failed to initialize distributed LDA >>>>>>>>>>> (%s)" % err) >>>>>>>>>>> RuntimeError: failed to initialize distributed LDA (message >>>>>>>>>>> used serializer that is not accepted: 1) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> And this is the log in the Console where Pyro serve is running >>>>>>>>>>> >>>>>>>>>>> Broadcast server running on 0.0.0.0:9091 >>>>>>>>>>> NS running on 0.0.0.0:9090 (0.0.0.0) >>>>>>>>>>> URI = PYRO:[email protected]:9090 >>>>>>>>>>> Exception in thread Pyro-Worker-140380205365968 : >>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>> File "/usr/lib/python2.7/threading.py", line 551, in__bootstrap_inner >>>>>>>>>>> self.run() >>>>>>>>>>> File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4- >>>>>>>>>>> 4.22-py2.7.egg/Pyro4/tpjobqueue.py", line 62, in run >>>>>>>>>>> self.job() >>>>>>>>>>> File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4- >>>>>>>>>>> 4.22-py2.7.egg/Pyro4/socketserver/threadpoolserver.py", line 34, >>>>>>>>>>> in __call__ >>>>>>>>>>> self.daemon.handleRequest(self.csock) >>>>>>>>>>> File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4- >>>>>>>>>>> 4.22-py2.7.egg/Pyro4/core.py", line 710, in handleRequest >>>>>>>>>>> raise errors.ProtocolError("message used serializer that is >>>>>>>>>>> not accepted: %d" % msg.serializer_id) >>>>>>>>>>> ProtocolError: message used serializer that is not accepted: 1 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Exception in thread Pyro-Worker-140380205223632 : >>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>> File "/usr/lib/python2.7/threading.py", line 551, in__bootstrap_inner >>>>>>>>>>> self.run() >>>>>>>>>>> File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4- >>>>>>>>>>> 4.22-py2.7.egg/Pyro4/tpjobqueue.py", line 62, in run >>>>>>>>>>> self.job() >>>>>>>>>>> File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4- >>>>>>>>>>> 4.22-py2.7.egg/Pyro4/socketserver/threadpoolserver.py", line 34, >>>>>>>>>>> in __call__ >>>>>>>>>>> self.daemon.handleRequest(self.csock) >>>>>>>>>>> File "/home/ritesh/.local/lib/python2.7/site-packages/Pyro4- >>>>>>>>>>> 4.22-py2.7.egg/Pyro4/core.py", line 710, in handleRequest >>>>>>>>>>> raise errors.ProtocolError("message used serializer that is >>>>>>>>>>> not accepted: %d" % msg.serializer_id) >>>>>>>>>>> ProtocolError: message used serializer that is not accepted: 1 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The log of the worker scripts and dispatcher scripts are as below >>>>>>>>>>> >>>>>>>>>>> INFO : running /home/ritesh/Documents/gensim/gensim/models/ >>>>>>>>>>> lda_worker.py >>>>>>>>>>> 2014-01-05 09:40:38,690 : INFO : gensim.lda_worker.fac0d0 >>>>>>>>>>> registered with nameserver (URI 'PYRO:gensim.lda_worker.fac0d0 >>>>>>>>>>> @127.0.0.1:55349') >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> INFO : running /home/ritesh/Documents/gensim/gensim/models/ >>>>>>>>>>> lda_worker.py >>>>>>>>>>> 2014-01-05 09:40:52,546 : INFO : gensim.lda_worker.a88a4d >>>>>>>>>>> registered with nameserver (URI 'PYRO:gensim.lda_worker.a88a4d >>>>>>>>>>> @127.0.0.1:34068') >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> INFO : running /home/ritesh/Documents/gensim/gensim/models/ >>>>>>>>>>> lda_dispatcher.py >>>>>>>>>>> 2014-01-05 09:42:39,969 : INFO : gensim.lda_dispatcher >>>>>>>>>>> registered with nameserver (URI 'PYRO:gensim.lda_dispatcher@12 >>>>>>>>>>> 7.0.0.1:40266') >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> It looks like a problem of Pyro (and its new methods of >>>>>>>>>>> serialization) to me. However I guess setting the two environment variables >>>>>>>>>>> was a workaround for this issue. >>>>>>>>>>> >>>>>>>>>>> I also tried using 4.18 version of Pyro but that does not work >>>>>>>>>>> either. The Python shell freezes and there is no response (I left it for >>>>>>>>>>> around 5 hours but it didnt come back to life!). >>>>>>>>>>> >>>>>>>>>>> I have also attached the corpus file that I am using with this >>>>>>>>>>> (had to add .txt extension to upload it here; I used the file without that >>>>>>>>>>> extension). >>>>>>>>>>> >>>>>>>>>>> Am I missing something here? Is there some problem in my >>>>>>>>>>> installations and setting things up? Or is it something else? >>>>>>>>>>> >>>>>>>>>>> And I was also wondering if it is possible to use systems with >>>>>>>>>>> different Operating Systems in one cluster (say, I use 2 computers with >>>>>>>>>>> Windows and 2 Ubuntu, each running the worker script and one of them the >>>>>>>>>>> dispatcher script)? >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>> in the Google Groups "gensim" group. >>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>> https://groups.google.com/d/topic/gensim/gA9d3HaQA24/unsubscribe. >>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>> to gensim+un...-/JYPxA39Uh5TLH3MbocFF+G/[email protected] >>>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Ritesh Kumar >>>>>>>>> Asst. Prof., Dept. of Linguistics, Dr. B.R. Ambedkar University, >>>>>>>>> Agra >>>>>>>>> Doctoral Candidate, Centre for Linguistics, Jawaharlal Nehru >>>>>>>>> University, New Delhi >>>>>>>>> >>>>>>>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "gensim" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/gensim/gA9d3HaQA24/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> gensim+un...-/JYPxA39Uh5TLH3MbocFF+G/[email protected] >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> >> >> -- >> Ritesh Kumar >> Asst. Prof., Dept. of Linguistics, Dr. B.R. Ambedkar University, Agra >> Doctoral Candidate, Centre for Linguistics, Jawaharlal Nehru University, >> New Delhi >> > -- You received this message because you are subscribed to the Google Groups "gensim" group. To unsubscribe from this group and stop receiving emails from it, send an email to gensim+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/[email protected] For more options, visit https://groups.google.com/groups/opt_out.