[gensim:1349] Re: LDA on large corpora: Pryo4, Atlas, and multiprocessing

Thank you for the report, Kyle.

On Sep 22, 10:26 pm, Kyle Jensen <[email protected]> wrote:
> 1) Pyro4, version 4.14 is broken for me on OSX Lion and Ubuntu 12.04.  In
> distributed mode, the LdaModel code cannot communicate with the dispatcher
> and it appears that lda_worker's do not property register with the name
> server.  Downgrading to 4.13 fixes this problem for me on both platforms.

I cannot replicate this problem, on Lion with Pyro 4.14 -- the
distributed LDA seems to work fine. Can you give more debug info?

Looking at release notes of Pyro 4.14, the only relevant changes as to
why registering should stop working:

* fixed some problems with MSG_WAITALL socket option on systems that
don?t properly
support it
* temporary workaround for threadpool scaling problem (lock-up): pool
is fixed at
THREADPOOL_MINTHREADS threads, until the thread pool has been
redesigned to get rid of
the issues.


> 2) Installing Atlas using apt-get on Ubuntu/Debian does not install a
> version that is optimized for your architecture.  You can follow the [very
> easy] instructions here to build
> one:http://mail.scipy.org/pipermail/scipy-user/2012-September/033055.html
>
> The tuning phase of building Atlas can take a long time, but it is worth it
> --- I saw the speed some matrix operations improve by 3 orders of magnitude.

That has been my experience as well -- optimizing system BLAS
libraries is well worth the time!


> Finally, two questions:
> A) I noticed that the LDA code spends an unusual amount of time
> in utils.grouper.  Anybody else see this?

That's the method that (lazily) reads in your corpus. Perhaps the data
is stored on a slow medium -- slow disk? Or the documents take time to
compute (on-the-fly corpus)?


> B) Any interest in a "distributed" version of LDA that uses multiprocessing
> (or concurrent.futures) rather than Pryo4.  It would only be useful for
> multi-core machines, but, in that case, would likely incur less overhead
> that using Pryo4.

Definitely! I think multi-core processing is actually the most common
scenario, so having that in gensim directly makes a lot of sense. Not
because of the overhead (Pyro4 is very efficient), but simply because
it'd be less complex -- one less moving piece, one less external
dependency.

Best,
Radim