Re: openmosix and distcc

Hi,

Did you forget to send an email to the openMosix sourceforge lists. (heh - i
often do that too)
The original emails are attached for anyone else reading my reply.

I decided to do a few quick tests for distcc and openMosix using 4 dual P3
650Mhz servers to get some timings.


Compiling a fresh untouched linux-2.4.26 kernel from www.kernel.org located
in /usr/src

 ~ # cd /usr/src
 src # tar -xvjpf linux-2.4.26.tar.bz2
 src # ln -s /usr/src/linux-2.4.26 /usr/src/linux
 src # cd linux

linux # make menuconfig

[ EXIT and SAVE with NO modifications to the config ]

linux # make dep

Now ready to start some comparison timings, I stopped openMosix

linux # /etc/init.d/openmosix stop
 * Stopping openMosix ...
 * openMosix:
All remote processes were expelled and no further remote processes accep  [
ok ]


Then, I removed ccache and distcc from PATH

linux # echo $PATH
/usr/lib/ccache/bin:/usr/lib/distcc/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/i386-pc-linux-gnu/gcc-bin/3.3.6

linux # export
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/i386-pc-linux-gnu/gcc-bin/3.3.6"


This first test is now without distcc and openMosix using only one dual cpu
server.
( using -j3 [ jobs = num of cpus + 1 ] )

linux # time make -j3 bzImage

warning: kernel is too big for standalone boot from floppy
make[1]: Leaving directory `/usr/src/linux-2.4.26/arch/i386/boot'

real    4m58.864s
user    9m11.450s
sys     0m35.510s


The next test is using only distcc with openMosix disabled on every node.
( using -j9 [ jobs = num of cpus + 1 ] )

export PATH="/usr/lib/distcc/bin:${PATH}"

linux # make clean

 linux # time make -j9 bzImage

warning: kernel is too big for standalone boot from floppy
make[1]: Leaving directory `/usr/src/linux-2.4.26/arch/i386/boot'

real    1m53.739s
user    1m17.400s
sys     0m21.250s


The next test is with distcc and openMosix enabled
( using -j9 [ jobs = num of cpus + 1 ] )

linux # /etc/init.d/openmosix start
 * Initializing openMosix ...                                             [
ok ]

linux # make clean

linux # time make -j9 bzImage

warning: kernel is too big for standalone boot from floppy
make[1]: Leaving directory `/usr/src/linux-2.4.26/arch/i386/boot'

real    2m5.922s
user    1m19.070s
sys     0m31.730s


As you can see, distcc speeds up compiling because thats its job.
Using only openMosix without distcc usually breaks with make -j(number of
cluster cpus) as openMosix is at its best migrating cluster friendly
processes and compiling isnt all that cluster friendly.
Distcc also seems to migrate on openMosix also, although it looks as if
migration has slowed down the compilation a little.

If you are running a cluster built for a bit of everything, then I would
definately recommend that distcc is used with openMosix, I would also
suggest ccache (it would have been unfair results if i had included ccache
in the above tests) and I also use dsh for easy administration of my nodes.

BTW, I use the linux-2.4.26 with the openmosix patch as openMosix 2.6.* is
still under development. (although I do boot openMosix-2.6.15 every now and
again to have a poke around)

Martin




On 06/04/06, Andreas Fladischer <[email protected]> wrote:
>
> oh i forgot something; do you use the 2.4 or the 2.6 kernel??
>
> Deadpan110 wrote:
>
> > Hi,
> >
> > I use openMosix and distcc (makes sence when using a distro like
> > gentoo that requires a lot of compiling)
> >
> > As you are probably aware that while not all processes will migrate
> > using openMosix (especially while compiling some things)
> > distcc seems to help matters a lot.
> >
> > I would recommend that if you have not used distcc before that you
> > make sure all your nodes are the same and have the same versions of
> > gcc, kernel headers etc, etc as this is the main cause of failed
> > distcc builds.
> >
> > Afterwards, it is possible to configure distcc for cross compiling.
> >
> > I have noticed that distcc can also migrate sometimes (so when a
> > compile is sent to another node with distcc, it too can migrate again
> > if openmosix spots that its not heavily IO bound)
> >
> > I would say distcc is a definite must and speeds up compilations on my
> > cluster.
> >
> > I would also recommend that you also use ccache to speed up compiling
> > and also install dsh to make administering your cluster easier.
> >
> > Martin
> >
> > On 06/04/06, *Andreas Fladischer* <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> >     hi!
> >
> >     have anybody experience with openmosix and distcc?
> >
> >     greetz
> >
> >
> >     -------------------------------------------------------
> >     This SF.Net email is sponsored by xPML, a groundbreaking scripting
> >     language
> >     that extends applications into web and mobile media. Attend the
> >     live webcast
> >     and join the prime developer group breaking into this new coding
> >     territory!
> >
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> >     <
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642>
> >     _______________________________________________
> >     openMosix-general mailing list
> >     openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> >     <mailto:openMosix-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org>
> >     https://lists.sourceforge.net/lists/listinfo/openmosix-general
> >     <https://lists.sourceforge.net/lists/listinfo/openmosix-general>
> >
> >
>
>