Re: homogeneous environments

On Wed, Apr 29, 2009 at 1:39 PM, Robert W. Anderson <
[email protected]> wrote:

> Fergus Henderson wrote:
>
>  On Tue, Apr 28, 2009 at 4:28 PM, Robert W. Anderson <
>> [email protected] <mailto:[email protected]>> wrote:
>>
>>
>>    I have an environment where we have many nodes potentially available
>>    for compilation, and all of them see the same file spaces via NFS.
>>     We are seeing decent performance out of distcc 3.1 using pump mode,
>>    but from reading the docs there may be big performance gains left to
>>    wring out in this special(?) case.
>>
>>    If I understand correctly, distcc's pump mode finds a set of header
>>    files necessary to send along with the source file to enable
>>    compilation on a remote node.  In a homogeneous environment, it
>>    seems both steps here are unnecessary if the master and slave nodes
>>    are more or less indistinguishable in terms of compiler, sources,
>>    and headers.
>>
>>    I think we could really achieve some screaming compile times (over
>>    thousands of source files) if these steps could be bypassed with the
>>    user's explicit acknowledgement that he is making assumptions about
>>    the homogeneity of his build server machines.
>>
>>    How extensive would the modifications be to support such an
>>    optimization?  It was not clear to me after a few minutes of poking
>>    around in the source, and thought I'd seek an expert opinion first.
>>
>>
>> Typically NFS is a lot slower than local file access.
>> So it's not clear that this approach would actually improve overall
>> performance.
>>
>> Distcc can work faster than NFS, because it sends all of the source files
>> at once, requiring only one round-trip between the client and the distcc
>> server for each compilation.  With NFS, you need a round-trip between the
>> distcc server and the NFS server for each header file that is included
>> (directly or indirectly) from the source file being compiled.
>>
>> Of course with distcc, if your source files are on NFS, the client needs
>> to do the same round-trips to the NFS server to fetch the files, but this is
>> not as bad as having the distcc servers do that, because the distcc client
>> need only fetch each file once for the whole build, not once for each
>> compilation in which it is referenced, and after that the file will probably
>> be cached.  In addition, the client machine is more likely to have source
>> files cached from previous builds, since on the client machine you're
>> probably compiling the same sources that you compiled last time, whereas on
>> the distcc server machines they are serving lots of different users who may
>> be compiling very different programs.
>>
>> Another issue with this approach is that there may also be additional
>> security considerations.  Currently distcc servers normally run as user
>> "distcc", which may not have access to the user's NFS files, so this
>> approach would not work if the source files are not world-readable.  Of
>> course it would be possible to address this issue by having the distcc
>> server authenticate the user, and then access the user's files on NFS as
>> that user, but that would require additional authentication, which would
>> have a performance impact.  For example one way to do it would be to use
>> distcc's ssh mode, but that mode has a major performance impact. (The
>> recently posted patches for GSSAPI support have less performance impact, but
>> there is still a significant impact.)
>>
>> For the approach that you are considering, you may not need to use distcc
>> at all;
>> a simple script using ssh may be sufficient, though the overheads of ssh
>> may be prohibitive (ssh connection sharing may help with that, although that
>> has security concerns of its own).
>> If you do want to modify distcc, I'd guess that the modifications needed
>> would be moderate in scope.
>>
>
> Fergus,
>
> Thanks for the clear and detailed reply.  First I should note that I am
> already using ssh mode (via rsh) because I was unable to make TCP mode work.
>  I don't know if this is some kind of port blocking restriction on my
> machine or what:
>
> distcc[1663] (dcc_pump_sendfile) ERROR: sendfile failed: Connection reset
> by peer
> distcc[1663] (dcc_writex) ERROR: failed to write: Broken pipe
> distcc[1663] Warning: failed to distribute source.c to host16,cpp,lzo,
> running locally instead


It looks like distccd is dropping the connection?
Have a look in the distccd log file (e.g. /var/log/syslog or
/var/log/messages on host16); there may be more information there.

Otherwise, maybe there is some issue with the sys_sendfile() implementation
on your system.
You may be able to make it work by using dcc_pump_readwrite() rather than
sys_sendfile().
See the source for the dcc_pump_sendfile() function in src/sendfile.c for
more info.

Perhaps getting TCP mode running should be my first performance priority.
>
> I just tried what you suggested in your last paragraph, manually
> distributing compiles via rsh, and am finding that it is, as you suspected,
> a little slower than distcc using pump mode.  Rather than pursue that any
> further, based on your comments, I would like to see if I can get TCP pump
> mode working first.


Sounds like a good plan.
-- 
Fergus Henderson <[email protected]>

__ 
distcc mailing list            http://distcc.samba.org/
To unsubscribe or change options: 
https://lists.samba.org/mailman/listinfo/distcc