Re: how can I not have to spend 40 minutes _waiting_ in the morning for each of 40 updates?

Please read Arthur's comments, as they all apply, including the comment about 
the subject.

chaitu wrote, On 10/08/2008 09:57 PM:
> Hi Todd,Mark, everyone,
> 
> I've another tricky situation to beat. Got this huge 3rd party code
> repository (+2G of code) that I need on each of my 40 build machines
> (for different platforms of course). This repository is compiled
> everytime there's a code change and the resulting libraries are used
> for compiling my product's code.
> 

Is this 2GB of very small text files or 2GB of large (>2MB) binary files?
Are you pulling in the source code for this library, or the resulting compiled 
library, or both by using CVS?

> The problem is that I need to fire cvs update on each of the machines
> early in the morning before I start building my product. And just the
> damn update takes over 40 minutes. And since all machines start this
> update process at around the same time, there's some bandwidth crunch
> being recorded.
> 

Ever hear of cron?
I would have the Unix hosts issuing the update calls for me, probably having 
each machine give the previous ~30 minutes to run before starting it's own, 
early in the AM such that they should be finishing around the time I normally 
arrive.
I think even windows has a time of day command runner, but IIRC some user has 
to be logged into the box for it to work.

Also are you using either the -z option with cvs or the -C option of ssh 
(note: use one or the other, not both)?  I think -C on ssh is the better one 
as it compresses all of the cvs traffic not just the file transfers. Note: 
IIRC you would have to put the -C in ~/.ssh/config, so the easy test is -z.

when one machine is running update by itself, i.e., the other 39 are not 
updating while it is, how long does an update take?  how long does that update 
take if the sandbox is already up to date?
try running:
cd my/sandbox/ ; \
/usr/bin/time -p -o updatetime cvs -z update ; \
/usr/bin/time -ap -o updatetime cvs -z update ; \
cat updatetime

note: some people need reminded that my/sandbox/ needs replaced with the 
location of their sandbox in the file system.

> I was thinking of this solution around the problem and would
> appreciate if you can chip in with your criticism.
> 
> Update on only 1 machine, and from the update log, get all the U and P
> entries and create a tarball which I can send over LAN to the other 39
> machines and unzip them.

if you are going to trap the [UP] entries, I would then take that data and 
issue cvs commands on the other machines instead.  using cvs commands will 
keep the metadata correct and would stop the server from looking through it's 
tree to see if OTHER files had changed, i.e., you would run the risk with this 
or your method that if someone checked in a change between your first machine 
running and the last, you could get old and inconsistent data.



Do you have 40 different architectures or different versions of the OS for 
which you are building?
if it is more of a thing where you are building 5 different released versions 
of your code for windows XP i386 on 5 different machines but against one 
version of the library, this is one place where I would build the lib on one 
XP machine and SMB share the _resulting_LIB_ to the other 4. with the caveat 
that I don't expect the versions of any other libs on those machines to affect 
the build of the lib you are building.


-- 
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter