Re: visualize build parallelism, duration of build steps?

On 10/18/21 16:45, Gabe Black wrote:
> Hi folks, sorry if this is a really obvious question, but is there a 
> command line flag or tool or something to visualize how parallel a SCons 
> build is, if there are any bottlenecks, if there are abnormally long 
> running build steps, etc?

The taskmastertrace is generally the way to extract this information. 
It's a bit like trying to drink from a firehose, though, so a tool to 
process it would likely help. Don't know if there are any of those 
floating around.


> I've had a suspicion for a few years that there might be a task scheduling bug where at certain points a super task seems to come along and stop any other tasks from being scheduled until it is done. 
There's been some discussion of this over time.  It appears the way 
scheduling is done may be prone to getting blocked by a long-running 
jobs.  One tuning of the algorithm was proposed in a PR which hasn't 
received much action - opinions about whether this had really pinpointed 
an underlying issue were not unanimous.

https://github.com/SCons/scons/pull/3386

There's also been talk about maybe not having all the members of the 
pool of job threads be handled identically, so there's a place you could 
put a long-running job to get it started as soon as possible, and not 
wait behind a bunch of shorter ones, which could then compete for the 
other threads and maybe get closer to finishing at the same times. 
Making decisions based on length of jobs would require some sort of 
profiling capability so the scheduler has the information available.

On the whole I think it's been posited that there are performance issues 
when the build gets very big, and not sure it's well understood why. 
The taskmaster is a pretty complex piece of code and it was heavily 
tuned "in the old days" (~15 years ago).  Be nice to have better ways to 
characterize what happens on modern hardware configurations with 
potentially larger build projects than when that work was done.