Re: SCONS -j12, console output is disordered

For whatever it's worth, I had an example to provide, but then after 
looking at it I decided it needed some work.  So now it's a little more 
self-contained, and I think it's a reasonable example of a SPAWN 
override.  We use it to save full test output into log files while 
filtering the console output to the more meaningful lines.  I expect it 
could be modified easily to cache the output from a single command and 
write the output to the console all at once.  Perhaps that would avoid 
your interleaved output problem.  Our implementation can be seen here:

https://github.com/NCAR/eol_scons/blob/master/eol_scons/spawner.py

Good luck,
Gary

On 12/25/23 13:36, Bill Deegan wrote:
> Please respond to the individual message, not the digest of messages.
>
> You can find information on SPAWN here:
> https://scons.org/doc/production/HTML/scons-man.html#cv-SPAWN
>
> I'm not aware of any easy to find examples, but you should be able to 
> figure it out based on the manpage 
> <https://scons.org/doc/production/HTML/scons-man.html#cv-SPAWN> 
> contents: (copied here)
>
>     A command interpreter function that will be called to execute
>     command line strings. The function must accept five arguments:
>
>     def spawn(shell, escape, cmd, args, env):
>
>     |shell|is a string naming the shell program to use,|escape|is a
>     function that can be called to escape shell special characters in
>     the command line,|cmd|is the path to the command to be
>     executed,|args|holds the arguments to the command and|env|is a
>     dictionary of environment variables defining the execution
>     environment in which the command should be executed.
>
> Here's some related stack overflow questions:
>     https://stackoverflow.com/questions/11139519/scons-save-redirect-gcc-textual-output-warnings
>
> Please let us know if you come up with a good solution.
>
> -Bill
>
> On Sun, Dec 24, 2023 at 11:51 PM Yu,Zhiwei (YFI,Shanghai,CN) via 
> Scons-users <[email protected]> wrote:
>
>     Hi Guys，
>
>             Thanks Mats, Duane and Bill, Thanks for your replay.
>
>             I find something new. My compiler is IAR, and default, IAR
>     will output warning and error messages to stderr, normal message
>     will be put into stdout.
>             So, if there are warnings in my code, normal msg and
>     warning will be intertwined in words.
>             Luckily, Param --only_stdout can make IAR output stream to
>     stdout.
>
>             But I have a new problem. The warnings msg is output by
>     multi lines. So they are intertwined in line. For example:
>                     file0 warnings line 0
>                     file0 warnings line 1
>                     file1 normal msg line 0
>                     file0 warnings line 2
>                     file0 warnings line 3
>             Actually, I want the lines is:
>                     file0 warnings line 0
>                     file0 warnings line 1
>                     file0 warnings line 2
>                     file0 warnings line 3
>                     file1 normal msg line 0
>
>             I try to add mutex for print(), but no effection.
>             Maybe, I should set env['SPAWN'], but I don't know how to
>     use it.
>
>             I have two questions:
>             Where can I find a sample about env['SPAWN']?
>             How can I print msg when scons building complete?
>
>
>     Best Regards!
>
>     Software Engineer
>     ------------------------------------
>     [email protected]
>
>
>
>     -----邮件原件-----
>     发件人: Scons-users <[email protected]> 代表
>     [email protected]
>     发送时间: 2023年12月24日 3:29
>     收件人: [email protected]
>     主题: Scons-users Digest, Vol 147, Issue 2
>
>     CAUTION: This email originated from outside of the organization.
>     DO NOT click links or open attachments unless you recognize the
>     sender and know the content is safe.
>
>
>     Send Scons-users mailing list submissions to
>     [email protected]
>
>     To subscribe or unsubscribe via the World Wide Web, visit
>     https://pairlist4.pair.net/mailman/listinfo/scons-users
>     or, via email, send a message with subject or body 'help' to
>     [email protected]
>
>     You can reach the person managing the list at
>     [email protected]
>
>     When replying, please edit your Subject line so it is more
>     specific than "Re: Contents of Scons-users digest..."
>
>
>     Today's Topics:
>
>        1. Re: SCONS -j12, console output is disordered (Duane Ellis)
>        2. Re: SCONS -j12, console output is disordered (Mats Wichmann)
>
>
>     ----------------------------------------------------------------------
>
>     Message: 1
>     Date: Sat, 23 Dec 2023 18:33:20 +0000
>     From: Duane Ellis <[email protected]>
>     To: SCons users mailing list <[email protected]>
>     Subject: Re: [Scons-users] SCONS -j12, console output is disordered
>     Message-ID:
>            
>     <PH0PR05MB84789EC7A7CE775049F19AD3CA9BA@PH0PR05MB8478.namprd05.prod.outlook.com>
>
>     Content-Type: text/plain; charset="utf-8"
>
>     SCONS could ?
>                     Capture the output of each sub process in the
>     standard way.
>                     Ie: use ?POPEN()? and COMMUNICATE with the sub
>     process.
>                        Thus the subprocess output goes to SCONS and
>     SCONS collects it in a buffer.
>                         You might want to put a timeout on all sub
>     processes, ie: 5 minutes?
>                         In case they output an error and want sometype
>     of input from the human to continue
>
>                     Then once the subprocess has completed, (Exit
>     success or Exit Error)
>                    The responsible runner thread would:
>                           Lock the console (mutex) so other SCONS
>     threads cannot use it.
>                           output stdout and stderr of the sub process
>                           Unlock the console so other SCONS threads
>     can use it.
>
>     From: Scons-users <[email protected]> on behalf of
>     Mats Wichmann <[email protected]>
>     Date: Saturday, December 23, 2023 at 8:51 AM
>     To: [email protected] <[email protected]>
>     Subject: Re: [Scons-users] SCONS -j12, console output is
>     disordered On 12/23/23 08:39, Yu,Zhiwei (YFI,Shanghai,CN) via
>     Scons-users wrote:
>     > Hi Guys?
>     >
>     > ??????I add -j12 param, then the console output is disordered.
>     > ??????I wonder that the reason is python's print function is not
>     > thread-safe or atomic operation, right?
>     > ??????How could I fix the problem when I use SCONS.
>
>     There are two sources of output:
>
>     (1) scons-generated (by default, the command line, modifiable by
>     setting the various COMSTR variables, or setting up a
>     PRINT_CMD_LINE_FUNC).
>     This text should not have any particular interleave problems.
>     (2) command-issued output.  This you can't do much about, because
>     SCons isn't really designed for that.  As jobs are classified as
>     ready to run, they're put on the queue; meanwhile when a job
>     runner slot opens up, it's assigned a job from the queue. Those
>     are run as subprocesses, and run independently; SCons doesn't
>     collect their output or any other information besides the exit
>     status.  So you're just seeing the effect of having 12 job runner
>     slots all firing off jobs as they become available - if those
>     generate their own output that can well interleave.
>
>     To not have interleaved command output, you'd need to convince
>     SCons to collect the output of the subprocesses rather than going
>     to the default stdout/stderr.  That is possible, but beyond the
>     scope of a first answer.
>
>     _______________________________________________
>     Scons-users mailing list
>     [email protected]
>     https://pairlist4.pair.net/mailman/listinfo/scons-users
>     -------------- next part --------------
>     An HTML attachment was scrubbed...
>     URL:
>     <https://pairlist4.pair.net/pipermail/scons-users/attachments/20231223/4c7188b9/attachment-0001.htm>
>
>     ------------------------------
>
>     Message: 2
>     Date: Sat, 23 Dec 2023 12:29:06 -0700
>     From: Mats Wichmann <[email protected]>
>     To: [email protected]
>     Subject: Re: [Scons-users] SCONS -j12, console output is disordered
>     Message-ID: <[email protected]>
>     Content-Type: text/plain; charset=UTF-8; format=flowed
>
>     On 12/23/23 11:33, Duane Ellis wrote:
>     > SCONS could ?
>     >
>     >  ??????????????? Capture the output of each sub process in the
>     standard way.
>     >
>     >  ??????????????? Ie: use ?POPEN()? and COMMUNICATE with the sub
>     process.
>     >
>     >  ???????????????????Thus the subprocess output goes to SCONS and
>     SCONS
>     > collects it in a buffer.
>
>     Yes, it could.  It even does so now in one very specific situation
>     - while evaluating a Configure context. However, even playing the
>     game with pushing in the value of the PSPAWN construction variable
>     in place of the SPAWN variable which is what the configure code
>     does is not a full solution, as scons would actually have to be
>     prepared to gather and do something with the output, which it
>     currently isn't.
>
>     >
>     >  ??????????????????? You might want to put a timeout on all sub
>     > processes, ie: 5 minutes?
>     >
>     >  ??????????????????? In case they output an error and want
>     sometype of
>     > input from the human to continue
>     >
>     >  ??????????????? Then once the subprocess has completed, (Exit
>     success
>     > or Exit Error)
>     >
>     >  ?????????????? The responsible runner thread would:
>     >
>     >  ????????????????????? Lock the console (mutex) so other SCONS
>     threads
>     > cannot use it.
>     >
>     >  ????????????????????? output stdout and stderr of the sub process
>     >
>     >  ????????????????????? Unlock the console so other SCONS threads
>     can use it.
>
>     This is the method the test runner uses - queues tests, has a pool
>     of runners to run them, and when one finished it takes a lock
>     before writing the collected output. But that's a different
>     usecase... the test runner *knows* it needs to collect output,
>     while SCons itself really isn't structured that way. Using logging
>     could also do this, as it handles locking between threads
>     internally (there's an experimental version of the test runner
>     that does that).
>
>     It *can* be done, but would require some effort.
>
>
>     ------------------------------
>
>     Subject: Digest Footer
>
>     _______________________________________________
>     Scons-users mailing list
>     [email protected]
>     https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>     ------------------------------
>
>     End of Scons-users Digest, Vol 147, Issue 2
>     *******************************************
>     _______________________________________________
>     Scons-users mailing list
>     [email protected]
>     https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
> _______________________________________________
> Scons-users mailing list
> [email protected]
> https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
[email protected]
https://pairlist4.pair.net/mailman/listinfo/scons-users