Re: SCONS -j12, console output is disordered

Gary,

Any chance that repo could be relicensed, or dual licensed MIT?
That would allow us to possibly pull in some of your logic to core scons..

-Bill

On Thu, Jan 4, 2024 at 9:33 AM Gary Granger <[email protected]> wrote:

> For whatever it's worth, I had an example to provide, but then after
> looking at it I decided it needed some work.  So now it's a little more
> self-contained, and I think it's a reasonable example of a SPAWN override.
> We use it to save full test output into log files while filtering the
> console output to the more meaningful lines.  I expect it could be modified
> easily to cache the output from a single command and write the output to
> the console all at once.  Perhaps that would avoid your interleaved output
> problem.  Our implementation can be seen here:
>
> https://github.com/NCAR/eol_scons/blob/master/eol_scons/spawner.py
>
> Good luck,
> Gary
>
> On 12/25/23 13:36, Bill Deegan wrote:
>
> Please respond to the individual message, not the digest of messages.
>
> You can find information on SPAWN here:
> https://scons.org/doc/production/HTML/scons-man.html#cv-SPAWN
>
> I'm not aware of any easy to find examples, but you should be able to
> figure it out based on the manpage
> <https://scons.org/doc/production/HTML/scons-man.html#cv-SPAWN> contents:
> (copied here)
>
> A command interpreter function that will be called to execute command line
> strings. The function must accept five arguments:
>
> def spawn(shell, escape, cmd, args, env):
>
> shell is a string naming the shell program to use, escape is a function
> that can be called to escape shell special characters in the command line,
>  cmd is the path to the command to be executed, args holds the arguments
> to the command and env is a dictionary of environment variables defining
> the execution environment in which the command should be executed.
> Here's some related stack overflow questions:
> https://stackoverflow.com/questions/11139519/scons-save-redirect-gcc-textual-output-warnings
> Please let us know if you come up with a good solution.
>
> -Bill
>
> On Sun, Dec 24, 2023 at 11:51 PM Yu,Zhiwei (YFI,Shanghai,CN) via
> Scons-users <[email protected]> wrote:
>
>> Hi Guys，
>>
>>         Thanks Mats, Duane and Bill, Thanks for your replay.
>>
>>         I find something new. My compiler is IAR, and default, IAR will
>> output warning and error messages to stderr, normal message will be put
>> into stdout.
>>         So, if there are warnings in my code, normal msg and warning will
>> be intertwined in words.
>>         Luckily, Param --only_stdout can make IAR output stream to stdout.
>>
>>         But I have a new problem. The warnings msg is output by multi
>> lines. So they are intertwined in line. For example:
>>                 file0 warnings line 0
>>                 file0 warnings line 1
>>                 file1 normal msg line 0
>>                 file0 warnings line 2
>>                 file0 warnings line 3
>>         Actually, I want the lines is:
>>                 file0 warnings line 0
>>                 file0 warnings line 1
>>                 file0 warnings line 2
>>                 file0 warnings line 3
>>                 file1 normal msg line 0
>>
>>         I try to add mutex for print(), but no effection.
>>         Maybe, I should set env['SPAWN'], but I don't know how to use it.
>>
>>         I have two questions:
>>         Where can I find a sample about env['SPAWN']?
>>         How can I print msg when scons building complete?
>>
>>
>> Best Regards!
>>
>> Software Engineer
>> ------------------------------------
>> [email protected]
>>
>>
>>
>> -----邮件原件-----
>> 发件人: Scons-users <[email protected]> 代表
>> [email protected]
>> 发送时间: 2023年12月24日 3:29
>> 收件人: [email protected]
>> 主题: Scons-users Digest, Vol 147, Issue 2
>>
>> CAUTION: This email originated from outside of the organization. DO NOT
>> click links or open attachments unless you recognize the sender and know
>> the content is safe.
>>
>>
>> Send Scons-users mailing list submissions to
>>         [email protected]
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         https://pairlist4.pair.net/mailman/listinfo/scons-users
>> or, via email, send a message with subject or body 'help' to
>>         [email protected]
>>
>> You can reach the person managing the list at
>>         [email protected]
>>
>> When replying, please edit your Subject line so it is more specific than
>> "Re: Contents of Scons-users digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Re: SCONS -j12, console output is disordered (Duane Ellis)
>>    2. Re: SCONS -j12, console output is disordered (Mats Wichmann)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Sat, 23 Dec 2023 18:33:20 +0000
>> From: Duane Ellis <[email protected]>
>> To: SCons users mailing list <[email protected]>
>> Subject: Re: [Scons-users] SCONS -j12, console output is disordered
>> Message-ID:
>>         <
>> PH0PR05MB84789EC7A7CE775049F19AD3CA9BA@PH0PR05MB8478.namprd05.prod.outlook.com
>> >
>>
>> Content-Type: text/plain; charset="utf-8"
>>
>> SCONS could ?
>>                 Capture the output of each sub process in the standard
>> way.
>>                 Ie: use ?POPEN()? and COMMUNICATE with the sub process.
>>                    Thus the subprocess output goes to SCONS and SCONS
>> collects it in a buffer.
>>                     You might want to put a timeout on all sub processes,
>> ie: 5 minutes?
>>                     In case they output an error and want sometype of
>> input from the human to continue
>>
>>                 Then once the subprocess has completed, (Exit success or
>> Exit Error)
>>                The responsible runner thread would:
>>                       Lock the console (mutex) so other SCONS threads
>> cannot use it.
>>                       output stdout and stderr of the sub process
>>                       Unlock the console so other SCONS threads can use
>> it.
>>
>> From: Scons-users <[email protected]> on behalf of Mats
>> Wichmann <[email protected]>
>> Date: Saturday, December 23, 2023 at 8:51 AM
>> To: [email protected] <[email protected]>
>> Subject: Re: [Scons-users] SCONS -j12, console output is disordered On
>> 12/23/23 08:39, Yu,Zhiwei (YFI,Shanghai,CN) via Scons-users wrote:
>> > Hi Guys?
>> >
>> > ??????I add -j12 param, then the console output is disordered.
>> > ??????I wonder that the reason is python's print function is not
>> > thread-safe or atomic operation, right?
>> > ??????How could I fix the problem when I use SCONS.
>>
>> There are two sources of output:
>>
>> (1) scons-generated (by default, the command line, modifiable by setting
>> the various COMSTR variables, or setting up a PRINT_CMD_LINE_FUNC).
>> This text should not have any particular interleave problems.
>> (2) command-issued output.  This you can't do much about, because SCons
>> isn't really designed for that.  As jobs are classified as ready to run,
>> they're put on the queue; meanwhile when a job runner slot opens up, it's
>> assigned a job from the queue. Those are run as subprocesses, and run
>> independently; SCons doesn't collect their output or any other information
>> besides the exit status.  So you're just seeing the effect of having 12 job
>> runner slots all firing off jobs as they become available - if those
>> generate their own output that can well interleave.
>>
>> To not have interleaved command output, you'd need to convince SCons to
>> collect the output of the subprocesses rather than going to the default
>> stdout/stderr.  That is possible, but beyond the scope of a first answer.
>>
>> _______________________________________________
>> Scons-users mailing list
>> [email protected]
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> https://pairlist4.pair.net/pipermail/scons-users/attachments/20231223/4c7188b9/attachment-0001.htm
>> >
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Sat, 23 Dec 2023 12:29:06 -0700
>> From: Mats Wichmann <[email protected]>
>> To: [email protected]
>> Subject: Re: [Scons-users] SCONS -j12, console output is disordered
>> Message-ID: <[email protected]>
>> Content-Type: text/plain; charset=UTF-8; format=flowed
>>
>> On 12/23/23 11:33, Duane Ellis wrote:
>> > SCONS could ?
>> >
>> >  ??????????????? Capture the output of each sub process in the standard
>> way.
>> >
>> >  ??????????????? Ie: use ?POPEN()? and COMMUNICATE with the sub process.
>> >
>> >  ???????????????????Thus the subprocess output goes to SCONS and SCONS
>> > collects it in a buffer.
>>
>> Yes, it could.  It even does so now in one very specific situation -
>> while evaluating a Configure context.  However, even playing the game with
>> pushing in the value of the PSPAWN construction variable in place of the
>> SPAWN variable which is what the configure code does is not a full
>> solution, as scons would actually have to be prepared to gather and do
>> something with the output, which it currently isn't.
>>
>> >
>> >  ??????????????????? You might want to put a timeout on all sub
>> > processes, ie: 5 minutes?
>> >
>> >  ??????????????????? In case they output an error and want sometype of
>> > input from the human to continue
>> >
>> >  ??????????????? Then once the subprocess has completed, (Exit success
>> > or Exit Error)
>> >
>> >  ?????????????? The responsible runner thread would:
>> >
>> >  ????????????????????? Lock the console (mutex) so other SCONS threads
>> > cannot use it.
>> >
>> >  ????????????????????? output stdout and stderr of the sub process
>> >
>> >  ????????????????????? Unlock the console so other SCONS threads can
>> use it.
>>
>> This is the method the test runner uses - queues tests, has a pool of
>> runners to run them, and when one finished it takes a lock before writing
>> the collected output. But that's a different usecase... the test runner
>> *knows* it needs to collect output, while SCons itself really isn't
>> structured that way. Using logging could also do this, as it handles
>> locking between threads internally (there's an experimental version of the
>> test runner that does that).
>>
>> It *can* be done, but would require some effort.
>>
>>
>> ------------------------------
>>
>> Subject: Digest Footer
>>
>> _______________________________________________
>> Scons-users mailing list
>> [email protected]
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>>
>> ------------------------------
>>
>> End of Scons-users Digest, Vol 147, Issue 2
>> *******************************************
>> _______________________________________________
>> Scons-users mailing list
>> [email protected]
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>
> _______________________________________________
> Scons-users mailing [email protected]://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
> _______________________________________________
> Scons-users mailing list
> [email protected]
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>

_______________________________________________
Scons-users mailing list
[email protected]
https://pairlist4.pair.net/mailman/listinfo/scons-users