Re: SCons working with Ninja

Don,

But if you're willing to give it a shot and report back your issues we can
put them into the queue..

-Bill

On Thu, Apr 15, 2021 at 1:52 PM Bill Deegan <[email protected]>
wrote:

> Don,
>
> If processing your sconscript files is taking a significant amount of
> time, you're almost always doing something wrong.
> Have you run with --debug=time? What output do you get?
>
> Also using heavyweight function actions will slow down the build, There's
> no real threading in python due to GIL.
> Another reason why it's best to move such out of process if possible.
> I think while this would work with scons/ninja, it would be vastly slower
> than moving that logic into scripts..
>
> This is IMHO one of the most common mistakes users make.
> SCons is python great, let me make a big blob of python logic in process
> to do this thing.
> That would not be something you could do with most other build systems,
> and for most such, it's not something  you should do.
>
> Honestly it's more like alpha tester..
> You're probably not a great candidate for the initial test.
>
> I'd prefer a more or less vanilla c/c++ build.
>
>
> On Thu, Apr 15, 2021 at 9:56 AM Don Baldwin <[email protected]> wrote:
>
>> Hi Bill,
>>
>>
>>
>> I think it would be great to be a beta tester of this, but will need to
>> check with my leads to make sure we can spare the resourced.  I’ll get back
>> to you on that.  Also, it will obviously only be if we can actually use the
>> feature, which we may not be able to if it doesn’t support function actions.
>>
>>
>>
>> I’m not sure of the details of the post-link processing, but they are
>> mostly doing file munging with Python scripts.  We have spoken with the
>> teams to convert their tools to a command line form, but they’re fairly
>> reluctant to do so.  Having the ability to call Python functions as actions
>> is a great feature, and no one wants to give that up.
>>
>>
>>
>> In our build system, parsing through our SConscript files takes a
>> significant amount of time, and that’s part of what we were hoping to avoid
>> with the Ninja builds.  I think we’d need a way to save the environments
>> for each action, if we’re to have to call back into SCons.  Is that the
>> approach you guys are taking here?
>>
>>
>>
>> Thanks,
>>
>> Don
>>
>>
>>
>>
>>
>> *From:* Scons-users <[email protected]> * On Behalf Of *Bill
>> Deegan
>> *Sent:* Wednesday, April 14, 2021 8:14 PM
>> *To:* SCons users mailing list <[email protected]>
>> *Subject:* [EXT] Re: [Scons-users] SCons working with Ninja
>>
>>
>>
>> The work in progress is generously contributed and upstreaming sponsored
>> by MongoDB, so the reference to someone's packaging of such would probably
>> be an earlier version of that code.
>>
>>
>>
>> As I understand it many of the build steps as possible are exported to
>> the ninja build file, the remainder will call back into SCons to build.
>>
>>
>>
>> Don - I'd be curious if you could share what post link steps have to be
>> implemented in python? Are they generating command lines? or doing file
>> munging in python?
>>
>>
>>
>> The plan is to add it to the next (4.1.1 or 4.2.0.. I need to review the
>> included changes since 4.1.0), it will NOT be enabled by default.
>>
>> As the code has only been used in only one production environment, I
>> expect there'll be a lot of bumps in the road before it's ready for general
>> usage.
>>
>>
>>
>> It's possible the implementation may be significantly changed before it's
>> enabled by default.
>>
>>
>>
>> If you're willing to be a PATIENT "beta" user and even contribute fixes..
>> , then we'd be glad to have the help.
>>
>> If not, then you're better waiting until it's had more usage to try it
>> out.
>>
>>
>>
>> Further, if you already have an implementation (based on the mongodb
>> work, or not) and can share any lessoned and code learned, that'd be great.
>>
>>
>>
>> We don't really have the resources to fully support this at the moment,
>> thus the "experimental feature" approach was chosen as a way to expose it
>> to motivated parties who might help us move it forward to production
>> quality.
>>
>>
>>
>> -Bill
>>
>> SCons Project Co-Manager.
>>
>>
>>
>>
>>
>> On Wed, Apr 14, 2021 at 1:56 PM Don Baldwin <[email protected]>
>> wrote:
>>
>> Thanks Gary.  However, in our case, we have several post-link steps that
>> are carried out as Python functions, so they are an unavoidable part of the
>> build process.
>>
>> -Don
>>
>> -----Original Message-----
>> From: Scons-users <[email protected]> On Behalf Of Gary
>> Granger
>> Sent: Wednesday, April 14, 2021 1:22 PM
>> To: SCons users mailing list <[email protected]>
>> Subject: [EXT] Re: [Scons-users] SCons working with Ninja
>>
>> Hello Don,
>>
>> For whatever it's worth, we have a tool to generate ninja files from
>> SCons, but I did not try to solve the problem of running python functions
>> from ninja.  Instead, the tool generates a ninja build file from the nodes,
>> and then it lets scons build just the nodes which do not use command line
>> actions.  Most of our python function builders generate source files and
>> version header files, so they are the leaves of the dependency graph and it
>> doesn't matter if ninja cannot build them. Building the few python actions
>> establishes a 'build baseline', then ninja builds the rest.  Obviously not
>> complete, since someone has to remember to rebuild the ninja file if the
>> python dependencies change. Although I suppose the tool could be modified
>> to list all the python targets with a rule to re-run scons to build just
>> those.  And maybe this does not help you, if most of your actions are not
>> command lines.
>>
>> That said, we don't use this much because in practice it's easier to just
>> stick with scons for everything.
>>
>> https://github.com/NCAR/eol_scons/blob/master/eol_scons/tools/ninja.py
>>
>> The ninja support mentioned by Mats is surprising news, I'll be
>> interested to see how that works.
>>
>> Gary
>>
>> On 4/14/21 1:42 PM, Don Baldwin wrote:
>> > Thanks Mats.  Will this be implemented in SCons 3.x as well as 4.x?  We
>> are planning on moving to 4.x fairly soon, but we aren't there yet, and
>> will probably not move our older projects to 4.x.  Also, do you have an ETA
>> of when this will be available?
>> >
>> > Also, what about existing tools that claim to support generating Ninja
>> scripts from SCons builds, such as:
>> >
>> >     * https://github.com/RedBeard0531/mongo_module_ninja
>> >
>> > Does anyone in this group know if that supports function actions?
>> >
>> > Lastly, we've been able to implement our own home-grown tools for doing
>> a build "play-back" similar to what Ninja does, but we've only gotten it to
>> work for command line actions.  I think the main hurdle for function
>> actions is reproducing the SCons environment.  We've tried pickling it
>> using both the native pickle library and 'dill', but have encountered
>> infinite loop errors and errors trying to pickle class objects.  Has anyone
>> else attempted this and gotten past these errors?  This is using SCons
>> 3.1.2.
>> >
>> > Thanks,
>> > Don
>>
>> _______________________________________________
>> Scons-users mailing list
>> [email protected]
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>> _______________________________________________
>> Scons-users mailing list
>> [email protected]
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>> _______________________________________________
>> Scons-users mailing list
>> [email protected]
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
>

_______________________________________________
Scons-users mailing list
[email protected]
https://pairlist4.pair.net/mailman/listinfo/scons-users