Re: Scons none-deterministic behavior for incremental build
Hua Yanghao <[email protected]>
| Newsgroups | gmane.comp.programming.tools.scons.user |
|---|---|
| Message-ID | <CAPTzt1CCK74U+vRq9069tPA0Z-Wff3X034=Xv9pSrkq8VdV3KA@mail.gmail.com> |
It is impossible to run on python2.7 within a reasonable time as a huge list of python3-only syntax/modules needs to be adapted ... On Wed, Dec 5, 2018 at 12:15 AM Bill Deegan <[email protected]> wrote: > > Does this happen if you use python 2.7? > > On Tue, Dec 4, 2018 at 6:00 PM Hua Yanghao <[email protected]> wrote: >> >> Apparently the MD5 signature is taking *turns* some how. >> >> You can see below the old md5 and new md5 checksum is somehow >> alternating between two fixed value ... >> How is scons calculating the md5 value? is it only depends on the file content? >> The md5sum of the "arch_init.c" is: >> md5sum firmware/arch/armv7/arch_init.c >> db624fbb365812d7ba58c87d49f44b6f firmware/arch/armv7/arch_init.c >> >> First build: >> e494890ea1b6e36acf275f3698a3f3f9 >> c706a016770a7c85ac42f16f79456797 >> Returning True >> in self.changed ... 2 >> return not r: False >> ['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o'] >> <SCons.Script.Main.BuildTask object at 0x7fdc934c7fd0> >> ______CC build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o >> >> Second build: >> c706a016770a7c85ac42f16f79456797 >> e494890ea1b6e36acf275f3698a3f3f9 >> Returning True >> in self.changed ... 2 >> return not r: False >> ['build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o'] >> <SCons.Script.Main.BuildTask object at 0x7f3fca5e4fd0> >> ______CC build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o >> On Tue, Dec 4, 2018 at 11:54 PM Hua Yanghao <[email protected]> wrote: >> > >> > That is generated by sconstruct, not from shell. Any potential issue >> > here? (It was working for a very long time before and only breaks >> > recently when I added some new features). >> > >> > In the meantime I have isolated the issue to be in Node.Node.changed() >> > call, where scons old & new signature for the C files seems not >> > matching (print("###### 3.1") is executed in error case): >> > 1463 if self.has_builder(): >> > |3 >> > 1464 print("####### 3") >> > |3 >> > 1465 import SCons.Util >> > |3 >> > 1466 newsig = SCons.Util.MD5signature(contents) >> > |3 >> > 1467 if bi.bactsig != newsig: >> > |3 >> > 1468 if t: Trace(': bactsig %s != newsig %s' % >> > (bi.bactsig, newsig)) |3 >> > 1469 print("####### 3.1") >> > |3 >> > 1470 result = True >> > On Tue, Dec 4, 2018 at 11:24 PM Bill Deegan <[email protected]> wrote: >> > > >> > > Generated by logic in your sconstruct/sconscripts? or do you shell out and run a script/program to generate it. >> > > >> > > >> > > On Tue, Dec 4, 2018 at 4:37 PM Hua Yanghao <[email protected]> wrote: >> > >> >> > >> Hi Bill, >> > >> config.h is generated file, not checked in, it is generated in the >> > >> build folder only. >> > >> Well at least I see scons is very odering-sensitive. For example, if >> > >> the order of SConscript processing changes (in the case of a set()), >> > >> even if there is no dependency actually changed, scons thinks the >> > >> target need to be rebuilt. >> > >> >> > >> any progress on these similar issues already? :) >> > >> >> > >> Best Regards, >> > >> Yanghao >> > >> On Tue, Dec 4, 2018 at 10:15 PM Bill Deegan <[email protected]> wrote: >> > >> > >> > >> > Is "build/configs/qemu_arm_vexpress/arm3/config/config.h" a generated file? Or checked in? >> > >> > >> > >> > It's very unlikely to be set ordering. set's should only be used for membership checks not iteration in the SCons codebase. >> > >> > >> > >> > We're actively debugging similar issue. >> > >> > >> > >> > -Bill >> > >> > >> > >> > On Tue, Dec 4, 2018 at 3:23 PM Hua Yanghao <[email protected]> wrote: >> > >> >> >> > >> >> I does that a lot ... but a quick try to make sure no >> > >> >> LIBS/LIBPATH/CPPPATH etc. is touched after Object() is being called, >> > >> >> still shows the same 50~70% rebuild of the entire project. >> > >> >> I am not using any parallel build, and somehow scons is having a very >> > >> >> none-deterministic behavior. >> > >> >> >> > >> >> Actually I fixed partially the issue (e.g. in simpler build the >> > >> >> incremental builds now works fine), due to python set() is having a >> > >> >> random order for every run. I am thinking maybe scons is somehow also >> > >> >> depends on the ordering of a set()? And when it happens to be the same >> > >> >> order then no rebuild? >> > >> >> On Tue, Dec 4, 2018 at 9:00 PM Oscar Anson <[email protected]> wrote: >> > >> >> > >> > >> >> > Hi, >> > >> >> > >> > >> >> > Not sure if this could be related, but a similar issue happened to me where I get a relink of a shared library without changing anything in the project. It happened that I was setting the environment LIBS and LIBPATH after calling the builder of the library. >> > >> >> > >> > >> >> > El mar., 4 dic. 2018 20:55, Hua Yanghao <[email protected]> escribió: >> > >> >> >> >> > >> >> >> Dear Scons Experts, >> > >> >> >> I recently hit on a very strange problem where scons incremental >> > >> >> >> builds sometimes OK, sometimes the entire project is re-built. >> > >> >> >> >> > >> >> >> scons: 3.0.1 >> > >> >> >> python: 3.6.7 >> > >> >> >> >> > >> >> >> --debug=explain gives below example trace: >> > >> >> >> scons: rebuilding >> > >> >> >> `build/configs/qemu_arm_vexpress/arm3/config/config.h' because >> > >> >> >> AlwaysBuild() is specified >> > >> >> >> __CONFIG build/configs/qemu_arm_vexpress/arm3/config/config.h >> > >> >> >> scons: rebuilding >> > >> >> >> `build/configs/qemu_arm_vexpress/arm3/firmware/arch/armv7/arch_init.o' >> > >> >> >> because: >> > >> >> >> `firmware/arch/armv7/arch_init.c' is no longer a dependency >> > >> >> >> `firmware/arch/armv7/arch_init.c' is a new dependency >> > >> >> >> >> > >> >> >> It seems some how the C file itself is removed and re-added as the >> > >> >> >> Object() dependency, and causes rebuild. However this is *NOT* >> > >> >> >> *always* the case. Sometime the rebuild doesn't happen. >> > >> >> >> >> > >> >> >> I failed to create a minimal example that can show this issue so far, >> > >> >> >> but in the production environment it is about 50% full-rebuild and 50% >> > >> >> >> incremental re-build. >> > >> >> >> >> > >> >> >> Any thoughts or hints? >> > >> >> >> >> > >> >> >> Thanks, >> > >> >> >> Yanghao >> > >> >> >> _______________________________________________ >> > >> >> >> Scons-users mailing list >> > >> >> >> [email protected] >> > >> >> >> https://pairlist4.pair.net/mailman/listinfo/scons-users >> > >> >> > >> > >> >> > _______________________________________________ >> > >> >> > Scons-users mailing list >> > >> >> > [email protected] >> > >> >> > https://pairlist4.pair.net/mailman/listinfo/scons-users >> > >> >> _______________________________________________ >> > >> >> Scons-users mailing list >> > >> >> [email protected] >> > >> >> https://pairlist4.pair.net/mailman/listinfo/scons-users >> > >> > >> > >> > _______________________________________________ >> > >> > Scons-users mailing list >> > >> > [email protected] >> > >> > https://pairlist4.pair.net/mailman/listinfo/scons-users >> > >> _______________________________________________ >> > >> Scons-users mailing list >> > >> [email protected] >> > >> https://pairlist4.pair.net/mailman/listinfo/scons-users >> > > >> > > _______________________________________________ >> > > Scons-users mailing list >> > > [email protected] >> > > https://pairlist4.pair.net/mailman/listinfo/scons-users >> _______________________________________________ >> Scons-users mailing list >> [email protected] >> https://pairlist4.pair.net/mailman/listinfo/scons-users > > _______________________________________________ > Scons-users mailing list > [email protected] > https://pairlist4.pair.net/mailman/listinfo/scons-users _______________________________________________ Scons-users mailing list [email protected] https://pairlist4.pair.net/mailman/listinfo/scons-users