Re: scons-4.3.0 bug reading scons-3.1.2 generated .sconsign.dblite

"Deploy" was a poor choice of words.  We are a large organization with a
pipeline of builds and quality.  Our fastest turnaround/lowest quality
guarantees are quick development builds for preliminary testing of targeted
features (non-production use, obviously).  We are long-time users of scons
and rely on the build correctness that scons can usually provide.  I
haven't seen another build tool that tries as hard as scons to encode even
the commands and the build environment as dependencies.  That's why I was
surprised that a DB compatibility check wasn't in the code.

I think an alternative solution of writing the scons version, or some other
compatibility version information, as a header in the .sconsign.dblite
would be much better than renaming files.  If the header is not recognized
as being compatible, nuke the file and start over.  Is a reproducer really
needed?  Even conceptually, an incompatible DB format between versions is a
known weakness.

Thanks,

Jay


On Tue, Mar 22, 2022 at 10:44 PM Bill Deegan <[email protected]>
wrote:

> Jay,
>
> You're deploying non-full builds (incremental) to the field? (built in the
> same sandbox as older versions?)
> I would not consider that a good practice.
>
> If we change the default sconsign naming we need to go through a
> deprecation cycle (we need to notify on next release, and then roll a major
> release to push the change), it will break lots of tests, and also affect
> users expecting the sconsign file naming. And even if we did this if you
> used a python 3.5+ with scons 3.0+ and did the same switch back and forth
> that you're doing, the issue would not be resolved.
>
> That said you are free to enable switching back and forth in your code by
> naming the sconsign file to include the python version.
> This would solve your issue.
>
> Could you make a small reproducer? Perhaps there's another way to avoid
> the bad outcome.
>
> -Bill
>
>
> On Mon, Mar 21, 2022 at 11:56 AM Jay West <[email protected]> wrote:
>
>> I'm not asking for the sconsign file to be made compatible between
>> versions.  What I'm asking for is to ensure that scons refuses to create
>> bad builds.  A full rebuild is acceptable.  Error'ing out is acceptable.
>> Silently creating a bad build is not acceptable.
>>
>> A developer may check out old code that must be run with scons2
>> (scons-3.1.2 with python2).  The developer may also check out the most
>> recent code that must run with scons3 (scons-4.3.0 with python3).  The
>> following happens:
>>
>>
>>    - Compile new code with scons3, followed by compiling old code with
>>    scons2.  scons2 errors out with a pickling error.  This is perfectly fine.
>>    The developer removes the .sconsign.dblite file and all is good.
>>    - Compile old code with scons2, followed by compiling new code with
>>    scon3.  The build is successful, but the build is corrupt.  *This is
>>    not acceptable.*
>>
>>
>> We have unknowingly deployed bad builds to the field, and this is causing
>> all sorts of havoc.  I have added the below to our SConstruct file for one
>> project.  Can something like this be added to scons?
>>
>> Thanks,
>>
>> Jay
>>
>>     def fix_scons_db():
>>         '''
>>         Fix scons2 vs scons3 DB backward compatibility problem
>>
>>         The dependency database from scons-3.1.2 cannot be read by
>> scons-4.3.0.  Instead of generating an error, scons-4.3.0 will blissfully
>> ignore the error and produce an incorrect build.
>>         In order to workaround this, we'll use use an scons
>> version-specific DB file.  Credit for the idea to:
>>
>> https://github.com/Autodesk/arnold-usd/pull/116/commits/a404d5f3400437beb4bad561ef1811538f7af022
>>         This solution also future proofs against further scons upgrades.
>>         '''
>>         scons_db_file_name = '.sconsign.{}'.format(SCons.__version__)
>>         SConsignFile(scons_db_file_name)
>>         scons_db_file_name += '.dblite' # scons adds this suffix
>>         scons_old_db_files = glob.glob('.sconsign.*')
>>         if scons_db_file_name in scons_old_db_files:
>>             scons_old_db_files.remove(scons_db_file_name)
>>         #print('scons DB file name:', scons_db_file_name, 'removing',
>> scons_old_db_files)
>>         if len(scons_old_db_files):
>>             subprocess.run(['rm -f .sconsign.*'], check=True, shell=True)
>>
>>
>> On Sun, Mar 20, 2022 at 9:36 PM Bill Deegan <[email protected]>
>> wrote:
>>
>>> There's no (reasonable) way to make the sconsign files be compatible
>>> between py3.5 and p2.7.
>>> The only way to not have invalid (as in crashes scons per your original
>>> email) is to name the sconsign files per python version.
>>> (Which I mentioned in my previous email, and is what the commit you
>>> pointed to).
>>>
>>> Of course as you discovered (most likely) you will rebuild items which
>>> were not out of date.
>>>
>>> The recommended way is to just validate the new scons & python
>>> combination on your build and then make the switch.
>>>
>>> -Bill
>>>
>>>
>>> On Sun, Mar 20, 2022 at 7:23 PM Jay West <[email protected]> wrote:
>>>
>>>> Actually, the workaround isn't sufficient.  We also need to delete all
>>>> the other versions of .sconsign.dblite* as they will be out of date.
>>>>
>>>> Jay
>>>>
>>>>
>>>> On Sun, Mar 20, 2022 at 6:58 PM Jay West <[email protected]> wrote:
>>>>
>>>>> If scons did a full rebuild, it would not be a problem.  The issue is
>>>>> that scons does NOT do a full rebuild, thinking that targets were up to
>>>>> date when they were really not.
>>>>>
>>>>> I see a workaround that someone else took:
>>>>> https://github.com/Autodesk/arnold-usd/pull/116/commits/a404d5f3400437beb4bad561ef1811538f7af022.
>>>>> I will be putting this into our SContruct file.  Can this be made the
>>>>> default behavior?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jay
>>>>>
>>>>>
>>>>> On Sat, Mar 19, 2022 at 11:09 AM Bill Deegan <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Switching from py2.7 -> py 3.5 invalidates the .sconsign files
>>>>>> contents.
>>>>>> You have to do a full build when you make that switch.
>>>>>> It's in the release notes somewhere.
>>>>>>
>>>>>> https://scons.org/scons-300-is-available.html
>>>>>>
>>>>>> There's no way around it.
>>>>>> The best you can do is name your SConsign file to include the python
>>>>>> version so at least they won't be invalid when you switch from one to
>>>>>> another, but you will see additional rebuilds which may not be strictly
>>>>>> necessary.
>>>>>>
>>>>>> -Bill
>>>>>>
>>>>>> On Fri, Mar 18, 2022 at 5:03 PM Jay West <[email protected]> wrote:
>>>>>>
>>>>>>> Yes, our scons-3.1.2 was using python-2.7.17, and our scons-4.3.1 is
>>>>>>> using python-3.9.10.  We get a pickling error when trying to run
>>>>>>> scons-3.1.2 after scons-4.3.1 which is good.  Running scons-4.3.1 after
>>>>>>> scons-3.1.2 runs without error, but results in bad builds, which is
>>>>>>> terrible.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 18, 2022 at 3:04 PM Bill Deegan <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Sounds like you also switched from python 2.7 to python 3.5 or
>>>>>>>> higher?
>>>>>>>>
>>>>>>>> On Fri, Mar 18, 2022 at 2:07 PM Jay West <[email protected]> wrote:
>>>>>>>>
>>>>>>>>> We are migrating from scons-3.1.2 to scons-4.3.0 and have
>>>>>>>>> encountered incorrect builds.
>>>>>>>>>
>>>>>>>>> After building with scons-3.1.2, changing some files, and then
>>>>>>>>> running scons-4.3.0, it appears that scons-4.3.0 cannot read some
>>>>>>>>> dependencies from the DB file.  Instead of error'ing out or ignoring the DB
>>>>>>>>> file, it happily generates an incorrect build.  However, running:
>>>>>>>>>
>>>>>>>>> sconsign .sconsign.dblite
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> gives
>>>>>>>>>
>>>>>>>>> TypeError: '<' not supported between instances of 'str' and 'bytes'
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Has anyone seen this issue?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Jay
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Scons-users mailing list
>>>>>>>>> [email protected]
>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> [email protected]
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> [email protected]
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> [email protected]
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> [email protected]
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> [email protected]
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>> _______________________________________________
>> Scons-users mailing list
>> [email protected]
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> [email protected]
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>

_______________________________________________
Scons-users mailing list
[email protected]
https://pairlist4.pair.net/mailman/listinfo/scons-users