Re: SCons 3.0, sconsign files and Py2 vs Py3.. how to handle sconsign imcompatabilities

Hi Bill,
the ascii related error (the one you get when generating using py2 first
then py3 second)
seems to be caused by
line 119, dblite.py, p = f.read()

python 2 reads then decodes a file to a string by default
but python 3 reads the file as binary without decoding.
it then feeds it into pickle.loads()
line 122, dblite.py, self._dict = pickle.loads(p)

I spotted some comments about something similar here
https://stackoverflow.com/questions/28218466/unpickling-a-python-2-object-with-python-3
https://stackoverflow.com/questions/30469575/how-to-pickle-and-unpickle-to-portable-string-in-python-3
I tried some simple decodes of the f.read to get it to work but I think it
might be a case of changing from pickle.loads to pickle.load with an
encoding parameter

For a future version of scons (past 3.0) it might be worth investigating
performance of different storage types.
Based on some of these links I think pickle 4 might be the fastest, then
json then pickle 2
https://stackoverflow.com/questions/2259270/pickle-or-json
https://konstantin.blog/2010/pickle-vs-json-which-is-faster/
https://stackoverflow.com/questions/23582489/python-pickle-protocol-choice

Many Thanks
Richard

On 10 September 2017 at 03:13, Bill Deegan <bill-cJFiu+DHMVC5azolltMz9laTQe2KTcn/@public.gmane.org>
wrote:

> Greetings,
>
> This is (I hope) the last issue gating 3.0 release.
>
> *If we set the default pickle version to 2. (Currently set to
> HIGHEST_AVAILABLE, which means 4 for py3.5+ and 2 for py2.7)*
>
> If you run scons with py2.7 and then run again with py3.x, you'll get the
> following:
> ...
> scons: *** [$DISTDIR/scons-local-3.0.0.alpha.20170821.tar.gz]
> UnicodeDecodeError : 'ascii' codec can't decode byte 0x80 in position 0:
> ordinal not in range(128)
>
> If you run scons with py3 and then rerun with py2.7, you'll get the
> following:
>  * It will rebuild everything you just built with py3
>
> *If we set the default pickle version  to HIGHEST_AVAILABLE, which means 4
> for py3.5+ and 2 for py2.7*
>
>
> Here's the options I see to deal with this:
> 1. Always append .py3 to PY3 scons runs' sconsign files
> 2. Always append .py# to sconsign files based on what version of python
> was used, ignore pre-existing .sconsigns from older versions
> 3. Do something complicated:
>    a. Leave pickle version as HIGHEST_AVAILABLE, detect version of pickle
> by reading first two bytes. If incompatible version detected, change file
> name to .py# and try reading that one.  Always write to .sconsign? Or write
> to file read.
>    b. ?
>
> Thoughts?
>
> -Bill
>
>
> _______________________________________________
> Scons-dev mailing list
> [email protected]
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>

_______________________________________________
Scons-dev mailing list
[email protected]
https://pairlist2.pair.net/mailman/listinfo/scons-dev