Re: [Fuego] Support for Debian Bullseye

> -----Original Message-----
> From: [email protected] <[email protected]>
> 
> Hello Tim,
> 
> We recently added support for Debian buster in Fuego.
> However, as you may already know Debian bullseye was recently released.
> 
> The new version does not support most of the Python 2x packages that Fuego relies on.
> Not even python-pip.

First, thanks for bringing up this issue.  The ever-changing nature of Linux
and distribution progress continues to present these challenges.  And now
is probably a good time to revisit the issues.

Bullseye was just released, so I think we have some time to adapt.  I believe
most embedded developers don't rapidly upgrade their development environments, but I don't really want to get stuck in the past.

Note that the docker container was introduced specifically to address this
type of distro churn.
 
> I think it is time to upgrade Fuego's python scripts to work on Python 3.x (in particular Bullseye python 3.9).
> However, I am afraid that we might break compatibility with older distributions. Regarding that,
> I considered these two options:
> 1) Port the python scripts and Dockerfile dependencies to use Python 3.x, and then see if a distribution fails
> 2) Focus only on Bullseye and leave the previous distributions working with Python 2.x
> 
> I think that the python3 libraries required by Fuego are available on all distros from Jessie to the latest. However,
> those libraries usually change behavior (we saw it in the case of python-openpyxl). Also, testing all the previous versions
> extensively would be too difficult and time consuming.

It would be good to review what python modules we use, and avoid
those with changing semantics.  I just hit this with another project,
where urllib changed semantics and APIs from python2 to python3.
You'd think that would be a pretty safe library, that the developers
would want to maintain backwards compatibility for!  But the
python community has a really terrible attitude about backwards
compatibility.

Although, openpyxl was particularly bad.

> 
> For that reason, I would like to propose that we go for the 2nd option.
I agree with this reasoning.  I'd rather not change a bunch of stuff from python 2.

I believe the main programs in Fuego that have these issues is 'ftc' and ltp_process.py, and the test Functional.fuego_release_test (which is broken now anyway).

The other areas where python is used extensively is the parser, and some tests that do post-processing of results into xml files
(which scripts seem, upon closer inspection) to be broken, anyway,
in a way unrelated to python issues). 
But none of the parser and xml post-processing scripts import anything exotic.

Well, here's a list of the imports from python scripts inside fuego-core/tests
sub-directories: (the first number is the number of times it was imported - usually
with 1 per script)
     1 atexit (fuego_release_test)
      1 docker (fuego_release_test)
      1 http (fuego_release_test)
      1 http.client (fuego_release_test)
      1 logging (fuego_release_test)
      1 getpass (fuego_release_test)
      1 matplotlib (iperf3)
      1 pexpect (fuego_release_test)
      1 pylab as plt (iperf3)
      1 subprocess (fuego_release_test)
      2 argparse (fuego_release_test)
      2 json
      2 io (fuego_release_test)
      2 os.path
      2 random
      2 requests (fuego_release_test)
      2 PIL (fuego_release_test)
      5 selenium fuego_release_test)
      5 openpyxl (LTP - LTP_process.py)
     32 xmltodict
     33 time
     75 re
     96 collections
    147 os
    148 sys

OK - so openpyxl and selenium are the weirdest ones there, and 
fuego_release_test has a long list of imports, but I'm not worried about
that test at the moment.

> 
> In that case, we need to decide what we do to for example support Buster and Bullseye at the same time. These are some
> options I considered
> 1) Detect the default python version on each script and then apply IF-ELSE when required.

Do you mean detect the Debian default version of python?
Has bullseye made /usr/bin/python a python3 interpreter?  that is going to break
a lot of things.

> 2) Create two versions of each python script (eg. ftc2.py and ftc3.py) and then use a link depending on which one we use
> 
> Each option has advantages and disadvantages so it is hard to choose. Option 2 will make maintainance quite hard.
> Option 1 would be good if the differences are small, but if there are too many differences it might be too much.
> 
> What do you think?

I think that the main issue here is ftc and ltp_process.py, but there are
a couple of other stragglers that may have issues.

I did a scan outside of the 'tests' directory and found that deorphan-runs.py imports 'jenkins', and that all other python programs in the core are using
fairly innocuous python modules, that I would expect even bullseye to have.

I note that pip is used to install 'filelock', 'flake8' and 'jenkins'.
filelock is used in the parser. Maybe that dependency could be eliminated.
flake8 is used for internal unit testing, and I suspect that could be migrated
to python3 easily, and 'jenkins' is primarily used by ftc.

I'd rather not migrate python2 code to python3 unless its needed.  There's
just way too much possibility to introduce subtle errors.

We already support a mix of python2 and python3 inside the docker container,
and I'd like to leave that alone.

I think I'd like to defer making any changes to the contents of the docker
container until we do our next container distro upgrade, which won't be
for a while.  Toshiba is mostly affected by native Fuego execution issues,
I believe.

Overall, I think I'd like to do some work on option 1, and see what happens.
I think for most python code in Fuego, we will likely be able to just
hardcode #!/usr/lib/python to #!/usr/lib/python2, if needed, fixing scripts to
that version of python (and the associated modules).

For ftc, I'd like to do a more detailed analysis of the imports, and where
the problems are, and see if I can just convert it to Python3.  The super-huge
issue here (speaking from experience with such conversions) will be handling the
change in string semantics.  The Unicode vs byte-string thing is a huge headache.

... [ OK - I just did some 2to3 analysis, and I get the following results
132 items need to be converted, including:
62 print statements
33 uses of list() need to be added
30 has_key() converted to 'in'
3 octal number conversions (0644 -> 0o644)
3 changes having to do with urllib
2 raw_input convertions to input
2 long() to int()
2 iteritems() to items()
2 changes related to importlib

That's actually not that much for the syntax (and the has_key conversion
is not strictly needed.)

Here are the imports for ftc:
I put stars next to the weird or likely problematic ones. A '-' indicates that the
module is not installed by default for python3 in the current 'stretch' Fuego
docker container.
*     1 ast
      1 csv
      1 fcntl
      1 getopt
      1 getpass
      1 glob
      1 json
      1 os
-     1 parsedatetime
      1 re
*     1 requests
      1 shutil
      1 signal
-     1 simplejson (not in python3 in old container, but there's a fallback to json if missing)
      1 subprocess
      1 sys
      1 tempfile
      1 time
*    1 urllib
-*    1 urlparse (from) (refactored in python3)
      2 datetime
-*    2 openpyxl (from) - for excel reports
      2 shutil (1 from)
-*    2 jenkins
-      2 yaml
-*    5 reportlab (from) - for PDF reports

The big question is whether it's possible to deal with the differences in these
to make ftc3 backwards compatible with the previous Fuego docker container
instances. (I'm not suggesting calling it ftc3, that's just a shorthand for 'the
python3 version of ftc).  That is, if someone has an old docker container, and they
do 'cd /fuego-core ; git pull' will things massively break.  I think maybe not,
but you are right, the testing to catch corner cases is a big issue.

I'm not sure whether some of these (openpyxl and reportlab) would be available
at all for python3 in the old containers.  Finding out what the status is of the
other non-core python modules, for python3, in the old containers, is potentially
a big issue.

Let me know what you think.
 -- Tim