Re: [Fuego] Report on python3 compatibility in Fuego (was something else)

Venkata,

Thanks very much for testing this out and reporting back.
 -- Tim


> -----Original Message-----
> From: [email protected] <[email protected]>
>=20
> Hi Tim,
>=20
> Thanks for improving the python3 compatibility problems, please see my in=
line comments.
>=20
> Thanks,
> Venkata.
>=20
> >-----Original Message-----
> >From: Bird, Tim <[email protected]>
> >Sent: 19 October 2022 23:15
> >To: pyla venkata(=1B$B#T#S#I#P=1B(B TMIEC ODG Porting) <Venkata.Pyla@tos=
hiba-
> >tsip.com>; [email protected]
> >Cc: nakkala sireesha(=1B$B#T#S#I#P=1B(B TMIEC ODG Porting) <sireesha.nak=
kala@toshiba-
> >tsip.com>; dinesh kumar(=1B$B#T#S#I#P=1B(B TMIEC ODG Porting)
> ><[email protected]>; hayashi kazuhiro(=1B$BNS=1B(B =1B$BOB9(=
=1B(B =1B$B""#S#W#C"~#A#C#T=1B(B)
> ><[email protected]>
> >Subject: Report on python3 compatibility in Fuego (was something else)
> >
> >OK - I'm hijacking this thread to so a general report on python3 compati=
bility in
> >Fuego.
> >
> >I recently completed a long stretch of work on this, and here are some n=
otes
> >about the work.
> >
> >I modified common.sh, functions.sh, and test-parser.sh to set and use PY=
_EXE I
> >experimented with doing an autodetect to use python3 or python (2).
> >But ultimately I decided that doing an runtime autodetect is not a good =
policy,
> >as it means that factors unrelated to Fuego (such as installing a versio=
n of
> >python) can affect Fuego in unexpected ways.
> >
> >So, I instead opted to create a script which can switch Fuego from pytho=
n to
> >python3 usage, or vice-versa.  That is, if a user switches Fuego to pyth=
on3, but
> >then discovers problems, they can switch back to python2.  The new utili=
ty is
> >called set-python, and is in fuego-core/scripts.  It also checks that th=
e required
> >libraries for the specified python version are present on the system.  I=
n order to
> >switch to a version of python, that version of python must be present on=
 your
> >system.
> >
> >The tool is "idempotent", meaning that if you run it twice with the same
> >arguments, it should not change anything the second time.  Specifically,=
 it
> >should not have any deleterious effects.  So, if you're already done a c=
onversion
> >to python3, you can run it again and it should not cause any problems.  =
Note
> >that you can also run the tool with '-c' to just check the status, witho=
ut making
> >any changes.
>=20
> 'set-python' tool is a nice tool that you prepared, it is checking and mo=
difying the shebang lines according to the system python
> version, this perfectly suited for us as well.
> I applied your changes in our project and I randomly checked and it is wo=
rking.
>=20
> We will further verify while executing some of the tests and if there are=
 any problems we will send fixes for them or report to you.
>=20
> >
> >The tool must be run in fuego-core/scripts, outside of any docker contai=
ner.
> >Use '-h'
> >to get usage help.  The script changes all shebang python lines in fuego=
 scripts,
> >with the exception of parser.py shebang lines, which are always ignored.
> >
> >I removed all the shebang lines from the parser core (fuego-
> >core/scripts/parser/*.py), but decided to leave the shebang lines in all=
 the
> >test/*/parser.py files.  They could be removed (and probably should be),=
 but I
> >did a lot of changes this release and wanted to push these changes out b=
efore
> >doing another large set of changes.  The shebang lines in parser.py file=
s are
> >never used, so they are superfluous, and don't reflect how the scripts w=
ill
> >actually be called.
> >
> >I also cleaned up remaining python3 incompatibilities in ftc and a few o=
ther
> >scripts.
> >Specifically, I handled the input vs. raw_input issue, and the urllib.pa=
rse vs.
> >urlparse issue.
> >
> >Please try out the latest master branch and let me know if you have any
> >problems, especially with python3 compatibility.
> >
> >If you could run set-python in your lab, and let me know what it shows y=
ou,
> >that would be helpful.
> >That is please do:
> >  $ cd fuego-core/scripts ; set-python python3 (or 'set-python -c python=
3') and
> >let me know the results.
>=20
> I executed the 'set-python' in our environment with your patches and belo=
w are the results
>=20
> 1)
> root@cloud:/fuego-core/scripts# ./set-python -c python3
> Python version I am executing=3D'python3' (3, 9, 2)
> Python version requested=3D'python3'
> Report:
>   Missing module jenkins
>=20
> Here are the lines that need to change!
> =3D=3D check-dependencies =3D=3D
> =3D=3D deorphan-runs.py =3D=3D
> =3D=3D ftc =3D=3D
> =3D=3D gen-page.py =3D=3D
> =3D=3D jdiff =3D=3D
> =3D=3D ../tests/Functional.LTP/ltp_process.py =3D=3D
> =3D=3D ../tests/Functional.apertis_sanity_check/make-script-from-yaml.py =
=3D=3D
> =3D=3D common.sh =3D=3D
> =3D=3D test-parser.sh =3D=3D
>=20
> 2)
> root@cloud:/fuego-core/scripts# ./set-python python3
> Python version I am executing=3D'python3' (3, 9, 2)
> Python version requested=3D'python3'
> Report:
>   Missing module jenkins
>=20
> Here are some line changes!
> =3D=3D check-dependencies =3D=3D
> #!/usr/bin/python
>  -- is now --
> #!/usr/bin/python3
> =3D=3D deorphan-runs.py =3D=3D
> #!/usr/bin/python
>  -- is now --
> #!/usr/bin/python3
> =3D=3D ftc =3D=3D
>  -- is now --
> #!/usr/bin/python3
> =3D=3D gen-page.py =3D=3D
> #!/usr/bin/python
>  -- is now --
> #!/usr/bin/python3
> =3D=3D jdiff =3D=3D
> #!/usr/bin/python
>  -- is now --
> #!/usr/bin/python3
> =3D=3D ../tests/Functional.LTP/ltp_process.py =3D=3D
> #!/usr/bin/python
>  -- is now --
> #!/usr/bin/python3
> =3D=3D ../tests/Functional.apertis_sanity_check/make-script-from-yaml.py =
=3D=3D
> #!/usr/bin/python
>  -- is now --
> #!/usr/bin/python3
> =3D=3D common.sh =3D=3D
> PY_EXE=3D/usr/bin/python
>  -- is now --
> PY_EXE=3D/usr/bin/python3
> =3D=3D test-parser.sh =3D=3D
> PY_EXE=3D/usr/bin/python
>  -- is now --
> PY_EXE=3D/usr/bin/python3
>=20
> I saw the shebang lines are changed to python3 and the tests are working.
>=20
> >
> >I haven't decided to switch Fuego to install using python3 by default, b=
ut I may
> >do so after I experiment with making serio python3 compatible, and when =
I do
> >the next major release.
> >I have modified the install scripts to load all the needed python3 libra=
ries, but
> >haven't done a docker test yet with these.
> >
> >Regards,
> > -- Tim
> >
> >> -----Original Message-----
> >> From: [email protected] <[email protected]>
> >> Sent: Monday, October 10, 2022 7:00 AM
> >> To: Bird, Tim <[email protected]>; [email protected]
> >> Cc: [email protected]; [email protected];
> >> [email protected]
> >> Subject: RE: [PATCH] python3: Use python3 in the scripts to fully
> >> migrate
> >>
> >> Hi Tim,
> >>
> >> Sorry for the late, I am on vacation last week.
> >>
> >> Please find my answers below.
> >>
> >> >-----Original Message-----
> >> >From: Bird, Tim <[email protected]>
> >> >Sent: 05 October 2022 05:52
> >> >To: pyla venkata(=1B$B#T#S#I#P=1B(B TMIEC ODG Porting) <Venkata.Pyla@=
toshiba-
> >> >tsip.com>; [email protected]
> >> >Cc: nakkala sireesha(=1B$B#T#S#I#P=1B(B TMIEC ODG Porting)
> >> ><sireesha.nakkala@toshiba- tsip.com>; dinesh kumar(=1B$B#T#S#I#P=1B(B=
 TMIEC ODG
> >> >Porting) <dinesh.kumar@toshiba- tsip.com>; hayashi kazuhiro(=1B$BNS=
=1B(B =1B$BOB9(=1B(B
> >> >=1B$B""#S#W#C"~#A#C#T=1B(B)
> >> ><[email protected]>
> >> >Subject: RE: [PATCH] python3: Use python3 in the scripts to fully
> >> >migrate
> >> >
> >> >OK -  I have some more comments on this patch.
> >> >
> >> >See inline below.
> >> >
> >> >> -----Original Message-----
> >> >> From: [email protected] <[email protected]>
> >> >>
> >> >> The python scripts are failing to run in an environment where
> >> >> `python` is not defined, though the scripts are migrated to work on
> >> >> python3 the shebang is still pointing to python which may be
> >> >> undefined
> >> >>
> >> >> Signed-off-by: venkata pyla <[email protected]>
> >> >> ---
> >> >>  scripts/check-dependencies              | 2 +-
> >> >
> >> >My first comment is that invocation using just the interpreter name
> >> >of 'python' is in a lot more places than just these 4 files, and the =
parser.py
> >scripts.
> >> >
> >> >I see it also in deorphan-runs.py, gen-page.py, jdiff, test_parser.sh
> >> >, generic_parser.py, test_filelock.py (in the scripts directory).
> >> >Many of these you are likely not using.
> >> Yes, many of the files were not used by Toshiba, so we fixed only in t=
he tests
> >where it is breaking.
> >>
> >> >
> >> >However, some of these are intended for use by Fuego users, for
> >> >special circumstances (like deorphan-runs.py and test_parser.sh).
> >>
> >>
> >> >
> >> >>  scripts/common.sh                       | 8 ++++----
> >> >>  scripts/ftc                             | 2 +-
> >> >>  scripts/functions.sh                    | 2 +-
> >> >>  tests/Benchmark.Dhrystone/parser.py     | 2 +-
> >> >>  tests/Benchmark.IOzone/parser.py        | 2 +-
> >> >>  tests/Benchmark.Stream/parser.py        | 2 +-
> >> >>  tests/Benchmark.bonnie/parser.py        | 2 +-
> >> >>  tests/Benchmark.cyclictest/parser.py    | 2 +-
> >> >>  tests/Benchmark.fio/parser.py           | 2 +-
> >> >>  tests/Benchmark.hackbench/parser.py     | 2 +-
> >> >>  tests/Benchmark.lmbench2/parser.py      | 2 +-
> >> >>  tests/Benchmark.migratetest/parser.py   | 2 +-
> >> >>  tests/Benchmark.pmqtest/parser.py       | 2 +-
> >> >>  tests/Benchmark.ptsematest/parser.py    | 2 +-
> >> >>  tests/Benchmark.signaltest/parser.py    | 2 +-
> >> >>  tests/Benchmark.sigwaittest/parser.py   | 2 +-
> >> >>  tests/Benchmark.svsematest/parser.py    | 2 +-
> >> >>  tests/Functional.LTP/fuego_test.sh      | 4 ++--
> >> >>  tests/Functional.LTP/ltp_process.py     | 2 +-
> >> >>  tests/Functional.LTP/parser.py          | 2 +-
> >> >>  tests/Functional.LTP_one_test/parser.py | 2 +-
> >> >> tests/Functional.autopkgtest/parser.py  | 2 +-
> >> >>  tests/Functional.linaro/parser.py       | 2 +-
> >> >>  24 files changed, 28 insertions(+), 28 deletions(-)
> >> >>
> >> >> diff --git a/scripts/check-dependencies
> >> >> b/scripts/check-dependencies index 583012e..7369bd4 100755
> >> >> --- a/scripts/check-dependencies
> >> >> +++ b/scripts/check-dependencies
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >
> >> >check-dependencies is used by Functional.LTP, so I'm surprised you
> >> >haven't run into this one.  Maybe you have and this program works fin=
e with
> >python3.
> >> We use LTP test and so we changed the shebang here, and as you
> >> suggested below it is good to use `run_python` which will internally f=
ind the
> >right interpreter as you wrote the code, then this shebang is not requir=
ed or we
> >can remove.
> >>
> >> >
> >> >check-dependencies is invoked directly (not using run_python), or by
> >> >calling the interpreter, but it is only called in one place
> >> >(Functional.LTP/fuego_test.sh), so maybe that one place could call
> >> >run_python, to make it so that this would work with a conditional
> >interpreter.
> >> >(see below for how I might make common.sh auto-detect the python
> >> >interpreter)
> >> This will be a good solution for the backward compatibility.
> >>
> >> >
> >> >>  # vim: set ts=3D4 sw=3D4 et :
> >> >>  #
> >> >>  # check_dependencies - check config dependencies listed in a file
> >> >> diff --git a/scripts/common.sh b/scripts/common.sh index
> >> >> f916d84..cf93ddb 100644
> >> >> --- a/scripts/common.sh
> >> >> +++ b/scripts/common.sh
> >> >> @@ -108,20 +108,20 @@ function run_python() {
> >> >>      if [ ! -z $ORIG_PATH ] ; then
> >> >>          dprint "run_python with PATH=3D$ORIG_PATH,
> >TOOLCHAIN=3D$TOOLCHAIN"
> >> >>          export TOOLCHAIN
> >> >> -        PATH=3D$ORIG_PATH TOOLCHAIN=3D$TOOLCHAIN python "$@"
> >> >> +        PATH=3D$ORIG_PATH TOOLCHAIN=3D$TOOLCHAIN python3 "$@"
> >> >For this one and the next, I'm thinking of using a variable to
> >> >indicate the
> >> >interpreter:
> >> >PYTHON_EXE, that I would detect outside of the run_python function,
> >> >with something like this:
> >> >
> >> >INTERPRETER_LIST=3D"python3 python python2"
> >> >for interpreter in $INTERPRETER_LIST ; do
> >> >    if [ -x /usr/bin/${interpreter} ] ; then
> >> >        PYTHON_EXE=3D"/usr/bin/${interpreter}"
> >> >        break
> >> >    fi
> >> >done
> >> >if [ -z "$PYTHON_EXE" ] ; then
> >> >    abort_job "No python interpreter found!"
> >> >fi
> >> >
> >> >this line would then become:
> >> >PATH=3D$ORIG_PATH TOOLCHAIN=3D$TOOLCHAIN $PYTHON_EXE "$@"
> >> >
> >> >Note the use of $ORIG_PATH.  Some toolchain setup scrips modify the
> >> >PATH so that the 'python' that is executed is one that the SDK for
> >> >the toolchain set up, with libs from the target sysroot.  This is not
> >> >desired for Fuego's use of python, which should always use the host's
> >> >python interpreter.  I'll have to see if this proposed change (which
> >> >would use the fullpath to the interpreter, instead of just the
> >> >interpreter found in the PATH), would affect this use of ORIG_PATH wi=
th
> >some toolchains.
> >> >
> >> >Let me know if you have any thoughts about this.
> >> >Specifically, does your toolchain setup script set ORIG_PATH?
> >> We are not using the ORIG_PATH variable, in our case the host environm=
ent
> >interpreter is sufficient to run the scripts.
> >>
> >> But in our patch we have modified in both the places when ORIG_PATH is
> >> defined and not defined because we wanted at least change python versi=
on
> >problem in the same file.
> >>
> >> >
> >> >>      else
> >> >>          dprint "run_python with TOOLCHAIN=3D$TOOLCHAIN"
> >> >>          export TOOLCHAIN
> >> >> -        TOOLCHAIN=3D$TOOLCHAIN python "$@"
> >> >> +        TOOLCHAIN=3D$TOOLCHAIN python3 "$@"
> >> >and this would become:
> >> >TOOLCHAIN=3D$TOOLCHAIN $PYTHON_EXE "$@"
> >> >
> >> >>      fi
> >> >>  }
> >> >>
> >> >>  function run_python_quiet() {
> >> >>      if [ ! -z $ORIG_PATH ]
> >> >>      then
> >> >> -        PATH=3D$ORIG_PATH python "$@"
> >> >> +        PATH=3D$ORIG_PATH python3 "$@"
> >> >As near as I can tell, run_python_quiet is only ever used by
> >> >overlays/base/base- params.fuegoclass (and it shows up in expected
> >> >values in Functional.fuego_ftc_test).
> >> >This is used to invoke sercp, serlogin, and sersh for serial port
> >> >connections to the target.
> >> >(that is, when the TRANSPORT=3Dserial)
> >> >
> >> >I have NOT tested sercp, serlogin and sersh for python3 compatibility
> >> >(but I already know they are NOT python3 compatible right now).
> >> >Given that they are managing serial port data flow on the  serial
> >> >port at a byte-at-a-time level, they will need extensive analysis to
> >> >make sure that the string handling does not break when they are run
> >> >with python3 (where strings default to Unicode instead of byte string=
s).
> >> >I have another project on github that also handles data flow on a
> >> >serial port, and it was very difficult to make it work correctly on b=
oth
> >python2 and python3.
> >> >This one I'll have to defer.
> >> >
> >> >If I understand correctly, the way that you use Fuego is by
> >> >installing it natively onto the device under test, and using
> >> >TRANSPORT=3Dlocal.  If you don't use the serial TRANSPORT or the seri=
al
> >> >port tools (serio suite of tools), then this shouldn't affect you.
> >> Yes, this change is not required for us, as mentioned above we modifie=
d all
> >python versions in the same file.
> >> You can ignore this change and thanks for letting me know about the fu=
nction
> >'run_python_quiet'
> >>
> >> >
> >> >In any event, this can not be changed to python3 yet.
> >> >>      else
> >> >> -        python "$@"
> >> >> +        python3 "$@"
> >> >Nor this one. (see above)
> >> >
> >> >>      fi
> >> >>  }
> >> >>
> >> >> diff --git a/scripts/ftc b/scripts/ftc index adce17b..4c5ff7e
> >> >> 100755
> >> >> --- a/scripts/ftc
> >> >> +++ b/scripts/ftc
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >
> >> >Have you been running ftc with this change already?
> >> >
> >> >ftc has a large number of dependencies on python modules, especially
> >> >in the area of report generation.  The docker container never
> >> >installs python3 versions of some of these esoteric modules, such as
> >reportlab and openpyxl.
> >> >
> >> >Just out of curiosity, do you import these modules in your
> >> >environment, or just not use the ftc functionality that depends on th=
em?
> >> We installed python3 dependencies in our target machine.
> >> as you may know we are not using the docker-container and installing
> >> the fuego-core directly on the target machine, and the python3 depende=
ncies
> >are already installed in the target machine.
> >>
> >> >
> >> >Also, ftc has a few remaining python3 incompatibilities:
> >> >  -  raw_input vs input
> >> >  - some urllib-related fixes, for the python3 refactoring of
> >> >ulrparse and urllib
> >> >
> >> >Is ftc functioning correctly with python3 now, for all your usage sce=
narios?
> >> It is working fine with our current usage scenarios, because in our
> >> environment we were not using the Jenkins or the features related to
> >> those functions I think we should fix them because they are obviously
> >required when completely migrate to pyhon3.
> >>
> >> >
> >> >>  #
> >> >>  # vim: set ts=3D4 sw=3D4 et :
> >> >>  #
> >> >> diff --git a/scripts/functions.sh b/scripts/functions.sh index
> >> >> 7afe423..5a6e0e5 100755
> >> >> --- a/scripts/functions.sh
> >> >> +++ b/scripts/functions.sh
> >> >> @@ -506,7 +506,7 @@ function build {
> >> >>          call_if_present test_build
> >> >>          ret=3D$?
> >> >>          build_end_time=3D$(date +"%s.%N")
> >> >> -        build_duration=3D$(python -c "print($build_end_time -
> >$build_start_time)")
> >> >> +        build_duration=3D$(python3 -c "print($build_end_time -
> >> >> + $build_start_time)")
> >> >If I do the PYTHON_EXE thing in common.sh, I should be able to use
> >> >that here as well.
> >> >
> >> >>
> >> >>          # test_build may change the current dir
> >> >>          # get back to root of build dir, before 'touch'
> >> >> diff --git a/tests/Benchmark.Dhrystone/parser.py
> >> >> b/tests/Benchmark.Dhrystone/parser.py
> >> >> index fd07cff..868f9de 100755
> >> >> --- a/tests/Benchmark.Dhrystone/parser.py
> >> >> +++ b/tests/Benchmark.Dhrystone/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >Finally, on all these parser shebangs, I'm not sure what to do.
> >> >
> >> >I experimented with creating a wrapper script, called run_python.sh,
> >> >that determines the correct interpreter to use, and changing these to=
:
> >> >#!/fuego-core/scripts/run_python.sh
> >> >
> >> >It works, but I don't know how robust it is.  These parser are never
> >> >invoked directly, as far as I can tell.  It might be better to
> >> >actually remove the shebang line to make it explicitly impossible to
> >> >treat the parser.py scripts as standalone programs.
> >> >
> >> >I'm not sure if I'd be breaking anyone's workflow with this or not,
> >> >so I hesitate to do that.
> >> >
> >> >What do you think?
> >> In our use cases also we were not directly running them, so we can rem=
ove
> >the shebang lines if no one else also not using it directly.
> >>
> >> >
> >> >>
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >> diff --git a/tests/Benchmark.IOzone/parser.py
> >> >> b/tests/Benchmark.IOzone/parser.py
> >> >> index b1631ae..a56aa0e 100755
> >> >> --- a/tests/Benchmark.IOzone/parser.py
> >> >> +++ b/tests/Benchmark.IOzone/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>
> >> >>  import os, sys
> >> >>  import common as plib
> >> >> diff --git a/tests/Benchmark.Stream/parser.py
> >> >> b/tests/Benchmark.Stream/parser.py
> >> >> index a564d74..926e611 100755
> >> >> --- a/tests/Benchmark.Stream/parser.py
> >> >> +++ b/tests/Benchmark.Stream/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >> diff --git a/tests/Benchmark.bonnie/parser.py
> >> >> b/tests/Benchmark.bonnie/parser.py
> >> >> index b68bc3e..962a7cd 100755
> >> >> --- a/tests/Benchmark.bonnie/parser.py
> >> >> +++ b/tests/Benchmark.bonnie/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >> diff --git a/tests/Benchmark.cyclictest/parser.py
> >> >> b/tests/Benchmark.cyclictest/parser.py
> >> >> index c29393e..4252654 100755
> >> >> --- a/tests/Benchmark.cyclictest/parser.py
> >> >> +++ b/tests/Benchmark.cyclictest/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >>
> >> >> diff --git a/tests/Benchmark.fio/parser.py
> >> >> b/tests/Benchmark.fio/parser.py index ab3ea34..fedb734 100644
> >> >> --- a/tests/Benchmark.fio/parser.py
> >> >> +++ b/tests/Benchmark.fio/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>  # See common.py for description of command-line arguments
> >> >>
> >> >>  import os, sys
> >> >> diff --git a/tests/Benchmark.hackbench/parser.py
> >> >> b/tests/Benchmark.hackbench/parser.py
> >> >> index a22c480..3ac9ba5 100755
> >> >> --- a/tests/Benchmark.hackbench/parser.py
> >> >> +++ b/tests/Benchmark.hackbench/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >> diff --git a/tests/Benchmark.lmbench2/parser.py
> >> >> b/tests/Benchmark.lmbench2/parser.py
> >> >> index f06f132..410b46e 100755
> >> >> --- a/tests/Benchmark.lmbench2/parser.py
> >> >> +++ b/tests/Benchmark.lmbench2/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >> diff --git a/tests/Benchmark.migratetest/parser.py
> >> >> b/tests/Benchmark.migratetest/parser.py
> >> >> index 627ec2d..21b7c56 100755
> >> >> --- a/tests/Benchmark.migratetest/parser.py
> >> >> +++ b/tests/Benchmark.migratetest/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >>
> >> >> diff --git a/tests/Benchmark.pmqtest/parser.py
> >> >> b/tests/Benchmark.pmqtest/parser.py
> >> >> index 05ee57b..a5c31ba 100755
> >> >> --- a/tests/Benchmark.pmqtest/parser.py
> >> >> +++ b/tests/Benchmark.pmqtest/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >> diff --git a/tests/Benchmark.ptsematest/parser.py
> >> >> b/tests/Benchmark.ptsematest/parser.py
> >> >> index 05ee57b..a5c31ba 100755
> >> >> --- a/tests/Benchmark.ptsematest/parser.py
> >> >> +++ b/tests/Benchmark.ptsematest/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >> diff --git a/tests/Benchmark.signaltest/parser.py
> >> >> b/tests/Benchmark.signaltest/parser.py
> >> >> index 1992b21..95d4340 100755
> >> >> --- a/tests/Benchmark.signaltest/parser.py
> >> >> +++ b/tests/Benchmark.signaltest/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >> diff --git a/tests/Benchmark.sigwaittest/parser.py
> >> >> b/tests/Benchmark.sigwaittest/parser.py
> >> >> index 25a0262..120b7b5 100755
> >> >> --- a/tests/Benchmark.sigwaittest/parser.py
> >> >> +++ b/tests/Benchmark.sigwaittest/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >>
> >> >> diff --git a/tests/Benchmark.svsematest/parser.py
> >> >> b/tests/Benchmark.svsematest/parser.py
> >> >> index 05ee57b..a5c31ba 100755
> >> >> --- a/tests/Benchmark.svsematest/parser.py
> >> >> +++ b/tests/Benchmark.svsematest/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >> diff --git a/tests/Functional.LTP/fuego_test.sh
> >> >> b/tests/Functional.LTP/fuego_test.sh
> >> >> index ef58bcc..95dea2c 100755
> >> >> --- a/tests/Functional.LTP/fuego_test.sh
> >> >> +++ b/tests/Functional.LTP/fuego_test.sh
> >> >> @@ -475,9 +475,9 @@ function test_processing {
> >> >>          #  ImportError: No module named style
> >> >>          if [ -n "$ORIG_PATH" ] ; then
> >> >>              # Use ORIG_PATH, if defined, so that python works prop=
erly
> >> >> -            PATH=3D$ORIG_PATH python ltp_process.py
> >> >> +            PATH=3D$ORIG_PATH python3 ltp_process.py
> >> >>          else
> >> >> -            python ltp_process.py
> >> >> +            python3 ltp_process.py
> >> >>          fi
> >> >>
> >> >>          [ -e results.xlsx ] && cp results.xlsx
> >> >> ${LOGDIR}/results.xlsx diff --git
> >> >> a/tests/Functional.LTP/ltp_process.py
> >> >> b/tests/Functional.LTP/ltp_process.py
> >> >> index 8e2b637..27bc856 100644
> >> >> --- a/tests/Functional.LTP/ltp_process.py
> >> >> +++ b/tests/Functional.LTP/ltp_process.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>  # -*- coding: UTF-8 -*-
> >> >>  from openpyxl import Workbook
> >> >>  from openpyxl.styles import Border, Side, PatternFill, Color,
> >> >> Alignment diff --git a/tests/Functional.LTP/parser.py
> >> >> b/tests/Functional.LTP/parser.py index 9ad7659..3467328 100755
> >> >> --- a/tests/Functional.LTP/parser.py
> >> >> +++ b/tests/Functional.LTP/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>  # -*- coding: UTF-8 -*-
> >> >>  import os, os.path, re, sys
> >> >>  import common as plib
> >> >> diff --git a/tests/Functional.LTP_one_test/parser.py
> >> >> b/tests/Functional.LTP_one_test/parser.py
> >> >> index 312bd5b..7002e35 100755
> >> >> --- a/tests/Functional.LTP_one_test/parser.py
> >> >> +++ b/tests/Functional.LTP_one_test/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>  # See common.py for description of command-line arguments
> >> >>
> >> >>  import os
> >> >> diff --git a/tests/Functional.autopkgtest/parser.py
> >> >> b/tests/Functional.autopkgtest/parser.py
> >> >> index ba3dea1..ba8d2a0 100755
> >> >> --- a/tests/Functional.autopkgtest/parser.py
> >> >> +++ b/tests/Functional.autopkgtest/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/bin/python
> >> >> +#!/bin/python3
> >> >>
> >> >>  import os, re, sys
> >> >>  import common as plib
> >> >> diff --git a/tests/Functional.linaro/parser.py
> >> >> b/tests/Functional.linaro/parser.py
> >> >> index 48b502b..5bbb5de 100755
> >> >> --- a/tests/Functional.linaro/parser.py
> >> >> +++ b/tests/Functional.linaro/parser.py
> >> >> @@ -1,4 +1,4 @@
> >> >> -#!/usr/bin/python
> >> >> +#!/usr/bin/python3
> >> >>
> >> >>  import os, sys, collections
> >> >>  import common as plib
> >> >> --
> >> >> 2.20.1
> >> >>
> >> >
> >> >Finally, before changing the shebang in all the parser.py scripts,
> >> >I'd like to implement the compatibility changes first.  That is,
> >> >change all the print statements to be parenthesized, BEFORE changing
> >> >the shebang lines.  This way the code is never left in a faulty state=
 (such that
> >a git bisect would fail).
> >> >
> >> >Let me know your thoughts on the issues above.
> >> >
> >> >It looks like at least some of you are on vacation the next few days,
> >> >but I'll wait to hear back before I start attacking this problem.
> >> >
> >> >By the way - what distribution of Linux are you testing?  Is this for
> >> >CIP? or for some Toshiba-internal distro of Linux?  (just curious).
> >> We are using on Debian based distribution with Bullseye version, this
> >> is one of the reason why we are changing the scripts to support python=
3,
> >because bullseye in not supporting python2.
> >>
> >> At the first we have changed in few tests and some common function
> >> scripts which we are using, because in our environment the fuego-core
> >> runs on test/target machine directly this patch works for us, and I am
> >> not confident these changes will work with docker- container, so the
> >> reason we have not shared this patch initially and only the shared
> >> some print statement fixes and other
> >> python2to3 compatible fixes.
> >>
> >> After studying your suggestion in the above, there are quite few other
> >conversions required to fully convert to python3.
> >> Let me know if I can do some of the changes.
> >>
> >> >
> >> > -- Tim
> >>