Re: [Fuego] Error in running the test after latest pull

> -----Original Message-----
> From: Pooja Sanjay More <[email protected]>
> Sent: Wednesday, December 9, 2020 12:08 AM
> To: [email protected]; Bird, Tim <[email protected]>; [email protected]
> Subject: Error in running the test after latest pull
> 
> Hi Tim,
> 
> While running a Functional.hello_world test on the rpi board  of pptlab  we are getting the following errors. We are getting this after the
> latest changes were pulled around 8-10 days back.
> 
> 
> Please find the logs below.
> 
This looks like 4 different issues:
 1. cannot change locale
 2. permission denied on build lock
 3. no test log on target
 4. file exists error on console log

One or more of these might be "cascade" errors, which means that some other
error is causing these, and the error is just a symptom of another error.

I'll comment on each of these inline below.

> Logs:
> 
> root@cpu-366U:/#  ftc run-test -b rpi -t Functional.hello_world
> Running test 'Functional.hello_world' on board 'rpi' using spec 'default'
> ===== doing fuego phase: pre_test =====
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

From the frequency of these, this looks like a host-side problem (and not a target-side
problem).  However, I could be wrong.

Can you do the following, and send me the results:
(inside the container):

ls -l /etc/environment ; cat /etc/environment
ls -l /etc/locale.gen ; cat /etc/locale.gen
ls -l /etc/locale.conf ; cat /etc/locale.conf

Mine are basically empty (except for the JENKINS_PORT) in my Docker container.

Please try one of the fixes below (inside the Docker container):
fix 1:
perl -pi -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/g' /etc/locale.gen
locale-gen en_US.UTF-8
update-locale en_US.UTF-8

fix 2:
Add support for en_US.UTF-8 inside the container.

echo "LC_ALL=en_US.UTF-8" >> /etc/environment
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
echo "LANG=en_US.UTF-8" > /etc/locale.conf
locale-gen en_US.UTF-8

(if you don't have locale-gen in the container, try "apt-get install locales")

If this doesn't fix it, try these steps on the raspberry pi board.

Please report back which of these fixes (if any) worked.

If it still doesn't fix it, then please get a full debug log of the run, and
send that.
$ ftc --debug run-test -b rpi -t hello_world

Note that you don't need to use the full test name "Functional.hello_world",
but only the shortened name.


> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> ===== doing fuego phase: build =====
> /fuego-core/scripts/functions.sh: line 403: /fuego-rw/buildzone/rpi.default.Functional.hello_world-/fuego.build.lock: Permission denied

This sounds like it was from a test that failed during the build phase and left
its lock file hanging around.

Try just removing it:
rm /fuego-rw/buildzone/rpi.default.Functional.hello_world-/fuego.build.lock

Also, there should be a toolchain name in the directory path for the build directory.
Does your rpi.board file specify a TOOLCHAIN setting?
# cat /fuego-rw/boards/rpi3-2.board | grep TOOL
TOOLCHAIN="debian-armhf"

(You can also use ftc to get this:
 # ftc query-board -b rpi3-2 -n TOOLCHAIN
debian-armhf

It looks like there's a bug in fuego-core/scripts/common.sh where the TOOLCHAIN
is not set before it is used to set the value of JOB_BUILD_DIR.  I made a fix and committed it.

I'm not sure what change I made that would have affect this.

> In Fuego signal_handler (due to error in build phase)
> ##### doing fuego phase: post_test (from signal handler) #####
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> scp: /home/pi/test/fuego.Functional.hello_world/Functional.hello_world.log: No such file or directory

I'd like to see the full log to diagnose by the test log is not being created on the target.
Please run the test with '--debug' and send the log.

> INFO: the test did not produce a test log on the target
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> ##### doing fuego phase: processing (from signal handler) #####
> ### WARNING: Program returned exit code ''
> ### WARNING: Log evaluation may be invalid
> !!! ERROR: ERROR: log_compare: pattern 'SUCCESS' found 0 times (expected greater or equal than 1)
> ERROR: test_processing returned an error
> chart config not found. Using default values.
> ERROR: results did not satisfy the threshold
> Traceback (most recent call last):
>   File "/usr/local/bin/ftc", line 6162, in <module>
>     main()
>   File "/usr/local/bin/ftc", line 6051, in main
>     rcode, run_id = do_run_test(conf, options)
>   File "/usr/local/bin/ftc", line 4710, in do_run_test
>     os.symlink(log_filename, jenkins_log_filename)
> OSError: [Errno 17] File exists

This looks like an issue with there being an already existing run for a test.
If so, I need to handle this issue better (or fail more gracefully).

Can you do the following, inside the container:

 $ ls -l /fuego-rw/logs/Functional.hello_world
and
 $ ls -l /var/lib/jenkins/jobs/*hello_world*/builds

and send me the results?
 
Sometimes this issue clears itself up as you execute more jobs
(from either Jenkins or using ftc on the command line).

However, ftc is supposed to catch this and avoid this error.

I added some code to try to adjust the build number better
depending on the state of both Jenkins build numbers and
Fuego log (/run) numbers.

Please do a 'git pull' on the fuego-core  repository, and try again.

Thanks very much for the bug report.
 -- Tim