Re: [Fuego] Error in running the test after latest pull

This looks like it was caused by a server error ( http error 500), but ftc should have handled
this.  I made a modification to ftc to handle this, and pushed the commit to the master branch.
Let me know if you see this again.

 -- Tim


> -----Original Message-----
> From: Pooja Sanjay More <[email protected]>
> 
> Hi Tim,
> 
> Issues got fixed with the suggested changes and with recent pull. We tested a case by logging in a request in
> 
> fserver for a test to be run on rpi. This test also went smooth, but the results could not be
> 
> uploaded to the server due to the fserver issue.
> 
> Please find the logs below:
> 
> ftc run-request --put-run
> Trying to get request 'request-2020-12-10_05:36:12.67-himanshu_wks:rpi' from server
> TRB: best_local_match=('', u'Benchmark.Dhrystone', '1.1-1')
> TRB: best_remote_match=None
> TRB: candidate=('', u'Benchmark.Dhrystone', '1.1-1')
> Executing test Benchmark.Dhrystone on board rpi (using spec default)
> Running test 'Benchmark.Dhrystone' on board 'rpi' using spec 'default'
> ===== doing fuego phase: pre_test =====
> ===== doing fuego phase: build ========
> The test is already built
> Fuego test_build duration=0 seconds
> ===== doing fuego phase: deploy =====
> ===== doing fuego phase: snapshot =====
> Firmware revision: 5.4.81-v7-40+
> ===== doing fuego phase: run =====
> -------------------------------------------------
> 
> Dhrystone Benchmark, Version 2.1 (Language: C)
> 
> Program compiled without 'register' attribute
> 
> Please give the number of runs through the benchmark:
> Execution starts, 10000000 runs through Dhrystone
> Execution ends
> 
> Final values of the variables used in the benchmark:
> 
> Int_Glob:            5
>         should be:   5
> Bool_Glob:           1
>         should be:   1
> Ch_1_Glob:           A
>         should be:   A
> Ch_2_Glob:           B
>         should be:   B
> Arr_1_Glob[8]:       7
>         should be:   7
> Arr_2_Glob[8][7]:    10000010
>         should be:   Number_Of_Runs + 10
> Ptr_Glob->
>   Ptr_Comp:          10326352
>         should be:   (implementation-dependent)
>   Discr:             0
>         should be:   0
>   Enum_Comp:         2
>         should be:   2
>   Int_Comp:          17
>         should be:   17
>   Str_Comp:          DHRYSTONE PROGRAM, SOME STRING
>         should be:   DHRYSTONE PROGRAM, SOME STRING
> Next_Ptr_Glob->
>   Ptr_Comp:          10326352
>         should be:   (implementation-dependent), same as above
>   Discr:             0
>         should be:   0
>   Enum_Comp:         1
>         should be:   1
>   Int_Comp:          18
>         should be:   18
>   Str_Comp:          DHRYSTONE PROGRAM, SOME STRING
>         should be:   DHRYSTONE PROGRAM, SOME STRING
> Int_1_Loc:           5
>         should be:   5
> Int_2_Loc:           13
>         should be:   13
> Int_3_Loc:           7
>         should be:   7
> Enum_Loc:            1
>         should be:   1
> Str_1_Loc:           DHRYSTONE PROGRAM, 1'ST STRING
>         should be:   DHRYSTONE PROGRAM, 1'ST STRING
> Str_2_Loc:           DHRYSTONE PROGRAM, 2'ND STRING
>         should be:   DHRYSTONE PROGRAM, 2'ND STRING
> 
> Microseconds for one run through Dhrystone:    1.0
> Dhrystones per Second:                      1000000.0
> 
> -------------------------------------------------
> ===== doing fuego phase: post_test =====
> ===== doing fuego phase: processing =====
> Going to check 1 fail cases for rpi.default.Benchmark.Dhrystone
> No fail cases detected for rpi.default.Benchmark.Dhrystone
> Fuego: requested test phases complete!
> Packaging run 'Benchmark.Dhrystone-default-16-rpi'
> run/
> run/testlog.txt
> run/machine-snapshot.txt
> run/build.xml
> run/syslog.after.txt
> run/prolog.sh
> run/devlog.txt
> run/spec.json
> run/consolelog.txt
> run/run.json
> run/syslog.before.txt
> Run packaged successfully, and is at: /tmp/run-Benchmark.Dhrystone-default-16-on-himanshu_wks:rpi.frp
> url = http://birdcloud.org/cgi-bin/fserver.py/?action=put_run
> file to send= run-Benchmark.Dhrystone-default-16-on-himanshu_wks:rpi.frp
> response from server = <Response [500]>
> Traceback (most recent call last):
>   File "/usr/local/bin/ftc", line 6175, in <module>
>     main()
>   File "/usr/local/bin/ftc", line 6037, in main
>     do_run_request(conf, options)
>   File "/usr/local/bin/ftc", line 3553, in do_run_request
>     do_put_run(conf, [run_id])
>   File "/usr/local/bin/ftc", line 4223, in do_put_run
>     result, content = put_run_fuego(conf, run_filepath)
> ValueError: need more than 1 value to unpack
> 
> Thanks and regards,
> Pooja More
> 
> 
> 
> On Thu, Dec 10, 2020 at 12:10 AM Bird, Tim <[email protected] <mailto:[email protected]> > wrote:
> 
> 
> 
> 
> 	> -----Original Message-----
> 	> From: Pooja Sanjay More <[email protected] <mailto:[email protected]> >
> 	> Sent: Wednesday, December 9, 2020 12:08 AM
> 	> To: [email protected] <mailto:[email protected]> ; Bird, Tim <[email protected] <mailto:[email protected]> >;
> [email protected] <mailto:[email protected]>
> 	> Subject: Error in running the test after latest pull
> 	>
> 	> Hi Tim,
> 	>
> 	> While running a Functional.hello_world test on the rpi board  of pptlab  we are getting the following errors. We are getting this
> after the
> 	> latest changes were pulled around 8-10 days back.
> 	>
> 	>
> 	> Please find the logs below.
> 	>
> 	This looks like 4 different issues:
> 	 1. cannot change locale
> 	 2. permission denied on build lock
> 	 3. no test log on target
> 	 4. file exists error on console log
> 
> 	One or more of these might be "cascade" errors, which means that some other
> 	error is causing these, and the error is just a symptom of another error.
> 
> 	I'll comment on each of these inline below.
> 
> 	> Logs:
> 	>
> 	> root@cpu-366U:/#  ftc run-test -b rpi -t Functional.hello_world
> 	> Running test 'Functional.hello_world' on board 'rpi' using spec 'default'
> 	> ===== doing fuego phase: pre_test =====
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 
> 	From the frequency of these, this looks like a host-side problem (and not a target-side
> 	problem).  However, I could be wrong.
> 
> 	Can you do the following, and send me the results:
> 	(inside the container):
> 
> 	ls -l /etc/environment ; cat /etc/environment
> 	ls -l /etc/locale.gen ; cat /etc/locale.gen
> 	ls -l /etc/locale.conf ; cat /etc/locale.conf
> 
> 	Mine are basically empty (except for the JENKINS_PORT) in my Docker container.
> 
> 	Please try one of the fixes below (inside the Docker container):
> 	fix 1:
> 	perl -pi -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/g' /etc/locale.gen
> 	locale-gen en_US.UTF-8
> 	update-locale en_US.UTF-8
> 
> 	fix 2:
> 	Add support for en_US.UTF-8 inside the container.
> 
> 	echo "LC_ALL=en_US.UTF-8" >> /etc/environment
> 	echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
> 	echo "LANG=en_US.UTF-8" > /etc/locale.conf
> 	locale-gen en_US.UTF-8
> 
> 
> 
> This worked for me.
> 
> 
> 	(if you don't have locale-gen in the container, try "apt-get install locales")
> 
> 	If this doesn't fix it, try these steps on the raspberry pi board.
> 
> 	Please report back which of these fixes (if any) worked.
> 
> 	If it still doesn't fix it, then please get a full debug log of the run, and
> 	send that.
> 	$ ftc --debug run-test -b rpi -t hello_world
> 
> 	Note that you don't need to use the full test name "Functional.hello_world",
> 	but only the shortened name.
> 
> 
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> ===== doing fuego phase: build =====
> 	> /fuego-core/scripts/functions.sh: line 403: /fuego-rw/buildzone/rpi.default.Functional.hello_world-/fuego.build.lock: Permission
> denied
> 
> 	This sounds like it was from a test that failed during the build phase and left
> 	its lock file hanging around.
> 
> 	Try just removing it:
> 	rm /fuego-rw/buildzone/rpi.default.Functional.hello_world-/fuego.build.lock
> 
> 	Also, there should be a toolchain name in the directory path for the build directory.
> 	Does your rpi.board file specify a TOOLCHAIN setting?
> 	# cat /fuego-rw/boards/rpi3-2.board | grep TOOL
> 	TOOLCHAIN="debian-armhf"
> 
> 	(You can also use ftc to get this:
> 	 # ftc query-board -b rpi3-2 -n TOOLCHAIN
> 	debian-armhf
> 
> 	It looks like there's a bug in fuego-core/scripts/common.sh where the TOOLCHAIN
> 	is not set before it is used to set the value of JOB_BUILD_DIR.  I made a fix and committed it.
> 
> 	I'm not sure what change I made that would have affect this.
> 
> 	> In Fuego signal_handler (due to error in build phase)
> 	> ##### doing fuego phase: post_test (from signal handler) #####
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> scp: /home/pi/test/fuego.Functional.hello_world/Functional.hello_world.log: No such file or directory
> 
> 	I'd like to see the full log to diagnose by the test log is not being created on the target.
> 	Please run the test with '--debug' and send the log.
> 
> 	> INFO: the test did not produce a test log on the target
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> 	> ##### doing fuego phase: processing (from signal handler) #####
> 	> ### WARNING: Program returned exit code ''
> 	> ### WARNING: Log evaluation may be invalid
> 	> !!! ERROR: ERROR: log_compare: pattern 'SUCCESS' found 0 times (expected greater or equal than 1)
> 	> ERROR: test_processing returned an error
> 	> chart config not found. Using default values.
> 	> ERROR: results did not satisfy the threshold
> 	> Traceback (most recent call last):
> 	>   File "/usr/local/bin/ftc", line 6162, in <module>
> 	>     main()
> 	>   File "/usr/local/bin/ftc", line 6051, in main
> 	>     rcode, run_id = do_run_test(conf, options)
> 	>   File "/usr/local/bin/ftc", line 4710, in do_run_test
> 	>     os.symlink(log_filename, jenkins_log_filename)
> 	> OSError: [Errno 17] File exists
> 
> 	This looks like an issue with there being an already existing run for a test.
> 	If so, I need to handle this issue better (or fail more gracefully).
> 
> 	Can you do the following, inside the container:
> 
> 	 $ ls -l /fuego-rw/logs/Functional.hello_world
> 	and
> 	 $ ls -l /var/lib/jenkins/jobs/*hello_world*/builds
> 
> 	and send me the results?
> 
> 	Sometimes this issue clears itself up as you execute more jobs
> 	(from either Jenkins or using ftc on the command line).
> 
> 	However, ftc is supposed to catch this and avoid this error.
> 
> 	I added some code to try to adjust the build number better
> 	depending on the state of both Jenkins build numbers and
> 	Fuego log (/run) numbers.
> 
> 	Please do a 'git pull' on the fuego-core  repository, and try again.
> 
> 	Thanks very much for the bug report.
> 	 -- Tim
> 
> 
> 
> 
> This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient, you
> are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly
> prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be
> secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The
> sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail
> transmission.