Re: [Fuego] Problem with /fuego-rw/tests in your lab
<[email protected]> Tue, 16 Mar 2021 17:48:36 +0000
| Newsgroups | dev.linux.lists.fuego |
|---|---|
| Message-ID | <BYAPR13MB250336909DDA64A90804A6D6FD6B9@BYAPR13MB2503.namprd13.prod.outlook.com> |
OK. thanks for this update. This is very useful. Based on your experience here, I believe I should change the logic for test selection for 'ftc run-test'. Right now, if the test is defined in both /fuego-rw/tests and /fuego-core/tests, then Fuego runs the one from /fuego-rw/tests. However, I think that it would be more correct to run the one with the latest version (if the test request specifies 'any' or 'latest', for the request test version). Looking at the code, it appears I'll have to do some refactoring to support this behavior. I try to work on that this week. Thanks again for the report. -- Tim > -----Original Message----- > From: Pavan Arun Deshpande <[email protected]> > > Hi Tim, > > Pulled latest fuego-core sources from the repository > > fuego_lab_check in fuego-rw/tests directory belonged to an older version and it was picking that for testing as ftc checks for its presence in > fuego-rw/tests directory. > > when I removed fuego_lab_check from fuego-rw/tests directory and ran the fuego_lab_check , then it ran the testcase from fuego- > core/tests directory which is of the latest version. > > Please find a copy of log messages generated while running Functional.fuego_lab_check. > > ===== doing fuego phase: pre_test ===== > ===== doing fuego phase: build ===== > The test is already built > Fuego test_build duration=0 seconds > ===== doing fuego phase: deploy ===== > ===== doing fuego phase: snapshot ===== > Firmware revision: 5.4.83-v8-93349cdff-2+ > ===== doing fuego phase: run ===== > ------------------------------------------------- > TAP version 13 > 1..1 > # Version of this test=1.4-2 > # Version of this test (hardcoded)=1.4-2 > # Location of this test=/fuego-core/tests/Functional.fuego_lab_check > ######### Process list of docker container: > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > root 1 0.0 0.0 19456 3836 pts/0 Ss+ Mar15 0:00 /bin/bash > jenkins 58 0.0 0.0 18708 0 ? S Mar15 0:00 /usr/bin/daemon --name=jenkins --inherit --env=JENKINS_HOME=/var/lib/jenkins > --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Djava.awt.headless=true - > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego > jenkins 60 0.7 8.7 9950392 1422876 ? Sl Mar15 10:01 /usr/bin/java -Djava.awt.headless=true - > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego > root 110 0.0 0.0 9588 0 ? Ss Mar15 0:00 /usr/bin/netserver > root 129 0.0 0.0 9440 8 ? Ss Mar15 0:00 iperf3 -V -s -D -f M > jenkins 176 0.5 1.0 8487928 166808 ? Sl Mar15 7:09 java -jar /fuego-core/slave.jar > jenkins 177 0.5 1.1 8425932 190836 ? Sl Mar15 7:13 java -jar /fuego-core/slave.jar > jenkins 13870 0.9 0.3 99016 55856 ? S 10:50 0:00 /usr/bin/python /usr/local/bin/ftc run-request --put-run --allow-upgrade > request-2021-03-16_10:48:31.97-pptlab:rpi > jenkins 13873 0.4 0.0 12416 4248 ? S 10:50 0:00 /bin/bash -e /fuego-core/scripts/main.sh > jenkins 13962 0.0 0.0 11192 3008 ? S 10:50 0:00 bash -c { ps -aux; echo $? > /tmp/13873-4538 ; } 2>&1 | tee -a /fuego- > rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt > jenkins 13963 0.0 0.0 11192 272 ? S 10:50 0:00 bash -c { ps -aux; echo $? > /tmp/13873-4538 ; } 2>&1 | tee -a /fuego- > rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt > jenkins 13964 0.0 0.0 5832 708 ? S 10:50 0:00 tee -a /fuego-rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt > jenkins 13965 0.0 0.0 38304 3368 ? R 10:50 0:00 ps -aux > jenkins 18240 0.0 0.0 11180 2960 ? S 02:05 0:00 /bin/sh -xe /tmp/jenkins4864556993423806461.sh > jenkins 18243 0.0 0.0 11256 3104 ? S 02:05 0:12 /bin/bash /fuego-core/scripts/poll_requests.sh -u -d 12h -w 120 > ok 1 - process list of docker container > ######### git describe of fuego-core > v1.5.9-16-gcb0dd1f > ok 2 - git describe of fuego-core > ######### Installed tests - outside of fuego-core > ==== ls -lad /fuego-rw > drwxr-xr-x 8 root root 4096 Mar 16 06:42 /fuego-rw > ==== ls -lad /fuego-rw/tests > drwxr-xr-x 2 root root 4096 Mar 16 06:58 /fuego-rw/tests > ==== ls -laR /fuego-rw/tests > /fuego-rw/tests: > total 8 > drwxr-xr-x 2 root root 4096 Mar 16 06:58 . > drwxr-xr-x 8 root root 4096 Mar 16 06:42 .. > ==== ls -lad /fuego-core/tests > drwxr-xr-x 197 root root 12288 Jan 12 10:17 /fuego-core/tests > ok 3 - remotely installed tests > ######### Console log of the last run > # last run=/fuego-rw/logs/Functional.hello_world/MINNOW.default.10.10 > # Log: > # ===== doing fuego phase: pre_test ===== > # ===== doing fuego phase: build ===== > # The test is already built > # Fuego test_build duration=0 seconds > # ===== doing fuego phase: deploy ===== > # ===== doing fuego phase: snapshot ===== > # Firmware revision: 5.4.81-intel-pk-standard > # ===== doing fuego phase: run ===== > # ------------------------------------------------- > # Hello World! > # > # This is a test program, to demonstrate test success and failure > # Test result: SUCCESS > # ------------------------------------------------- > # ===== doing fuego phase: post_test ===== > # ===== doing fuego phase: processing ===== > # log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1) > # chart config not found. Using default values. > # Fuego: requested test phases complete! > ######### end of log > ok 4 - console log of last test run > ------------------------------------------------- > ===== doing fuego phase: post_test ===== > scp: /home/pi/test/fuego.Functional.fuego_lab_check/Functional.fuego_lab_check.log: No such file or directory > INFO: the test did not produce a test log on the target > ===== doing fuego phase: processing ===== > chart config not found. Using default values. > Fuego: requested test phases complete! > Packaging run 'Functional.fuego_lab_check-default-11-rpi' > Run packaged successfully, and is at: /tmp/run-Functional.fuego_lab_check-default-11-on-pptlab:rpi.frp > Run package run-Functional.fuego_lab_check-default-11-on-pptlab:rpi.frp was accepted by the server. > Run Functional.fuego_lab_check-default-11-rpi put to server > > Thanks and regards > > Pavan Arun Deshpande > > > > On Mon, Mar 15, 2021 at 8:56 PM Srivatsan S <[email protected] <mailto:[email protected]> > wrote: > > > Tim, > > We couldn't do a git pull today to get the latest changes as we had some local changes to get the raspi_kinstall working. Currently, > we have moved to a 64-bit kernel image for rpi board and have made some local changes to the script in Functional.raspi_kinstall to get it > working with a 64-bit kernel image. > > We will do a git pull tomorrow and run our changes on top of it and get back to you. > > Thanks, > Srivatsan > > > On Mon, Mar 15, 2021 at 8:23 PM <[email protected] <mailto:[email protected]> > wrote: > > > This is good to hear. Thanks for getting this resolved. > The errors that Fuego was returning were not very useful > for determining this problem. I'd like to think about how > this condition could be checked for, before or during test > execution, and reported back with a more useful error message. > > > # Version of this test=1.2-4 > It looks like you are still running an old version of fuego_lab_check. > Can you please 'git pull' the latest master branch of fuego-core? > > I plan to do more testing this week. > > Thanks. > -- Tim > > > > -----Original Message----- > > From: Pavan Arun Deshpande <[email protected] <mailto:[email protected]> > > > > > Hi Tim, > > > > The issue regarding Functional.fuego_lab_check has been resolved. > > The issue was due to the overlapping build numbers as concerned files were already present in the container. > > Therefore the server was giving an error message saying that " Already have a file cannot proceed". > > Uploading failed because of the above mentioned issue. > > I was able to resolve this after removing the previous run files. > > > > Functional.reboot also had the same issue and this also has been resolved. > > > > This has been verified with multiple tests and iterations. > > > > Please find a copy of log messages generated while running Functional.fuego_lab_check > > > > ===== doing fuego phase: pre_test ===== > > ===== doing fuego phase: build ===== > > The test is already built > > Fuego test_build duration=0 seconds > > ===== doing fuego phase: deploy ===== > > ===== doing fuego phase: snapshot ===== > > Firmware revision: 5.4.83-v8-93349cdff-1+ > > ===== doing fuego phase: run ===== > > ------------------------------------------------- > > TAP version 13 > > 1..1 > > # Version of this test=1.2-4 > > # Location of this test=/fuego-rw/tests/Functional.fuego_lab_check > > ######### Process list of docker container: > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 1 0.0 0.0 19424 4544 pts/0 Ss+ 11:25 0:00 /bin/bash > > jenkins 58 0.0 0.0 18708 184 ? S 11:25 0:00 /usr/bin/daemon --name=jenkins --inherit -- > env=JENKINS_HOME=/var/lib/jenkins > > --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Djava.awt.headless=true - > > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar > > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego > > jenkins 60 2.4 10.8 9946068 1769860 ? Sl 11:25 1:31 /usr/bin/java -Djava.awt.headless=true - > > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar > > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego > > root 110 0.0 0.0 9588 144 ? Ss 11:25 0:00 /usr/bin/netserver > > root 129 0.0 0.0 9440 96 ? Ss 11:25 0:00 iperf3 -V -s -D -f M > > jenkins 176 0.6 0.9 8487928 152384 ? Sl 11:25 0:23 java -jar /fuego-core/slave.jar > > jenkins 177 0.6 0.9 8421364 150824 ? Sl 11:25 0:24 java -jar /fuego-core/slave.jar > > jenkins 336 0.0 0.0 11180 2852 ? S 11:27 0:00 /bin/sh -xe /tmp/jenkins184800571287563138.sh > > jenkins 340 0.0 0.0 11256 3104 ? S 11:27 0:01 /bin/bash /fuego-core/scripts/poll_requests.sh -u -d 12h -w > 120 > > jenkins 4769 0.6 0.3 99016 56048 ? S 12:28 0:00 /usr/bin/python /usr/local/bin/ftc run-request --put-run -- > allow-upgrade > > request-2021-03-15_12:27:36.07-pptlab:rpi > > jenkins 4772 0.3 0.0 12404 4168 ? S 12:28 0:00 /bin/bash -e /fuego-core/scripts/main.sh > > jenkins 4855 0.0 0.0 11192 2848 ? S 12:28 0:00 bash -c { ps -aux; echo $? > /tmp/4772-7340 ; } 2>&1 | tee - > a /fuego- > > rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt > > jenkins 4856 0.0 0.0 11192 272 ? S 12:28 0:00 bash -c { ps -aux; echo $? > /tmp/4772-7340 ; } 2>&1 | tee -a > /fuego- > > rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt > > jenkins 4857 0.0 0.0 5832 656 ? S 12:28 0:00 tee -a /fuego- > rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt > > jenkins 4858 0.0 0.0 38304 3316 ? R 12:28 0:00 ps -aux > > ok 1 - process list of docker container > > ######### git describe of fuego-core > > v1.5.9-9-g79f9df5 > > ok 2 - git describe of fuego-core > > ######### Installed tests - outside of fuego-core > > total 4 > > drwxr-xr-x 2 root root 4096 Jan 29 04:40 Functional.fuego_lab_check > > ok 3 - remotely installed tests > > ######### Console log of the last run > > # last run=/fuego-rw/logs/Functional.fuego_board_check/rpi.default.91.91 > > # Log: > > # ===== doing fuego phase: pre_test ===== > > # ===== doing fuego phase: build ===== > > # The test is already built > > # Fuego test_build duration=0 seconds > > # ===== doing fuego phase: deploy ===== > > # ===== doing fuego phase: snapshot ===== > > # Firmware revision: 5.4.83-v8-93349cdff-1+ > > # ===== doing fuego phase: run ===== > > # ------------------------------------------------- > > # == Fuego target feature scan results == > > # Information: > > # account information: uid=0(root) gid=0(root) groups=0(root) > > # working directory: /home/pi/test/fuego.Functional.fuego_board_check > > # PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin > > # ------------------------------------- > > # Scan results: > > # HAS_PROGRAM_CAT=1 > > # HAS_PROGRAM_DF=1 > > # HAS_PROGRAM_FIND=1 > > # HAS_PROGRAM_FREE=1 > > # HAS_PROGRAM_GREP=1 > > # HAS_PROGRAM_HEAD=1 > > # HAS_PROGRAM_LOGGER=1 > > # HAS_PROGRAM_LOGREAD=0 > > # HAS_PROGRAM_MKDIR=1 > > # HAS_PROGRAM_MOUNT=1 > > # HAS_PROGRAM_PS=1 > > # HAS_PROGRAM_RM=1 > > # HAS_PROGRAM_RMDIR=1 > > # HAS_PROGRAM_SYNC=1 > > # HAS_PROGRAM_TAIL=1 > > # HAS_PROGRAM_TEE=1 > > # HAS_PROGRAM_TOUCH=1 > > # HAS_PROGRAM_TRUE=1 > > # HAS_PROGRAM_UMOUNT=1 > > # HAS_PROGRAM_UNAME=1 > > # HAS_PROGRAM_UPTIME=1 > > # HAS_PROGRAM_XARGS=1 > > # HAS_PROGRAM_BRACKET=1 > > # HAS_MKDIR_P=1 > > # HAS_RM_RF=1 > > # HAS_GREP_F=1 > > # HAS_SBIN_REBOOT=1 > > # HAS_BIN_TRUE=1 > > # HAS_SBIN_ROUTE=1 > > # HAS_VAR_LOG_MESSAGES=0 > > # HAS_VAR_LOG_SYSLOG=1 > > # HAS_PROC_INTERRUPTS=1 > > # HAS_PROC_DROP_CACHES=1 > > # HAS_PROC_OOM_SCORE_ADJ=0 > > # HAS_PROC_CONFIG_GZ=0 > > # HAS_MEMORY=935532 K > > # HAS_STORAGE=15022716 K > > # Process 1 is program: /sbin/init > > # ------------------------------------- > > # Based on this scan, > > # the recommended DISTRIB value should be: > > # nosyslogd.dist > > # ------------------------------------- > > # RESULT=SUCCESS > > # All required programs, files, features, and permissions are present > > # ------------------------------------- > > # 10+0 records in > > # 10+0 records out > > # 10000 bytes (10 kB, 9.8 KiB) copied, 0.000395734 s, 25.3 MB/s > > # Data from host: > > # TRANSPORT=ssh > > # Time to put 10K file: 0.68 > > # Time to get 10K file: 0.69 > > # Time to do 10 cmds: 6.74 > > # Average time per cmd: 0.67 > > # ------------------------------------------------- > > # ===== doing fuego phase: post_test ===== > > # ===== doing fuego phase: processing ===== > > # log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1) > > # chart config not found. Using default values. > > # Fuego: requested test phases complete! > > ######### end of log > > ok 4 - console log of last test run > > ------------------------------------------------- > > ===== doing fuego phase: post_test ===== > > scp: /home/pi/test/fuego.Functional.fuego_lab_check/Functional.fuego_lab_check.log: No such file or directory > > INFO: the test did not produce a test log on the target > > ===== doing fuego phase: processing ===== > > chart config not found. Using default values. > > Fuego: requested test phases complete! > > Packaging run 'Functional.fuego_lab_check-default-10-rpi' > > Run packaged successfully, and is at: /tmp/run-Functional.fuego_lab_check-default-10-on-pptlab:rpi.frp > > Run package run-Functional.fuego_lab_check-default-10-on-pptlab:rpi.frp was accepted by the server. > > Run Functional.fuego_lab_check-default-10-rpi put to server > > > > Thanks and regards > > > > Pavan Arun deshpande > > > > > > On Sat, Mar 13, 2021 at 3:54 AM <[email protected] <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]> > > wrote: > > > > > > There is a problem with running the test: Functional.fuego_lab_check > > in your lab. I'll send the error in another message. > > > > Here is the error from the testlog: > > > > ls: cannot access '/fuego-rw/tests': No such file or directory > > ---- > > > > And here is the report from the console log: > > > > ######### Installed tests (outside of fuego-core) > > ls: cannot access '/fuego-rw/tests': No such file or directory > > In Fuego signal_handler (due to error in run phase) > > ##### doing fuego phase: post_test (from signal handler) ##### > > ... > > ----- > > > > This output appears to be from an older version of Functional.fuego_lab_check. > > One of the things I am testing with this test is the ability to update the version > > of a test in your lab. Since there's a problem with the /fuego-rw/tests directory, > > this operation is not working. > > > > Can you please do 3 things: > > 1) please check your docker container and make sure that it has the directory > > /fuego-rw/tests > > and that this directory is owned by user jenkins and group jenkins? > > > > 2) can you 'git pull' the latest fuego-core from the master repository, to pick > > up the latest version of the Functional.fuego_lab_check test? > > > > Please make sure that you can run this test locally (from the /fuego-core/tests > > directory). That is, just run this test from the updated fuego-core repository > > in your lab, and confirm that it runs without error. > > > > Once you have done that, I will try some tests to see if I can cause your lab > > to run upgraded tests. > > > > Thanks, > > -- Tim > > > > > > > > > > > > This message contains confidential information and is intended only for the individual(s) named. If you are not the > intended recipient, you > > are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and > attached file/s is strictly > > prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be > guaranteed to be > > secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or > contain viruses. The > > sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a > result of e-mail > > transmission. > > > > > This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient, you > are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly > prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be > secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The > sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail > transmission.