Re: [Fuego] Problem with /fuego-rw/tests in your lab
<[email protected]> Tue, 16 Mar 2021 17:52:22 +0000
| Newsgroups | dev.linux.lists.fuego |
|---|---|
| Message-ID | <BYAPR13MB25039B7F12517D5964966B3AFD6B9@BYAPR13MB2503.namprd13.prod.outlook.com> |
Actually a slight correction below... > -----Original Message----- > From: Fuego <[email protected]> On Behalf Of [email protected] > > OK. thanks for this update. This is very useful. > > Based on your experience here, I believe I should change the logic for test selection > for 'ftc run-test'. That should be: "for 'ftc run-request'". > Right now, if the test is defined in both /fuego-rw/tests and /fuego-core/tests, > then Fuego runs the one from /fuego-rw/tests. However, I think that it would be more > correct to run the one with the latest version (if the test request specifies 'any' or 'latest', for > the request test version). > > Looking at the code, it appears I'll have to do some refactoring to support this behavior. > > I try to work on that this week. Thanks again for the report. > -- Tim > > > -----Original Message----- > > From: Pavan Arun Deshpande <[email protected]> > > > > Hi Tim, > > > > Pulled latest fuego-core sources from the repository > > > > fuego_lab_check in fuego-rw/tests directory belonged to an older version and it was picking that for testing as ftc checks for its presence > in > > fuego-rw/tests directory. > > > > when I removed fuego_lab_check from fuego-rw/tests directory and ran the fuego_lab_check , then it ran the testcase from fuego- > > core/tests directory which is of the latest version. > > > > Please find a copy of log messages generated while running Functional.fuego_lab_check. > > > > ===== doing fuego phase: pre_test ===== > > ===== doing fuego phase: build ===== > > The test is already built > > Fuego test_build duration=0 seconds > > ===== doing fuego phase: deploy ===== > > ===== doing fuego phase: snapshot ===== > > Firmware revision: 5.4.83-v8-93349cdff-2+ > > ===== doing fuego phase: run ===== > > ------------------------------------------------- > > TAP version 13 > > 1..1 > > # Version of this test=1.4-2 > > # Version of this test (hardcoded)=1.4-2 > > # Location of this test=/fuego-core/tests/Functional.fuego_lab_check > > ######### Process list of docker container: > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > root 1 0.0 0.0 19456 3836 pts/0 Ss+ Mar15 0:00 /bin/bash > > jenkins 58 0.0 0.0 18708 0 ? S Mar15 0:00 /usr/bin/daemon --name=jenkins --inherit -- > env=JENKINS_HOME=/var/lib/jenkins > > --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Djava.awt.headless=true - > > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar > > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego > > jenkins 60 0.7 8.7 9950392 1422876 ? Sl Mar15 10:01 /usr/bin/java -Djava.awt.headless=true - > > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar > > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego > > root 110 0.0 0.0 9588 0 ? Ss Mar15 0:00 /usr/bin/netserver > > root 129 0.0 0.0 9440 8 ? Ss Mar15 0:00 iperf3 -V -s -D -f M > > jenkins 176 0.5 1.0 8487928 166808 ? Sl Mar15 7:09 java -jar /fuego-core/slave.jar > > jenkins 177 0.5 1.1 8425932 190836 ? Sl Mar15 7:13 java -jar /fuego-core/slave.jar > > jenkins 13870 0.9 0.3 99016 55856 ? S 10:50 0:00 /usr/bin/python /usr/local/bin/ftc run-request --put-run --allow-upgrade > > request-2021-03-16_10:48:31.97-pptlab:rpi > > jenkins 13873 0.4 0.0 12416 4248 ? S 10:50 0:00 /bin/bash -e /fuego-core/scripts/main.sh > > jenkins 13962 0.0 0.0 11192 3008 ? S 10:50 0:00 bash -c { ps -aux; echo $? > /tmp/13873-4538 ; } 2>&1 | tee -a /fuego- > > rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt > > jenkins 13963 0.0 0.0 11192 272 ? S 10:50 0:00 bash -c { ps -aux; echo $? > /tmp/13873-4538 ; } 2>&1 | tee -a /fuego- > > rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt > > jenkins 13964 0.0 0.0 5832 708 ? S 10:50 0:00 tee -a /fuego- > rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt > > jenkins 13965 0.0 0.0 38304 3368 ? R 10:50 0:00 ps -aux > > jenkins 18240 0.0 0.0 11180 2960 ? S 02:05 0:00 /bin/sh -xe /tmp/jenkins4864556993423806461.sh > > jenkins 18243 0.0 0.0 11256 3104 ? S 02:05 0:12 /bin/bash /fuego-core/scripts/poll_requests.sh -u -d 12h -w 120 > > ok 1 - process list of docker container > > ######### git describe of fuego-core > > v1.5.9-16-gcb0dd1f > > ok 2 - git describe of fuego-core > > ######### Installed tests - outside of fuego-core > > ==== ls -lad /fuego-rw > > drwxr-xr-x 8 root root 4096 Mar 16 06:42 /fuego-rw > > ==== ls -lad /fuego-rw/tests > > drwxr-xr-x 2 root root 4096 Mar 16 06:58 /fuego-rw/tests > > ==== ls -laR /fuego-rw/tests > > /fuego-rw/tests: > > total 8 > > drwxr-xr-x 2 root root 4096 Mar 16 06:58 . > > drwxr-xr-x 8 root root 4096 Mar 16 06:42 .. > > ==== ls -lad /fuego-core/tests > > drwxr-xr-x 197 root root 12288 Jan 12 10:17 /fuego-core/tests > > ok 3 - remotely installed tests > > ######### Console log of the last run > > # last run=/fuego-rw/logs/Functional.hello_world/MINNOW.default.10.10 > > # Log: > > # ===== doing fuego phase: pre_test ===== > > # ===== doing fuego phase: build ===== > > # The test is already built > > # Fuego test_build duration=0 seconds > > # ===== doing fuego phase: deploy ===== > > # ===== doing fuego phase: snapshot ===== > > # Firmware revision: 5.4.81-intel-pk-standard > > # ===== doing fuego phase: run ===== > > # ------------------------------------------------- > > # Hello World! > > # > > # This is a test program, to demonstrate test success and failure > > # Test result: SUCCESS > > # ------------------------------------------------- > > # ===== doing fuego phase: post_test ===== > > # ===== doing fuego phase: processing ===== > > # log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1) > > # chart config not found. Using default values. > > # Fuego: requested test phases complete! > > ######### end of log > > ok 4 - console log of last test run > > ------------------------------------------------- > > ===== doing fuego phase: post_test ===== > > scp: /home/pi/test/fuego.Functional.fuego_lab_check/Functional.fuego_lab_check.log: No such file or directory > > INFO: the test did not produce a test log on the target > > ===== doing fuego phase: processing ===== > > chart config not found. Using default values. > > Fuego: requested test phases complete! > > Packaging run 'Functional.fuego_lab_check-default-11-rpi' > > Run packaged successfully, and is at: /tmp/run-Functional.fuego_lab_check-default-11-on-pptlab:rpi.frp > > Run package run-Functional.fuego_lab_check-default-11-on-pptlab:rpi.frp was accepted by the server. > > Run Functional.fuego_lab_check-default-11-rpi put to server > > > > Thanks and regards > > > > Pavan Arun Deshpande > > > > > > > > On Mon, Mar 15, 2021 at 8:56 PM Srivatsan S <[email protected] <mailto:[email protected]> > wrote: > > > > > > Tim, > > > > We couldn't do a git pull today to get the latest changes as we had some local changes to get the raspi_kinstall working. Currently, > > we have moved to a 64-bit kernel image for rpi board and have made some local changes to the script in Functional.raspi_kinstall to get > it > > working with a 64-bit kernel image. > > > > We will do a git pull tomorrow and run our changes on top of it and get back to you. > > > > Thanks, > > Srivatsan > > > > > > On Mon, Mar 15, 2021 at 8:23 PM <[email protected] <mailto:[email protected]> > wrote: > > > > > > This is good to hear. Thanks for getting this resolved. > > The errors that Fuego was returning were not very useful > > for determining this problem. I'd like to think about how > > this condition could be checked for, before or during test > > execution, and reported back with a more useful error message. > > > > > # Version of this test=1.2-4 > > It looks like you are still running an old version of fuego_lab_check. > > Can you please 'git pull' the latest master branch of fuego-core? > > > > I plan to do more testing this week. > > > > Thanks. > > -- Tim > > > > > > > -----Original Message----- > > > From: Pavan Arun Deshpande <[email protected] <mailto:[email protected]> > > > > > > > Hi Tim, > > > > > > The issue regarding Functional.fuego_lab_check has been resolved. > > > The issue was due to the overlapping build numbers as concerned files were already present in the container. > > > Therefore the server was giving an error message saying that " Already have a file cannot proceed". > > > Uploading failed because of the above mentioned issue. > > > I was able to resolve this after removing the previous run files. > > > > > > Functional.reboot also had the same issue and this also has been resolved. > > > > > > This has been verified with multiple tests and iterations. > > > > > > Please find a copy of log messages generated while running Functional.fuego_lab_check > > > > > > ===== doing fuego phase: pre_test ===== > > > ===== doing fuego phase: build ===== > > > The test is already built > > > Fuego test_build duration=0 seconds > > > ===== doing fuego phase: deploy ===== > > > ===== doing fuego phase: snapshot ===== > > > Firmware revision: 5.4.83-v8-93349cdff-1+ > > > ===== doing fuego phase: run ===== > > > ------------------------------------------------- > > > TAP version 13 > > > 1..1 > > > # Version of this test=1.2-4 > > > # Location of this test=/fuego-rw/tests/Functional.fuego_lab_check > > > ######### Process list of docker container: > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > > root 1 0.0 0.0 19424 4544 pts/0 Ss+ 11:25 0:00 /bin/bash > > > jenkins 58 0.0 0.0 18708 184 ? S 11:25 0:00 /usr/bin/daemon --name=jenkins --inherit -- > > env=JENKINS_HOME=/var/lib/jenkins > > > --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Djava.awt.headless=true - > > > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar > > > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego > > > jenkins 60 2.4 10.8 9946068 1769860 ? Sl 11:25 1:31 /usr/bin/java -Djava.awt.headless=true - > > > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar > > > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego > > > root 110 0.0 0.0 9588 144 ? Ss 11:25 0:00 /usr/bin/netserver > > > root 129 0.0 0.0 9440 96 ? Ss 11:25 0:00 iperf3 -V -s -D -f M > > > jenkins 176 0.6 0.9 8487928 152384 ? Sl 11:25 0:23 java -jar /fuego-core/slave.jar > > > jenkins 177 0.6 0.9 8421364 150824 ? Sl 11:25 0:24 java -jar /fuego-core/slave.jar > > > jenkins 336 0.0 0.0 11180 2852 ? S 11:27 0:00 /bin/sh -xe /tmp/jenkins184800571287563138.sh > > > jenkins 340 0.0 0.0 11256 3104 ? S 11:27 0:01 /bin/bash /fuego-core/scripts/poll_requests.sh -u -d 12h -w > > 120 > > > jenkins 4769 0.6 0.3 99016 56048 ? S 12:28 0:00 /usr/bin/python /usr/local/bin/ftc run-request --put-run -- > > allow-upgrade > > > request-2021-03-15_12:27:36.07-pptlab:rpi > > > jenkins 4772 0.3 0.0 12404 4168 ? S 12:28 0:00 /bin/bash -e /fuego-core/scripts/main.sh > > > jenkins 4855 0.0 0.0 11192 2848 ? S 12:28 0:00 bash -c { ps -aux; echo $? > /tmp/4772-7340 ; } 2>&1 | tee - > > a /fuego- > > > rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt > > > jenkins 4856 0.0 0.0 11192 272 ? S 12:28 0:00 bash -c { ps -aux; echo $? > /tmp/4772-7340 ; } 2>&1 | tee -a > > /fuego- > > > rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt > > > jenkins 4857 0.0 0.0 5832 656 ? S 12:28 0:00 tee -a /fuego- > > rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt > > > jenkins 4858 0.0 0.0 38304 3316 ? R 12:28 0:00 ps -aux > > > ok 1 - process list of docker container > > > ######### git describe of fuego-core > > > v1.5.9-9-g79f9df5 > > > ok 2 - git describe of fuego-core > > > ######### Installed tests - outside of fuego-core > > > total 4 > > > drwxr-xr-x 2 root root 4096 Jan 29 04:40 Functional.fuego_lab_check > > > ok 3 - remotely installed tests > > > ######### Console log of the last run > > > # last run=/fuego-rw/logs/Functional.fuego_board_check/rpi.default.91.91 > > > # Log: > > > # ===== doing fuego phase: pre_test ===== > > > # ===== doing fuego phase: build ===== > > > # The test is already built > > > # Fuego test_build duration=0 seconds > > > # ===== doing fuego phase: deploy ===== > > > # ===== doing fuego phase: snapshot ===== > > > # Firmware revision: 5.4.83-v8-93349cdff-1+ > > > # ===== doing fuego phase: run ===== > > > # ------------------------------------------------- > > > # == Fuego target feature scan results == > > > # Information: > > > # account information: uid=0(root) gid=0(root) groups=0(root) > > > # working directory: /home/pi/test/fuego.Functional.fuego_board_check > > > # PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin > > > # ------------------------------------- > > > # Scan results: > > > # HAS_PROGRAM_CAT=1 > > > # HAS_PROGRAM_DF=1 > > > # HAS_PROGRAM_FIND=1 > > > # HAS_PROGRAM_FREE=1 > > > # HAS_PROGRAM_GREP=1 > > > # HAS_PROGRAM_HEAD=1 > > > # HAS_PROGRAM_LOGGER=1 > > > # HAS_PROGRAM_LOGREAD=0 > > > # HAS_PROGRAM_MKDIR=1 > > > # HAS_PROGRAM_MOUNT=1 > > > # HAS_PROGRAM_PS=1 > > > # HAS_PROGRAM_RM=1 > > > # HAS_PROGRAM_RMDIR=1 > > > # HAS_PROGRAM_SYNC=1 > > > # HAS_PROGRAM_TAIL=1 > > > # HAS_PROGRAM_TEE=1 > > > # HAS_PROGRAM_TOUCH=1 > > > # HAS_PROGRAM_TRUE=1 > > > # HAS_PROGRAM_UMOUNT=1 > > > # HAS_PROGRAM_UNAME=1 > > > # HAS_PROGRAM_UPTIME=1 > > > # HAS_PROGRAM_XARGS=1 > > > # HAS_PROGRAM_BRACKET=1 > > > # HAS_MKDIR_P=1 > > > # HAS_RM_RF=1 > > > # HAS_GREP_F=1 > > > # HAS_SBIN_REBOOT=1 > > > # HAS_BIN_TRUE=1 > > > # HAS_SBIN_ROUTE=1 > > > # HAS_VAR_LOG_MESSAGES=0 > > > # HAS_VAR_LOG_SYSLOG=1 > > > # HAS_PROC_INTERRUPTS=1 > > > # HAS_PROC_DROP_CACHES=1 > > > # HAS_PROC_OOM_SCORE_ADJ=0 > > > # HAS_PROC_CONFIG_GZ=0 > > > # HAS_MEMORY=935532 K > > > # HAS_STORAGE=15022716 K > > > # Process 1 is program: /sbin/init > > > # ------------------------------------- > > > # Based on this scan, > > > # the recommended DISTRIB value should be: > > > # nosyslogd.dist > > > # ------------------------------------- > > > # RESULT=SUCCESS > > > # All required programs, files, features, and permissions are present > > > # ------------------------------------- > > > # 10+0 records in > > > # 10+0 records out > > > # 10000 bytes (10 kB, 9.8 KiB) copied, 0.000395734 s, 25.3 MB/s > > > # Data from host: > > > # TRANSPORT=ssh > > > # Time to put 10K file: 0.68 > > > # Time to get 10K file: 0.69 > > > # Time to do 10 cmds: 6.74 > > > # Average time per cmd: 0.67 > > > # ------------------------------------------------- > > > # ===== doing fuego phase: post_test ===== > > > # ===== doing fuego phase: processing ===== > > > # log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1) > > > # chart config not found. Using default values. > > > # Fuego: requested test phases complete! > > > ######### end of log > > > ok 4 - console log of last test run > > > ------------------------------------------------- > > > ===== doing fuego phase: post_test ===== > > > scp: /home/pi/test/fuego.Functional.fuego_lab_check/Functional.fuego_lab_check.log: No such file or directory > > > INFO: the test did not produce a test log on the target > > > ===== doing fuego phase: processing ===== > > > chart config not found. Using default values. > > > Fuego: requested test phases complete! > > > Packaging run 'Functional.fuego_lab_check-default-10-rpi' > > > Run packaged successfully, and is at: /tmp/run-Functional.fuego_lab_check-default-10-on-pptlab:rpi.frp > > > Run package run-Functional.fuego_lab_check-default-10-on-pptlab:rpi.frp was accepted by the server. > > > Run Functional.fuego_lab_check-default-10-rpi put to server > > > > > > Thanks and regards > > > > > > Pavan Arun deshpande > > > > > > > > > On Sat, Mar 13, 2021 at 3:54 AM <[email protected] <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]> > > wrote: > > > > > > > > > There is a problem with running the test: Functional.fuego_lab_check > > > in your lab. I'll send the error in another message. > > > > > > Here is the error from the testlog: > > > > > > ls: cannot access '/fuego-rw/tests': No such file or directory > > > ---- > > > > > > And here is the report from the console log: > > > > > > ######### Installed tests (outside of fuego-core) > > > ls: cannot access '/fuego-rw/tests': No such file or directory > > > In Fuego signal_handler (due to error in run phase) > > > ##### doing fuego phase: post_test (from signal handler) ##### > > > ... > > > ----- > > > > > > This output appears to be from an older version of Functional.fuego_lab_check. > > > One of the things I am testing with this test is the ability to update the version > > > of a test in your lab. Since there's a problem with the /fuego-rw/tests directory, > > > this operation is not working. > > > > > > Can you please do 3 things: > > > 1) please check your docker container and make sure that it has the directory > > > /fuego-rw/tests > > > and that this directory is owned by user jenkins and group jenkins? > > > > > > 2) can you 'git pull' the latest fuego-core from the master repository, to pick > > > up the latest version of the Functional.fuego_lab_check test? > > > > > > Please make sure that you can run this test locally (from the /fuego-core/tests > > > directory). That is, just run this test from the updated fuego-core repository > > > in your lab, and confirm that it runs without error. > > > > > > Once you have done that, I will try some tests to see if I can cause your lab > > > to run upgraded tests. > > > > > > Thanks, > > > -- Tim > > > > > > > > > > > > > > > > > > This message contains confidential information and is intended only for the individual(s) named. If you are not the > > intended recipient, you > > > are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and > > attached file/s is strictly > > > prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be > > guaranteed to be > > > secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or > > contain viruses. The > > > sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a > > result of e-mail > > > transmission. > > > > > > > > > > This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient, > you > > are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly > > prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to > be > > secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The > > sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail > > transmission. > > _______________________________________________ > Fuego mailing list > [email protected] > https://lists.linuxfoundation.org/mailman/listinfo/fuego