Re: [Fuego] Problem with /fuego-rw/tests in your lab
Pavan Arun Deshpande <[email protected]> Tue, 16 Mar 2021 17:14:15 +0530
| Newsgroups | dev.linux.lists.fuego |
|---|---|
| Message-ID | <CAMQEYMd9N_mpK3RRRdaRs=AJ4cXzVD4Ob0JHQUd9NU_vKdhgsQ@mail.gmail.com> |
Hi Tim,
Pulled latest fuego-core sources from the repository
fuego_lab_check in fuego-rw/tests directory belonged to an older version
and it was picking that for testing as ftc checks for its presence in
fuego-rw/tests directory.
when I removed fuego_lab_check from fuego-rw/tests directory and ran the
fuego_lab_check , then it ran the testcase from fuego-core/tests directory
which is of the latest version.
Please find a copy of log messages generated while running
Functional.fuego_lab_check.
===== doing fuego phase: pre_test =====
===== doing fuego phase: build =====
The test is already built
Fuego test_build duration=0 seconds
===== doing fuego phase: deploy =====
===== doing fuego phase: snapshot =====
Firmware revision: 5.4.83-v8-93349cdff-2+
===== doing fuego phase: run =====
-------------------------------------------------
TAP version 13
1..1
# Version of this test=1.4-2
# Version of this test (hardcoded)=1.4-2
# Location of this test=/fuego-core/tests/Functional.fuego_lab_check
######### Process list of docker container:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 19456 3836 pts/0 Ss+ Mar15 0:00 /bin/bash
jenkins 58 0.0 0.0 18708 0 ? S Mar15 0:00
/usr/bin/daemon --name=jenkins --inherit
--env=JENKINS_HOME=/var/lib/jenkins
--output=/var/log/jenkins/jenkins.log
--pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java
-Djava.awt.headless=true -Djenkins.install.runSetupWizard=false
-Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar
/usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war
--httpPort=8090 --prefix=/fuego
jenkins 60 0.7 8.7 9950392 1422876 ? Sl Mar15 10:01
/usr/bin/java -Djava.awt.headless=true
-Djenkins.install.runSetupWizard=false
-Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar
/usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war
--httpPort=8090 --prefix=/fuego
root 110 0.0 0.0 9588 0 ? Ss Mar15 0:00
/usr/bin/netserver
root 129 0.0 0.0 9440 8 ? Ss Mar15 0:00
iperf3 -V -s -D -f M
jenkins 176 0.5 1.0 8487928 166808 ? Sl Mar15 7:09 java
-jar /fuego-core/slave.jar
jenkins 177 0.5 1.1 8425932 190836 ? Sl Mar15 7:13 java
-jar /fuego-core/slave.jar
jenkins 13870 0.9 0.3 99016 55856 ? S 10:50 0:00
/usr/bin/python /usr/local/bin/ftc run-request --put-run
--allow-upgrade request-2021-03-16_10:48:31.97-pptlab:rpi
jenkins 13873 0.4 0.0 12416 4248 ? S 10:50 0:00
/bin/bash -e /fuego-core/scripts/main.sh
jenkins 13962 0.0 0.0 11192 3008 ? S 10:50 0:00 bash
-c { ps -aux; echo $? > /tmp/13873-4538 ; } 2>&1 | tee -a
/fuego-rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt
jenkins 13963 0.0 0.0 11192 272 ? S 10:50 0:00 bash
-c { ps -aux; echo $? > /tmp/13873-4538 ; } 2>&1 | tee -a
/fuego-rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt
jenkins 13964 0.0 0.0 5832 708 ? S 10:50 0:00 tee
-a /fuego-rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt
jenkins 13965 0.0 0.0 38304 3368 ? R 10:50 0:00 ps -aux
jenkins 18240 0.0 0.0 11180 2960 ? S 02:05 0:00
/bin/sh -xe /tmp/jenkins4864556993423806461.sh
jenkins 18243 0.0 0.0 11256 3104 ? S 02:05 0:12
/bin/bash /fuego-core/scripts/poll_requests.sh -u -d 12h -w 120
ok 1 - process list of docker container
######### git describe of fuego-core
v1.5.9-16-gcb0dd1f
ok 2 - git describe of fuego-core
######### Installed tests - outside of fuego-core
==== ls -lad /fuego-rw
drwxr-xr-x 8 root root 4096 Mar 16 06:42 /fuego-rw
==== ls -lad /fuego-rw/tests
drwxr-xr-x 2 root root 4096 Mar 16 06:58 /fuego-rw/tests
==== ls -laR /fuego-rw/tests
/fuego-rw/tests:
total 8
drwxr-xr-x 2 root root 4096 Mar 16 06:58 .
drwxr-xr-x 8 root root 4096 Mar 16 06:42 ..
==== ls -lad /fuego-core/tests
drwxr-xr-x 197 root root 12288 Jan 12 10:17 /fuego-core/tests
ok 3 - remotely installed tests
######### Console log of the last run
# last run=/fuego-rw/logs/Functional.hello_world/MINNOW.default.10.10
# Log:
# ===== doing fuego phase: pre_test =====
# ===== doing fuego phase: build =====
# The test is already built
# Fuego test_build duration=0 seconds
# ===== doing fuego phase: deploy =====
# ===== doing fuego phase: snapshot =====
# Firmware revision: 5.4.81-intel-pk-standard
# ===== doing fuego phase: run =====
# -------------------------------------------------
# Hello World!
#
# This is a test program, to demonstrate test success and failure
# Test result: SUCCESS
# -------------------------------------------------
# ===== doing fuego phase: post_test =====
# ===== doing fuego phase: processing =====
# log_compare: pattern 'SUCCESS' found 1 times (expected greater or
equal than 1)
# chart config not found. Using default values.
# Fuego: requested test phases complete!
######### end of log
ok 4 - console log of last test run
-------------------------------------------------
===== doing fuego phase: post_test =====
scp: /home/pi/test/fuego.Functional.fuego_lab_check/Functional.fuego_lab_check.log:
No such file or directory
INFO: the test did not produce a test log on the target
===== doing fuego phase: processing =====
chart config not found. Using default values.
Fuego: requested test phases complete!
Packaging run 'Functional.fuego_lab_check-default-11-rpi'
Run packaged successfully, and is at:
/tmp/run-Functional.fuego_lab_check-default-11-on-pptlab:rpi.frp
Run package run-Functional.fuego_lab_check-default-11-on-pptlab:rpi.frp
was accepted by the server.
Run Functional.fuego_lab_check-default-11-rpi put to server
Thanks and regards
Pavan Arun Deshpande
On Mon, Mar 15, 2021 at 8:56 PM Srivatsan S <[email protected]>
wrote:
> Tim,
>
> We couldn't do a git pull today to get the latest changes as we had some
> local changes to get the raspi_kinstall working. Currently, we have moved
> to a 64-bit kernel image for rpi board and have made some local changes to
> the script in Functional.raspi_kinstall to get it working with a 64-bit
> kernel image.
>
> We will do a git pull tomorrow and run our changes on top of it and get
> back to you.
>
> Thanks,
> Srivatsan
>
>
> On Mon, Mar 15, 2021 at 8:23 PM <[email protected]> wrote:
>
>> This is good to hear. Thanks for getting this resolved.
>> The errors that Fuego was returning were not very useful
>> for determining this problem. I'd like to think about how
>> this condition could be checked for, before or during test
>> execution, and reported back with a more useful error message.
>>
>> > # Version of this test=1.2-4
>> It looks like you are still running an old version of fuego_lab_check.
>> Can you please 'git pull' the latest master branch of fuego-core?
>>
>> I plan to do more testing this week.
>>
>> Thanks.
>> -- Tim
>>
>>
>> > -----Original Message-----
>> > From: Pavan Arun Deshpande <[email protected]>
>> >
>> > Hi Tim,
>> >
>> > The issue regarding Functional.fuego_lab_check has been resolved.
>> > The issue was due to the overlapping build numbers as concerned files
>> were already present in the container.
>> > Therefore the server was giving an error message saying that " Already
>> have a file cannot proceed".
>> > Uploading failed because of the above mentioned issue.
>> > I was able to resolve this after removing the previous run files.
>> >
>> > Functional.reboot also had the same issue and this also has been
>> resolved.
>> >
>> > This has been verified with multiple tests and iterations.
>> >
>> > Please find a copy of log messages generated while running
>> Functional.fuego_lab_check
>> >
>> > ===== doing fuego phase: pre_test =====
>> > ===== doing fuego phase: build =====
>> > The test is already built
>> > Fuego test_build duration=0 seconds
>> > ===== doing fuego phase: deploy =====
>> > ===== doing fuego phase: snapshot =====
>> > Firmware revision: 5.4.83-v8-93349cdff-1+
>> > ===== doing fuego phase: run =====
>> > -------------------------------------------------
>> > TAP version 13
>> > 1..1
>> > # Version of this test=1.2-4
>> > # Location of this test=/fuego-rw/tests/Functional.fuego_lab_check
>> > ######### Process list of docker container:
>> > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>> > root 1 0.0 0.0 19424 4544 pts/0 Ss+ 11:25 0:00
>> /bin/bash
>> > jenkins 58 0.0 0.0 18708 184 ? S 11:25 0:00
>> /usr/bin/daemon --name=jenkins --inherit --env=JENKINS_HOME=/var/lib/jenkins
>> > --output=/var/log/jenkins/jenkins.log
>> --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java
>> -Djava.awt.headless=true -
>> > Djenkins.install.runSetupWizard=false
>> -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar
>> > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war
>> --httpPort=8090 --prefix=/fuego
>> > jenkins 60 2.4 10.8 9946068 1769860 ? Sl 11:25 1:31
>> /usr/bin/java -Djava.awt.headless=true -
>> > Djenkins.install.runSetupWizard=false
>> -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar
>> > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war
>> --httpPort=8090 --prefix=/fuego
>> > root 110 0.0 0.0 9588 144 ? Ss 11:25 0:00
>> /usr/bin/netserver
>> > root 129 0.0 0.0 9440 96 ? Ss 11:25 0:00 iperf3
>> -V -s -D -f M
>> > jenkins 176 0.6 0.9 8487928 152384 ? Sl 11:25 0:23 java
>> -jar /fuego-core/slave.jar
>> > jenkins 177 0.6 0.9 8421364 150824 ? Sl 11:25 0:24 java
>> -jar /fuego-core/slave.jar
>> > jenkins 336 0.0 0.0 11180 2852 ? S 11:27 0:00
>> /bin/sh -xe /tmp/jenkins184800571287563138.sh
>> > jenkins 340 0.0 0.0 11256 3104 ? S 11:27 0:01
>> /bin/bash /fuego-core/scripts/poll_requests.sh -u -d 12h -w 120
>> > jenkins 4769 0.6 0.3 99016 56048 ? S 12:28 0:00
>> /usr/bin/python /usr/local/bin/ftc run-request --put-run --allow-upgrade
>> > request-2021-03-15_12:27:36.07-pptlab:rpi
>> > jenkins 4772 0.3 0.0 12404 4168 ? S 12:28 0:00
>> /bin/bash -e /fuego-core/scripts/main.sh
>> > jenkins 4855 0.0 0.0 11192 2848 ? S 12:28 0:00 bash
>> -c { ps -aux; echo $? > /tmp/4772-7340 ; } 2>&1 | tee -a /fuego-
>> > rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt
>> > jenkins 4856 0.0 0.0 11192 272 ? S 12:28 0:00 bash
>> -c { ps -aux; echo $? > /tmp/4772-7340 ; } 2>&1 | tee -a /fuego-
>> > rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt
>> > jenkins 4857 0.0 0.0 5832 656 ? S 12:28 0:00 tee -a
>> /fuego-rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt
>> > jenkins 4858 0.0 0.0 38304 3316 ? R 12:28 0:00 ps -aux
>> > ok 1 - process list of docker container
>> > ######### git describe of fuego-core
>> > v1.5.9-9-g79f9df5
>> > ok 2 - git describe of fuego-core
>> > ######### Installed tests - outside of fuego-core
>> > total 4
>> > drwxr-xr-x 2 root root 4096 Jan 29 04:40 Functional.fuego_lab_check
>> > ok 3 - remotely installed tests
>> > ######### Console log of the last run
>> > # last run=/fuego-rw/logs/Functional.fuego_board_check/rpi.default.91.91
>> > # Log:
>> > # ===== doing fuego phase: pre_test =====
>> > # ===== doing fuego phase: build =====
>> > # The test is already built
>> > # Fuego test_build duration=0 seconds
>> > # ===== doing fuego phase: deploy =====
>> > # ===== doing fuego phase: snapshot =====
>> > # Firmware revision: 5.4.83-v8-93349cdff-1+
>> > # ===== doing fuego phase: run =====
>> > # -------------------------------------------------
>> > # == Fuego target feature scan results ==
>> > # Information:
>> > # account information: uid=0(root) gid=0(root) groups=0(root)
>> > # working directory: /home/pi/test/fuego.Functional.fuego_board_check
>> > # PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>> > # -------------------------------------
>> > # Scan results:
>> > # HAS_PROGRAM_CAT=1
>> > # HAS_PROGRAM_DF=1
>> > # HAS_PROGRAM_FIND=1
>> > # HAS_PROGRAM_FREE=1
>> > # HAS_PROGRAM_GREP=1
>> > # HAS_PROGRAM_HEAD=1
>> > # HAS_PROGRAM_LOGGER=1
>> > # HAS_PROGRAM_LOGREAD=0
>> > # HAS_PROGRAM_MKDIR=1
>> > # HAS_PROGRAM_MOUNT=1
>> > # HAS_PROGRAM_PS=1
>> > # HAS_PROGRAM_RM=1
>> > # HAS_PROGRAM_RMDIR=1
>> > # HAS_PROGRAM_SYNC=1
>> > # HAS_PROGRAM_TAIL=1
>> > # HAS_PROGRAM_TEE=1
>> > # HAS_PROGRAM_TOUCH=1
>> > # HAS_PROGRAM_TRUE=1
>> > # HAS_PROGRAM_UMOUNT=1
>> > # HAS_PROGRAM_UNAME=1
>> > # HAS_PROGRAM_UPTIME=1
>> > # HAS_PROGRAM_XARGS=1
>> > # HAS_PROGRAM_BRACKET=1
>> > # HAS_MKDIR_P=1
>> > # HAS_RM_RF=1
>> > # HAS_GREP_F=1
>> > # HAS_SBIN_REBOOT=1
>> > # HAS_BIN_TRUE=1
>> > # HAS_SBIN_ROUTE=1
>> > # HAS_VAR_LOG_MESSAGES=0
>> > # HAS_VAR_LOG_SYSLOG=1
>> > # HAS_PROC_INTERRUPTS=1
>> > # HAS_PROC_DROP_CACHES=1
>> > # HAS_PROC_OOM_SCORE_ADJ=0
>> > # HAS_PROC_CONFIG_GZ=0
>> > # HAS_MEMORY=935532 K
>> > # HAS_STORAGE=15022716 K
>> > # Process 1 is program: /sbin/init
>> > # -------------------------------------
>> > # Based on this scan,
>> > # the recommended DISTRIB value should be:
>> > # nosyslogd.dist
>> > # -------------------------------------
>> > # RESULT=SUCCESS
>> > # All required programs, files, features, and permissions are present
>> > # -------------------------------------
>> > # 10+0 records in
>> > # 10+0 records out
>> > # 10000 bytes (10 kB, 9.8 KiB) copied, 0.000395734 s, 25.3 MB/s
>> > # Data from host:
>> > # TRANSPORT=ssh
>> > # Time to put 10K file: 0.68
>> > # Time to get 10K file: 0.69
>> > # Time to do 10 cmds: 6.74
>> > # Average time per cmd: 0.67
>> > # -------------------------------------------------
>> > # ===== doing fuego phase: post_test =====
>> > # ===== doing fuego phase: processing =====
>> > # log_compare: pattern 'SUCCESS' found 1 times (expected greater or
>> equal than 1)
>> > # chart config not found. Using default values.
>> > # Fuego: requested test phases complete!
>> > ######### end of log
>> > ok 4 - console log of last test run
>> > -------------------------------------------------
>> > ===== doing fuego phase: post_test =====
>> > scp:
>> /home/pi/test/fuego.Functional.fuego_lab_check/Functional.fuego_lab_check.log:
>> No such file or directory
>> > INFO: the test did not produce a test log on the target
>> > ===== doing fuego phase: processing =====
>> > chart config not found. Using default values.
>> > Fuego: requested test phases complete!
>> > Packaging run 'Functional.fuego_lab_check-default-10-rpi'
>> > Run packaged successfully, and is at:
>> /tmp/run-Functional.fuego_lab_check-default-10-on-pptlab:rpi.frp
>> > Run package run-Functional.fuego_lab_check-default-10-on-pptlab:rpi.frp
>> was accepted by the server.
>> > Run Functional.fuego_lab_check-default-10-rpi put to server
>> >
>> > Thanks and regards
>> >
>> > Pavan Arun deshpande
>> >
>> >
>> > On Sat, Mar 13, 2021 at 3:54 AM <[email protected] <mailto:
>> [email protected]> > wrote:
>> >
>> >
>> > There is a problem with running the test:
>> Functional.fuego_lab_check
>> > in your lab. I'll send the error in another message.
>> >
>> > Here is the error from the testlog:
>> >
>> > ls: cannot access '/fuego-rw/tests': No such file or directory
>> > ----
>> >
>> > And here is the report from the console log:
>> >
>> > ######### Installed tests (outside of fuego-core)
>> > ls: cannot access '/fuego-rw/tests': No such file or directory
>> > In Fuego signal_handler (due to error in run phase)
>> > ##### doing fuego phase: post_test (from signal handler) #####
>> > ...
>> > -----
>> >
>> > This output appears to be from an older version of
>> Functional.fuego_lab_check.
>> > One of the things I am testing with this test is the ability to
>> update the version
>> > of a test in your lab. Since there's a problem with the
>> /fuego-rw/tests directory,
>> > this operation is not working.
>> >
>> > Can you please do 3 things:
>> > 1) please check your docker container and make sure that it has
>> the directory
>> > /fuego-rw/tests
>> > and that this directory is owned by user jenkins and group
>> jenkins?
>> >
>> > 2) can you 'git pull' the latest fuego-core from the master
>> repository, to pick
>> > up the latest version of the Functional.fuego_lab_check test?
>> >
>> > Please make sure that you can run this test locally (from the
>> /fuego-core/tests
>> > directory). That is, just run this test from the updated
>> fuego-core repository
>> > in your lab, and confirm that it runs without error.
>> >
>> > Once you have done that, I will try some tests to see if I can
>> cause your lab
>> > to run upgraded tests.
>> >
>> > Thanks,
>> > -- Tim
>> >
>> >
>> >
>> >
>> >
>> > This message contains confidential information and is intended only for
>> the individual(s) named. If you are not the intended recipient, you
>> > are notified that disclosing, copying, distributing or taking any
>> action in reliance on the contents of this mail and attached file/s is
>> strictly
>> > prohibited. Please notify the sender immediately and delete this e-mail
>> from your system. E-mail transmission cannot be guaranteed to be
>> > secured or error-free as information could be intercepted, corrupted,
>> lost, destroyed, arrive late or incomplete, or contain viruses. The
>> > sender therefore does not accept liability for any errors or omissions
>> in the contents of this message, which arise as a result of e-mail
>> > transmission.
>>
>>
--
This
message contains confidential information and is intended only
for the
individual(s) named. If you are not the intended
recipient, you are
notified that disclosing, copying, distributing or taking any
action in
reliance on the contents of this mail and attached file/s is strictly
prohibited. Please notify the
sender immediately and delete this e-mail
from your system. E-mail transmission
cannot be guaranteed to be secured or
error-free as information could be
intercepted, corrupted, lost, destroyed,
arrive late or incomplete, or contain
viruses. The sender therefore does
not accept liability for any errors or
omissions in the contents of this
message, which arise as a result of e-mail
transmission.