[Fuego] 回覆: How do I reboot the DUT if system hang?

Hi Tim,

Thanks soooo much for your reply.It's so helpful.
But sorry for my late reply.

I follow your instructions to setup hardware reboot and it worked!
I use USBRelay(a usb device can control usb port to power up/down) to control board if board hang.

step1: Install fuego with usb privileged option.for example
$./install.sh --priv myfuego

step2: 
Here is my setting as below under ~/fuego/fuego-ro/boards/dut.board 

BOARD_CONTROL="custom"

override-func ov_board_control_reboot() {
    echo "Turning off power to board (in ov_board_control_reboot)"
    sleep 3
    #USBRelay_on.py let DUT power off and on via USB port(ex. /dev/ttyUSB1)
    #chmod 666 /dev/ttyUSB1 if you encounter access deny with USB port.
    cd /home
    python3 USBRelay_on.py
    echo "Turning on power to board(in ov_board_control_reboot)"
    sleep 40
}

While the DUT is no response or no network,then fuego start the hardware reboot by " python3 USBRelay_on.py"
So all the testcases can be separate if specific testcase caused the DUT hang.

Thanks again.
Daniel Lin

-----原始]件-----
寄件者: [email protected] <[email protected]> 
寄件日期: 2021年2月3日 上午 04:19
收件者: Daniel Lin 林源祥 <[email protected]>; [email protected]
主旨: RE: [Fuego] How do I reboot the DUT if system hang?

> -----Original Message-----
> From: Fuego <[email protected]> On Behalf Of Daniel Lin ???
> 
> Hi Tim & all,
> 
> Currently I often encounter the system hang while testing in fuego.
> When the system hang  happened,the following the test cases can not be ran.
> I read the about the PDU & PDU daemon on fuego pdfs,but I don’t understand what is it.
> 
> So anyone can point me how to reboot the DUT if system hang via hardware or so-called PDU?
> 

Hello Daniel,

Here is a some information about how Fuego handles board reboot.

By default, Fuego provides the 'target_reboot' command to perform a software
reboot of the board.  This executes "/sbin/reboot" on the board.   When Fuego
tries to reboot a board, it first attempts a software reboot (using 'target_reboot'), and if the board is unresponsive after that, it performs a hardware reboot using the overlay function 'ov_board_control_reboot'.  This overlay function is located in the file: fuego-core/scripts/overlays/base-board.fuegoclass.

This routine can be overridden in a board file, with an override-func definition.
(I'll show an example of that below).

The 'ov_board_control_reboot' function (by default) calls 'ftc power-cycle'
for a board, which in turn uses information in the board file to perform the actual board power switching to reboot the board.  ftc currently has support for two different board control systems: 'pdudaemon' and 'ttc'.  I will describe these more below.

To answer your question, there are there are two main options for configuring and implementing hardware reboot for a board in Fuego:

Option 1)  override the ov_board_control_reboot function, by placing an override-func in the Fuego board file.
Option 2) add support for your board to either pdudaemon or ttc, and then configure the board file to reference those board controller systems

Unless you are already using pdudaemon or ttc to control boards in your lab, I think that it makes more sense to choose option 1. 

Option 1 requires that you be able to cause a hardware reboot or to power cycle your board using command line programs or shell scripts that are available
inside your Fuego docker container.   This means you must have some pre-existing
method of rebooting your board.  It does not matter what the method is, as long as Fuego can use it from inside the docker container, as user 'jenkins' (which is the user id in effect during test execution inside the docker container).

If you do not have a way to control power to your board, then you will need to come up with something.  In my own lab, I have a mixture of USB-based control boards, serial-based control boards, web power switches, and web relay devices that are used for different boards in my lab.  For each of these I have a command line program which is used to control the power for a particular board under test.

If you have a script called "reboot-my-board" that can perform a reboot of your board, then make sure that program is callable from inside the docker container.  I would recommend putting it into /usr/local/bin inside the container.

You also need to set the value of BOARD_CONTROL in the board file, to indicate that you have your own ov_board_control_reboot() function.  In this case, set the value to 'custom'.

BOARD_CONTROL="custom"

Then add an override-func for ov_board_control_reboot() in the board file for the board that you need to reboot.  If your board is named "mybbb" the board file would be: /fuego-ro/boards/mybbb.board.  The function declaration in the board file might look something like this:

override-func ov_board_control_reboot() {
     reboot-my-board ; sleep 20
}

Whether you need a sleep or not depends on what 'reboot-my-board' does.  In general, it is safest to add the delay for hardware reboot into the ov_board_control_reboot function, if it is not provided by the called program.

You do not have to use just a single command.  Basically, if you can make the reboot happen with a series of commands, then you can put those into your override function in your board file.

Below is an example from my own lab. 
In this particular example, I am using a controller board which is attached to the Fuego host via a serial port, and which accepts the commands "v" and "V" on the serial line to turn off (or on, respectively) the voltage to the board under test.

override-func ov_board_control_reboot() {
    echo "Turning off power to board (in ov_board_control_reboot)"
    echo v >/dev/serial/by-id/[email protected]_CDB_Assist_00000042-if02
    sleep 3
    echo "Turning on power to board (in ov_board_control_reboot)"
    echo V >/dev/serial/by-id/[email protected]_CDB_Assist_00000042-if02
    echo "Waiting for board to boot (in ov_board_control_reboot)"
    sleep 40
}

My definition includes some debug echo statements to help see what is going on.  These messages would appear in the console log if this function is called during test execution.

Option 2) Using an existing board control system.
The other option is to use an existing board control system, such as pdudaemon or ttc.  In general, pdudaemon is a network-based control system, and ttc is a command-line based control system. The description of setting up and configuring either of these systems is outside the scope of this e-mail.  However, one important note is that it must be possible to effectively use these systems from inside the Fuego docker container.  For pdudaemon, this is usually not an issue, since it can be accessed via the network (Fuego does not use pduclient).
For ttc, this means that any helper scripts that are used by ttc must be available inside the container (and must work from inside the container).  If ttc or it's helpers needs to communicate via a serial port or USB port, then a privileged docker container should be used to allow for this.
(see Fuego's install.sh '--priv' option).

Even though setting up pdudaemon or ttc is outside the scope of this message, here is some information about how these systems are integrated into Fuego:

If using pdudaemon as your board control system, then add the following to the Fuego board file for the board under test:
BOARD_CONTROL="pdudaemon"
PDUDAEMON_HOSTNAME="<the hostname>"
PDUDAEMON_PORT="<the port>"
PDUDAEMON_DELAY="<value in seconds>"

The delay value is optional.

If using ttc as your board control system, then add the following to the Fuego board file for the board under test:
BOARD_CONTROL="ttc"
TTC_TARGET="<the target name>"

TTC_TARGET is not needed if the ttc target name is the same as the Fuego board name.

Once properly configured, and the settings are in the Fuego board file, you should be able to do the following operations on a board, using the 'ftc' command.
  ftc power-cycle -b <board-name>
  ftc power-off -b <board-name>
  ftc power-on -b <board-name>

Executing these command line operations is a good way to test that the configuration is correct for your selected board control system.

Finally, there is a test in Fuego to validate that software and hardware reboot for you board are working correctly.  It is the test 'Functional.reboot'.  Please try it on your system after you have configured one of the above options, and let me know your results.

I hope this is helpful.  Please do not hesitate to ask if you have any additional questions.
 -- Tim

Resources:
pdudaemon: https://github.com/pdudaemon/pdudaemon
ttc: https://github.com/tbird20d/ttc

Note that the Fuego docker container already has the 'ttc' command installed, but you need to configure 'ttc' to use it.