Re: [Fuego] How do I reboot the DUT if system hang?

> -----Original Message-----
> From: Fuego <[email protected]> On Behalf Of Daniel Lin ???
> 
> Hi Tim & all,
> 
> Currently I often encounter the system hang while testing in fuego.
> When the system hang  happened,the following the test cases can not be ran.
> I read the about the PDU & PDU daemon on fuego pdfs,but I don’t understand what is it.
> 
> So anyone can point me how to reboot the DUT if system hang via hardware or so-called PDU?
> 

Hello Daniel,

Here is a some information about how Fuego handles board reboot.

By default, Fuego provides the 'target_reboot' command to perform a software
reboot of the board.  This executes "/sbin/reboot" on the board.   When Fuego
tries to reboot a board, it first attempts a software reboot (using 'target_reboot'),
and if the board is unresponsive after that, it performs a hardware reboot using
the overlay function 'ov_board_control_reboot'.  This overlay function is
located in the file: fuego-core/scripts/overlays/base-board.fuegoclass.

This routine can be overridden in a board file, with an override-func definition.
(I'll show an example of that below).

The 'ov_board_control_reboot' function (by default) calls 'ftc power-cycle'
for a board, which in turn uses information in the board file to perform the
actual board power switching to reboot the board.  ftc currently has support for two
different board control systems: 'pdudaemon' and 'ttc'.  I will describe these more below.

To answer your question, there are there are two main options for configuring
and implementing hardware reboot for a board in Fuego:

Option 1)  override the ov_board_control_reboot function, by placing an override-func
in the Fuego board file.
Option 2) add support for your board to either pdudaemon or ttc, and then
configure the board file to reference those board controller systems

Unless you are already using pdudaemon or ttc to control boards in your lab,
I think that it makes more sense to choose option 1. 

Option 1 requires that you be able to cause a hardware reboot or to power cycle
your board using command line programs or shell scripts that are available
inside your Fuego docker container.   This means you must have some pre-existing
method of rebooting your board.  It does not matter what the method is, as long
as Fuego can use it from inside the docker container, as user 'jenkins' (which is
the user id in effect during test execution inside the docker container).

If you do not have a way to control power to your board, then you will need to
come up with something.  In my own lab, I have a mixture of USB-based control
boards, serial-based control boards, web power switches, and web relay devices
that are used for different boards in my lab.  For each of these I have a command
line program which is used to control the power for a particular board under test.

If you have a script called "reboot-my-board" that can perform a reboot of
your board, then make sure that program is callable from inside the docker
container.  I would recommend putting it into /usr/local/bin inside the container.

You also need to set the value of BOARD_CONTROL in the board file, to indicate
that you have your own ov_board_control_reboot() function.  In this case, set
the value to 'custom'.

BOARD_CONTROL="custom"

Then add an override-func for ov_board_control_reboot() in the board file for
the board that you need to reboot.  If your board is named "mybbb" the board file
would be: /fuego-ro/boards/mybbb.board.  The function declaration in the
board file might look something like this:

override-func ov_board_control_reboot() {
     reboot-my-board ; sleep 20
}

Whether you need a sleep or not depends on what 'reboot-my-board' does.  In
general, it is safest to add the delay for hardware reboot into the
ov_board_control_reboot function, if it is not provided by the called program.

You do not have to use just a single command.  Basically, if you can make the reboot
happen with a series of commands, then you can put those into your override function
in your board file.

Below is an example from my own lab. 
In this particular example, I am using a controller board which is attached to the Fuego host
via a serial port, and which accepts the commands "v" and "V" on the serial
line to turn off (or on, respectively) the voltage to the board under test.

override-func ov_board_control_reboot() {
    echo "Turning off power to board (in ov_board_control_reboot)"
    echo v >/dev/serial/by-id/[email protected]_CDB_Assist_00000042-if02
    sleep 3
    echo "Turning on power to board (in ov_board_control_reboot)"
    echo V >/dev/serial/by-id/[email protected]_CDB_Assist_00000042-if02
    echo "Waiting for board to boot (in ov_board_control_reboot)"
    sleep 40
}

My definition includes some debug echo statements to help see what
is going on.  These messages would appear in the console log
if this function is called during test execution.

Option 2) Using an existing board control system.
The other option is to use an existing board control system, such as pdudaemon
or ttc.  In general, pdudaemon is a network-based control system, and ttc is a
command-line based control system. The description of setting up and configuring
either of these systems is outside the scope of this e-mail.  However, one
important note is that it must be possible to effectively use these systems from
inside the Fuego docker container.  For pdudaemon, this is usually not an issue,
since it can be accessed via the network (Fuego does not use pduclient).
For ttc, this means that any helper scripts that are used by ttc must be available
inside the container (and must work from inside the container).  If ttc or it's
helpers needs to communicate via a serial port or USB port, then a 
privileged docker container should be used to allow for this.
(see Fuego's install.sh '--priv' option).

Even though setting up pdudaemon or ttc is outside the scope of this
message, here is some information about how these systems are integrated
into Fuego:

If using pdudaemon as your board control system, then add the following to
the Fuego board file for the board under test:
BOARD_CONTROL="pdudaemon"
PDUDAEMON_HOSTNAME="<the hostname>"
PDUDAEMON_PORT="<the port>"
PDUDAEMON_DELAY="<value in seconds>"

The delay value is optional.

If using ttc as your board control system, then add the following to the Fuego board
file for the board under test:
BOARD_CONTROL="ttc"
TTC_TARGET="<the target name>"

TTC_TARGET is not needed if the ttc target name is the same as the Fuego board name.

Once properly configured, and the settings are in the Fuego board file, you should be
able to do the following operations on a board, using the 'ftc' command.
  ftc power-cycle -b <board-name>
  ftc power-off -b <board-name>
  ftc power-on -b <board-name>

Executing these command line operations is a good way to test that the
configuration is correct for your selected board control system.

Finally, there is a test in Fuego to validate that software and hardware reboot
for you board are working correctly.  It is the test 'Functional.reboot'.  Please
try it on your system after you have configured one of the above options, and
let me know your results.

I hope this is helpful.  Please do not hesitate to ask if you have any
additional questions.
 -- Tim

Resources:
pdudaemon: https://github.com/pdudaemon/pdudaemon
ttc: https://github.com/tbird20d/ttc

Note that the Fuego docker container already has the 'ttc' command installed,
but you need to configure 'ttc' to use it.