From: Eugene Loh <[email protected]>
Signed-off-by: Eugene Loh <[email protected]>
---
Build | 4 +-
doc/tutorial/1.IntroducingDTrace.md | 380 ++++
.../2.TracingOperatingSystemBehavior.md | 1741 +++++++++++++++++
.../3.TracingUserSpaceApplications.md | 832 ++++++++
doc/tutorial/4.GoingFurtherWithDTrace.md | 14 +
doc/tutorial/index.md | 15 +
6 files changed, 2985 insertions(+), 1 deletion(-)
create mode 100644 doc/tutorial/1.IntroducingDTrace.md
create mode 100644 doc/tutorial/2.TracingOperatingSystemBehavior.md
create mode 100644 doc/tutorial/3.TracingUserSpaceApplications.md
create mode 100644 doc/tutorial/4.GoingFurtherWithDTrace.md
create mode 100644 doc/tutorial/index.md
diff --git a/Build b/Build
index 220c97845..434f8ad92 100644
--- a/Build
+++ b/Build
@@ -3,14 +3,16 @@
install::
mkdir -p $(INSTDOCDIR)
+ mkdir -p $(INSTDOCDIR)/tutorial
mkdir -p $(INSTDOCDIR)/userguide
mkdir -p $(INSTDOCDIR)/userguide/explanation
mkdir -p $(INSTDOCDIR)/userguide/how-to
mkdir -p $(INSTDOCDIR)/userguide/reference
mkdir -p $(INSTDOCDIR)/examples/
mkdir -p $(INSTDOCDIR)/examples/language_features
- $(call describe-install-target,$(INSTDOCDIR),README INCOMPATIBILITIES NEWS userguide examples)
+ $(call describe-install-target,$(INSTDOCDIR),README INCOMPATIBILITIES NEWS tutorial userguide examples)
install -m 644 README INCOMPATIBILITIES NEWS $(INSTDOCDIR)
+ install -m 644 doc/tutorial/*.md $(INSTDOCDIR)/tutorial
install -m 644 doc/userguide/index.md $(INSTDOCDIR)/userguide
install -m 644 doc/userguide/explanation/* $(INSTDOCDIR)/userguide/explanation
install -m 644 doc/userguide/how-to/* $(INSTDOCDIR)/userguide/how-to
diff --git a/doc/tutorial/1.IntroducingDTrace.md b/doc/tutorial/1.IntroducingDTrace.md
new file mode 100644
index 000000000..c91cf88e4
--- /dev/null
+++ b/doc/tutorial/1.IntroducingDTrace.md
@@ -0,0 +1,380 @@
+
+## Introducing DTrace
+
+This chapter introduces the dynamic tracing (DTrace) facility,
+available on Linux.
+You can use DTrace to examine the behavior of the operating system
+and of user-space programs, particularly those that have been
+instrumented with DTrace probes.
+DTrace has been implemented on Linux using eBPF,
+allowing DTrace to run as a user-space tool,
+albeit as root for most uses.
+
+### About This Tutorial
+
+This tutorial includes a variety of DTrace scripts and describes
+different ways in which you can use DTrace.
+Several examples have additional exercises that offer further
+practice in using DTrace.
+You should already have a good understanding of Linux administration
+and system programming, and broad experience using a programming language,
+such as C or C++, and a scripting language, such as Python.
+If you are not familiar with terms such as *system call*, *type*,
+*cast*, *signal*, *struct*, or *pointer*,
+you might have difficulty in understanding some of the examples
+or completing some of the exercises in this tutorial.
+However, each exercise provides a sample solution in case you do get stuck.
+You are encouraged to experiment with the examples to develop your
+skills at creating DTrace scripts.
+
+>Caution:
+>
+>To run the examples and perform the exercises in this tutorial,
+you need to have `root` access to a system.
+Only the `root` user or a user with `sudo` access to run commands
+as `root` can use the `dtrace` utility.
+As `root`, you have total power over a
+system and so have total responsibility for that system.
+Although DTrace is designed so that you can use it safely
+without needing to worry about corrupting the operating system
+or other processes,
+there are ways to circumvent some of the default, built-in safety measures.
+>
+>To minimize risk, perform the examples and exercises in this
+tutorial on a system other than a production system.
+>
+>Also, make sure that `/usr/sbin` is before `/usr/bin` in the root path,
+or specify `/usr/sbin/dtrace` explicitly.
+
+The examples in this tutorial demonstrate the different ways that
+you can perform dynamic tracing of your system: by entering a
+simple D program as an argument to `dtrace` on
+the command line, by using the `dtrace` command
+to run a script that contains a D program, or by using an
+executable D script that contains a *hashbang*
+(`#!` or *shebang*) invocation of `dtrace`.
+When you create your own D programs,
+you can choose which method best suits your needs.
+
+### About DTrace
+
+DTrace is a comprehensive dynamic tracing facility that was first
+developed for use on the Solaris operating system (now known as
+Oracle Solaris) and subsequently ported to Oracle Linux.
+You can use
+DTrace to explore the operation of your system to better
+understand how it works, to track down performance problems across
+many layers of software, or to locate the causes of aberrant
+behavior.
+
+Using DTrace, you can record data at previously instrumented
+places of interest, which are referred to as
+*probes*, in kernel and user-space programs.
+A probe is a location to which DTrace can bind a request to perform
+a set of actions, such as recording a stack trace, a timestamp, or
+the argument to a function.
+Probes function like programmable
+sensors that record information.
+When a probe is triggered, DTrace
+gathers data that you have designated in a D script and reports
+this data back to you.
+
+Using DTrace's D programming language, you can query the system
+probes to provide immediate and concise answers to any number of
+questions that you might formulate.
+
+A D program describes the actions that occur if one or more
+specified probes is triggered.
+A probe is uniquely specified by
+the name of the DTrace provider that publishes the probe, the name
+of the module, library, or user-space program in which the probe
+is located, the name of the function in which the probe is
+located, and the name of the probe itself, which usually describes
+some operation or functionality that you can trace.
+Because you do
+not need to specify probes exactly, this allows DTrace to perform
+the same action for a number of different probes.
+Full and
+explicit representation of a single probe in the D language takes
+the form *provider*:*module*:*function*:*name*.
+
+When you use the `dtrace` command to run a D
+program, you invoke the compiler for the D language.
+The compiled code, eBPF instructions, is loaded into the kernel
+and attached to the appropriate probes.
+DTrace handles any runtime errors that might occur
+during your D program's execution, including dividing by zero,
+dereferencing invalid memory, and so on, and reports them to you.
+
+The *module* is the kernel module in which a probe appears or,
+in the case of user code, the load object. Some probes, such
+as clock-triggered profiling interrupts, have no module associated
+with them.
+
+Unless you explicitly permit DTrace to perform potentially
+destructive actions, you cannot construct an unsafe program that
+would cause DTrace to inadvertently damage either the operating
+system or any process that is running on your system. These safety
+features enable you to use DTrace in a production environment
+without worrying about crashing or corrupting your system. If you
+make a programming mistake, DTrace reports the error and
+deactivates your program's probes. You can then correct your
+program and try again.
+
+For more information about using DTrace, see the
+[DTrace User Guide](../userguide/index.md).
+
+### About DTrace Providers
+
+Here are the providers in the Oracle Linux implementation of DTrace:
+
+```
+ kernel user space
+ +-----------------------------+
+ | dtrace | I want to control my D program.
+ +----------------+------------+
+ | | syscall | I want to see how I call the kernel.
+ +----------------+------------+
+ | cpc profile | I want to look at resource usage.
+ +----------------+------------+
+ | ip tcp udp | |
+ | io lockstat | | I know the semantics of the code.
+ | sdt rawtp | usdt | (I know what probes are in the code.)
+ | proc sched | |
+ +----------------+------------+
+ | fbt rawfbt | pid | I know the functions of the code.
+ +----------------+------------+
+```
+
+That is, these providers provide:
+
+- `dtrace`: probes that relate to DTrace itself,
+ such as `BEGIN`, `ERROR`, and `END`.
+ You can use these probes to initialize DTrace's state before
+ tracing begins, process its state after tracing has completed,
+ and handle unexpected execution errors in other probes.
+- `syscall`: probes that fire at the entry to and return from every system call.
+ Because system calls are the primary interface between user-level
+ applications and the operating system kernel, these probes can
+ offer you an insight into the interaction between applications and
+ the system.
+- `cpc`: probes that fire when hardware resource counters (or
+ software emulation thereof) trip, giving you an idea of
+ where statistically hardware events are triggered.
+- `profile`: probes that fire at fixed and specified time intervals.
+ You can use these probes to sample some aspect of a system's state.
+- `ip`: probes that fire for important steps in the IP protocol,
+ for both IPv4 and IPv6.
+- `tcp`: probes that fire for important steps in the TCP protocol,
+ for both IPv4 and IPv6.
+- `udp`: probes that fire for important steps in the the UDP protocol,
+ for both IPv4 and IPv6.
+- `io`: probes that relate to data input and output.
+ The `io` provider enables quick exploration of behavior
+ observed through I/O monitoring.
+- `lockstat`: probes that fire for important steps in lock handling,
+ including for mutexes, read-write locks, and spin locks.
+- `sdt`: probes that fire when statically defined tracepoints in the Linux kernel
+ are executed.
+- `rawtp`: probes that fire when statically defined tracepoints in the Linux kernel
+ are executed, but report raw arguments.
+- `proc`: probes that fire for process creation and termination, LWP creation
+ and termination, execution of new programs, and signal handling.
+- `sched`: probes that fire for important steps in CPU scheduling.
+ Because CPUs are the one resource that all threads must consume,
+ the `sched` provider is very useful for understanding systemic behavior.
+- `usdt`: probes that fire when user-space statically defined trace points
+ are encountered.
+- `fbt`: probes that fire for function boundary tracing (FBT) -- that is,
+ when kernel functions are entered or return.
+- `rawfbt`: probes that are like `fbt`, but are implemented with `kprobes`,
+ include synthetic functions (such as compiler-generated optimized
+ variants of functions with `.` suffixes.), and have unconverted and
+ untyped probe arguments.
+- `pid`: probes that fire for the specified process id (`pid`) upon
+ entry into or return from a function.
+
+See [DTrace Providers](../userguide/reference/dtrace_providers.md) in
+the [DTrace User Guide](../userguide/index.md)
+for more information about providers and their probes.
+
+### Preparing to Install and Configure DTrace
+
+DTrace is available for Linux from
+[github](https://github.com/oracle/dtrace-utils/tree/stable).
+Or, there are
+[DTrace packages built for Oracle Linux](https://www.oracle.com/linux/downloads/linux-dtrace.html).
+
+#### Example: Displaying Probes for a Provider
+
+The following example shows how you would display the probes
+for a provider, such as `proc`, by using the `dtrace` command.
+
+```
+# dtrace -l -P proc
+ ID PROVIDER MODULE FUNCTION NAME
+ 53 proc vmlinux start
+ 52 proc vmlinux signal-send
+ 51 proc vmlinux signal-handle
+ 50 proc vmlinux signal-discard
+ 49 proc vmlinux signal-clear
+ 48 proc vmlinux lwp-start
+ 47 proc vmlinux lwp-exit
+ 46 proc vmlinux lwp-create
+ 45 proc vmlinux exit
+ 44 proc vmlinux exec-success
+ 43 proc vmlinux exec-failure
+ 42 proc vmlinux exec
+ 41 proc vmlinux create
+```
+
+These probes enable you to monitor how the system creates
+processes, executes programs, and handles signals.
+
+The output shows the numeric identifier of the probe, the name
+of the probe provider, the name of the probe module, the name
+of the function that contains the probe (none for this provider),
+and the name of the probe itself.
+
+The full name of a probe is *provider*:*module*:*function*:*name*,
+for example, `proc:vmlinux::create`.
+
+When probes are listed, a missing field indicates there is no
+corresponding value.
+When you specify a probe, omitting a field is equivalent to using
+the wildcard `*`.
+
+#### Exercise: Enabling and Listing DTrace Probes
+
+Try listing the probes of the `syscall` provider.
+Notice that both `entry` and `return` probes are provided for each system call.
+
+#### Solution to Exercise: Enabling and Listing DTrace Probes
+
+```
+# dtrace -l -P syscall
+ ID PROVIDER MODULE FUNCTION NAME
+ 5055 syscall vmlinux socket entry
+ 5054 syscall vmlinux socket return
+ 5053 syscall vmlinux socketpair entry
+ 5052 syscall vmlinux socketpair return
+ 5051 syscall vmlinux bind entry
+ 5050 syscall vmlinux bind return
+ 5049 syscall vmlinux listen entry
+ 5048 syscall vmlinux listen return
+ ...
+ 4375 syscall vmlinux ioperm entry
+ 4374 syscall vmlinux ioperm return
+ 4373 syscall vmlinux iopl entry
+ 4372 syscall vmlinux iopl return
+ 4371 syscall vmlinux rt_sigreturn entry
+ 4370 syscall vmlinux rt_sigreturn return
+ 4369 syscall vmlinux arch_prctl entry
+ 4368 syscall vmlinux arch_prctl return
+```
+
+The probe ID numbers can differ from run to run.
+
+### Running a Simple DTrace Program
+
+The following example shows how you could use a simple D program
+that is in a file called `hello.d`.
+
+#### Example: Simple D Program That Uses the BEGIN Probe (hello.d)
+
+```
+/* hello.d -- A simple D program that uses the BEGIN probe */
+
+BEGIN
+{
+ /* This is a C-style comment */
+ trace("hello, world");
+ exit(0);
+}
+```
+
+A D program consists of a series of clauses, where each clause
+describes one or more probes to enable, and an optional set of actions
+to perform when the probe fires. The actions are listed as a series
+of statements enclosed in braces `{}` following the probe name.
+Each statement ends with a semicolon (`;`).
+
+In this example, the function `trace` directs DTrace to record the
+specified argument, the string ”hello, world”, when the `BEGIN` probe fires,
+and then print it out.
+The function `exit()` tells DTrace to cease tracing and exit the `dtrace`
+command.
+
+The full name of the `BEGIN` probe is `dtrace:::BEGIN`.
+The `dtrace` provider provides three probes: `dtrace:::BEGIN`,
+`dtrace:::END`, and `dtrace:::ERROR`.
+Because these probe names are unique to the `dtrace` provider,
+their names can be shortened to `BEGIN`, `END`, and `ERROR`.
+
+When you have saved your program, you can run it by using the
+`dtrace` command with the `-s` option,
+which specifies the name of the file that contains the D program:
+
+```
+# dtrace -s hello.d
+dtrace: script 'hello.d' matched 1 probe
+CPU ID FUNCTION:NAME
+ 0 1 :BEGIN hello, world
+```
+
+DTrace interprets and runs the script. You will notice
+that in addition to the string `"hello,world"`,
+the default behavior of DTrace is to display information about
+the CPU on which the script was running when a probe fired,
+the ID of the probe, the name of the function that contains
+the probe, and the name of the probe itself.
+The function name is displayed as blank for `BEGIN`,
+as DTrace provides this probe.
+
+You can suppress the probe information in a number of different
+ways, for example, by specifying the `-q` quiet option:
+
+```
+# dtrace -q -s hello.d
+hello, world
+```
+
+#### Exercise: Using the END Probe
+
+Copy the `hello.d` program to the file `goodbye.d`.
+Edit this file so that it traces the string "goodbye, world"
+and uses the `END` probe instead of `BEGIN`.
+When you run this new script, you need to type `Ctrl-C` to cause the
+probe to fire and exit `dtrace`.
+
+#### Solution to Exercise and Example: Using the END Probe
+
+The following is an example of a simple D program that
+demonstrates the use of the `END` probe:
+
+```
+/* goodbye.d -- Simple D program that demonstrates the END probe */
+
+END
+{
+ trace("goodbye, world");
+}
+```
+
+The default output is:
+
+```
+# dtrace -s goodbye.d
+dtrace: script 'goodbye.d' matched 1 probe
+^C
+CPU ID FUNCTION:NAME
+ 3 2 :END goodbye, world
+```
+
+Or, with the `-q` quiet option:
+
+```
+# dtrace -q -s ./goodbye.d
+^C
+goodbye, world
+```
diff --git a/doc/tutorial/2.TracingOperatingSystemBehavior.md b/doc/tutorial/2.TracingOperatingSystemBehavior.md
new file mode 100644
index 000000000..b3445ab6d
--- /dev/null
+++ b/doc/tutorial/2.TracingOperatingSystemBehavior.md
@@ -0,0 +1,1741 @@
+
+## Tracing Operating System Behavior
+
+This chapter provides examples of D programs that you can use to
+investigate what is happening in the operating system.
+
+### Tracing Process Creation
+
+The `proc` probes enable you to trace process creation and termination,
+execution of new program images, and signal processing on a system. See
+[Proc Provider](../userguide/reference/dtrace_providers_proc.md)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
+for a description of the `proc` probes and their arguments.
+
+#### Example: Monitoring the System as Programs Are Executed (execcalls.d)
+
+The following example shows the D program, `execcalls.d`, which uses
+`proc` probes to monitor the system as it executes process images:
+
+```
+/* execcalls.d -- Monitor the system as it executes programs */
+
+proc:::exec
+{
+ trace(stringof(args[0]));
+}
+```
+
+The `args[0]` argument to the `exec` probe is set to the path name of the
+program that is being executed.
+You use the `stringof()` function to convert the type from
+`char *` to the D type `string`.
+
+Type the `dtrace -s execcalls.d` command to run the D program in one window.
+Then start different programs from another window,
+while observing the output from `dtrace` in the first window.
+To stop tracing after a few seconds have elapsed,
+type `Ctrl-C` in the window that is running `dtrace`.
+
+```
+# dtrace -s execcalls.d
+dtrace: script 'execcalls.d' matched 1 probe
+CPU ID FUNCTION:NAME
+ 3 42 :exec /usr/sbin/sshd
+ 1 42 :exec /bin/bash
+ 0 42 :exec /usr/bin/id
+ 2 42 :exec /usr/bin/hostnamectl
+ 1 42 :exec /usr/lib/systemd/systemd-hostnamed
+ 3 42 :exec /usr/bin/register-python-argcomplete
+ 3 42 :exec /usr/libexec/grepconf.sh
+ 0 42 :exec /usr/bin/grep
+ 2 42 :exec /usr/bin/dircolors
+ 3 42 :exec /usr/bin/grep
+ 3 42 :exec /usr/libexec/grepconf.sh
+^C
+```
+
+The activity here shows a login to the same system (from another
+terminal) while the script is running.
+
+The probe `proc:::exec` fires whenever the system executes a new program and
+the associated action uses `trace()` to display the path name of the program.
+
+#### Exercise: Suppressing Verbose Output From DTrace
+
+Run the `execcalls.d` program again, but this time add the
+`-q` option to suppress all output except output from `trace()`.
+Notice how DTrace displays only what you traced with `trace()`.
+
+#### Solution to Exercise: Suppressing Verbose Output From DTrace
+
+```
+# dtrace -q -s execcalls.d
+/usr/sbin/sshd/bin/bash/usr/bin/id/usr/bin/hostnamectl/usr/lib/systemd/systemd-h
+ostnamed/usr/bin/register-python-argcomplete/usr/libexec/grepconf.sh/usr/bin/gre
+p/usr/bin/dircolors/usr/bin/grep/usr/libexec/grepconf.sh^C
+```
+
+### Tracing System Calls
+
+System calls are the interface between user programs and the
+kernel, which perform operations on the programs' behalf.
+
+The next example shows the next D program,
+`syscalls.d`, which uses
+`syscall` probes to record
+`open()` system call activity on a system.
+
+#### Example: Recording open() System Calls on a System (syscalls.d)
+
+```
+/* syscalls.d -- Record open() system calls on a system */
+
+syscall::open:entry
+{
+ printf("%-16s %-16s\n",execname,copyinstr(arg0));
+}
+syscall::openat*:entry
+{
+ printf("%-16s %-16s\n",execname,copyinstr(arg1));
+}
+```
+
+In this example, the `printf()` function is used to display the
+name of the executable that is calling `open()` and the path name
+of the file that it is attempting to open.
+
+>Note:
+>
+>Use the `copyinstr()` function to convert the first argument (`arg0`)
+in the `open()` call to a string.
+Whenever a probe accesses a pointer to data in the address space of a user process,
+you must use one of the `copyin()`, `copyinstr()`, or `copyinto()` functions
+to copy the data from user space to a DTrace buffer in kernel space.
+In this example, it is appropriate to use `copyinstr()`,
+as the pointer refers to a character array.
+If the string is not null-terminated, you also need to specify the
+length of the string to `copyinstr()`, for example,
+`copyinstr(arg1, arg2 + 1)`, for a system call such as `write()`,
+to copy arg2 bytes and then add a NUL terminating character after those bytes,
+which are possibly already NUL-terminated.
+See
+[Pointers and Address Spaces](../userguide/reference/dtrace-ref-PointersandScalarArrays.md#pointers_and_address_spaces)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md).
+
+```
+# dtrace -q -s syscalls.d
+irqbalance /proc/interrupts
+irqbalance /proc/stat
+irqbalance /proc/interrupts
+irqbalance /proc/stat
+irqbalance /proc/interrupts
+irqbalance /proc/stat
+NetworkManager /proc/sys/net/ipv6/conf/enX0/mtu
+irqbalance /proc/interrupts
+irqbalance /proc/stat
+systemd /proc/1200459/cgroup
+NetworkManager /proc/sys/net/ipv6/conf/enX0/mtu
+^C
+```
+
+#### Exercise: Using the printf() Function to Format Output
+
+Amend the arguments to the `printf()` function so that `dtrace`
+also prints the process ID and user ID for the process.
+Use a conversion specifier such as `%-4d`.
+
+See
+[printf](../userguide/reference/function_printf.md)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
+for a description of the function.
+
+The process ID and user ID are available as the variables
+`pid` and `uid`. Use the
+`BEGIN` probe to create a header for the
+output.
+
+#### Solution to Exercise: Using the printf() Function to Format Output
+
+```
+/* syscalls1.d -- Modified version of syscalls.d that displays more information */
+
+BEGIN
+{
+ printf("%-7s %-4s %-16s %-16s\n","PID","UID","EXECNAME","FILENAME");
+ mypid = pid;
+}
+
+syscall::open:entry
+/pid != mypid/
+{
+ printf("%-7d %-4d %-16s %-16s\n",pid,uid, execname,copyinstr(arg0));
+}
+syscall::openat*:entry
+/pid != mypid/
+{
+ printf("%-7d %-4d %-16s %-16s\n",pid,uid, execname,copyinstr(arg1));
+}
+```
+
+Note how this solution uses similar formatting strings to output
+the header and the data.
+Also, we add *predicates* to exclude probes that fire on the `dtrace`
+process itself.
+
+```
+# dtrace -q -s syscalls1.d
+PID UID EXECNAME FILENAME
+2852652 0 nm-daemon-helpe /etc/ld.so.cache
+2852652 0 nm-daemon-helpe /lib64/libgcc_s.so.1
+2852652 0 nm-daemon-helpe /lib64/libc.so.6
+2852652 0 nm-daemon-helpe /etc/nsswitch.conf
+2852652 0 nm-daemon-helpe /etc/host.conf
+2852652 0 nm-daemon-helpe /etc/resolv.conf
+2852652 0 nm-daemon-helpe /etc/hosts
+2852653 0 nm-daemon-helpe /etc/ld.so.cache
+2852653 0 nm-daemon-helpe /lib64/libgcc_s.so.1
+2852653 0 nm-daemon-helpe /lib64/libc.so.6
+2852653 0 nm-daemon-helpe /etc/nsswitch.conf
+2852653 0 nm-daemon-helpe /etc/host.conf
+2852653 0 nm-daemon-helpe /etc/resolv.conf
+2852653 0 nm-daemon-helpe /etc/hosts
+1200456 0 irqbalance /proc/interrupts
+1200456 0 irqbalance /proc/stat
+1 0 systemd /proc/1200459/cgroup
+1 0 systemd /proc/1200460/cgroup
+^C
+```
+
+### Performing an Action at Specified Intervals
+
+The `profile` provider includes `tick` probes that you can use
+to sample some aspect of a system's state at regular intervals.
+
+#### Example: Using `tick.d`
+
+The following is an example of the `tick.d` program.
+
+```
+/* tick.d -- Perform an action at regular intervals */
+
+BEGIN
+{
+ i = 0;
+}
+
+profile:::tick-1sec
+{
+ printf("i = %d\n",++i);
+}
+
+END
+{
+ trace(i);
+}
+```
+
+In this example, the program initializes (and thereby implicitly
+declares) the variable `i` when the D program starts,
+increments the variable and prints its value once every second,
+and displays the final value of `i` when the program exits.
+
+When you run this program, it produces output that is similar to
+the following, until you type `Ctrl-C`:
+
+```
+# dtrace -s tick.d
+dtrace: script 'tick.d' matched 3 probes
+CPU ID FUNCTION:NAME
+ 1 5315 :tick-1sec i = 1
+
+ 1 5315 :tick-1sec i = 2
+
+ 1 5315 :tick-1sec i = 3
+
+ 1 5315 :tick-1sec i = 4
+
+ 1 5315 :tick-1sec i = 5
+
+ 1 5315 :tick-1sec i = 6
+
+^C
+ 1 5315 :tick-1sec i = 7
+
+ 0 2 :END 7
+```
+
+To suppress all of the output except the output from
+`printf()` and `trace()`, specify the `-q` option:
+
+```
+# dtrace -q -s tick.d
+i = 1
+i = 2
+i = 3
+i = 4
+^C
+i = 5
+5
+```
+
+#### Exercise: Using tick Probes
+
+List the available `profile` provider probes.
+Experiment with using a different `tick` probe.
+Replace the `trace()` call in
+`END` with a `printf()` call.
+
+See
+[Profile Provider](../userguide/reference/dtrace_providers_profile.md)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
+for a description of the probes.
+
+#### Solution to Exercise and Example: Using tick Probes
+
+```
+# dtrace -l -P profile
+ ID PROVIDER MODULE FUNCTION NAME
+ 5 profile profile-97
+ 6 profile profile-199
+ 7 profile profile-499
+ 8 profile profile-997
+ 9 profile profile-1999
+ 10 profile profile-4001
+ 11 profile profile-4999
+ 12 profile tick-1
+ 13 profile tick-10
+ 14 profile tick-100
+ 15 profile tick-500
+ 16 profile tick-1000
+ 17 profile tick-5000
+5315 profile tick-1sec
+5316 profile tick-10sec
+```
+
+#### Example: Modified Version of tick.d
+
+```
+/* tick1.d -- Modified version of tick.d */
+
+BEGIN
+{
+ i = 0;
+}
+
+/* tick-500ms fires every 500 milliseconds */
+profile:::tick-500ms
+{
+ printf("i = %d\n",++i);
+}
+
+END
+{
+ printf("\nFinal value of i = %d\n",i);
+}
+```
+
+This example uses the `tick-500ms` probe,
+which fires twice per second.
+
+```
+# dtrace -s tick1.d
+dtrace: script 'tick1.d' matched 3 probes
+CPU ID FUNCTION:NAME
+ 2 642 :tick-500ms i = 1
+
+ 2 642 :tick-500ms i = 2
+
+ 2 642 :tick-500ms i = 3
+
+ 2 642 :tick-500ms i = 4
+
+^C
+ 2 642 :tick-500ms i = 5
+
+ 3 2 :END
+Final value of i = 5
+```
+
+### Using Predicates to Select Actions
+
+Predicates are logic statements that choose whether DTrace invokes
+the actions that are associated with a probe. You can use
+predicates to focus tracing analysis on specific contexts under
+which a probe fires.
+
+#### Example: Using daterun.d
+
+The following example shows an executable DTrace script, `daterun.d`, which
+displays the file descriptor, output string, and string length specified to
+the `write()` system call whenever the `date` command is run on the system.
+
+```
+#!/usr/sbin/dtrace -qs
+
+/* daterun.d -- Display arguments to write() when date runs */
+
+syscall::write:entry
+/execname == "date"/
+{
+ printf("%s(%d, %s, %d)\n", probefunc, arg0, copyinstr(arg1, arg2 + 1), arg2);
+}
+```
+
+In the example, the predicate is `/execname == "date"/`, which
+specifies that if the probe `syscall::write:entry` is triggered,
+DTrace runs the associated action only if the name of the executable is `date`.
+The `copyinstr()` string will have room for `arg2+1` bytes,
+including a NUL-terminated byte in case the first `arg2` bytes need it.
+
+Make the script executable by changing its mode:
+
+```
+# chmod +x daterun.d
+```
+
+If you run the script from one window,
+while typing the `date` command in another window,
+output similar to the following is displayed in the first window:
+
+```
+# ./daterun.d
+write(1, Tue Dec 9 11:14:43 GMT 2025
+, 29)
+```
+
+#### Example: Listing Available syscall Provider Probes
+
+The following example shows how you would list available
+`syscall` provider probes.
+
+```
+# dtrace -l -P syscall | less
+ ID PROVIDER MODULE FUNCTION NAME
+ 18 syscall vmlinux read entry
+ 19 syscall vmlinux read return
+ 20 syscall vmlinux write entry
+ 21 syscall vmlinux write return
+ 22 syscall vmlinux open entry
+ 23 syscall vmlinux open return
+ 24 syscall vmlinux close entry
+ 25 syscall vmlinux close return
+ 26 syscall vmlinux newstat entry
+ 27 syscall vmlinux newstat return
+...
+ 648 syscall vmlinux pkey_alloc entry
+ 649 syscall vmlinux pkey_alloc return
+ 650 syscall vmlinux pkey_free entry
+ 651 syscall vmlinux pkey_free return
+ 652 syscall vmlinux statx entry
+ 653 syscall vmlinux statx return
+ 654 syscall vmlinux waitfd entry
+ 655 syscall vmlinux waitfd return
+```
+
+#### Exercise: Using syscall Probes
+
+Experiment by adapting the `daterun.d` script for another program.
+Make the new script produce output when the system is running `w`.
+
+#### Solution to Exercise: Using syscall Probes
+
+```
+#!/usr/sbin/dtrace -qs
+
+/* wrun.d -- Modified version of daterun.d for the w command */
+
+syscall::write:entry
+/execname == "w"/
+{
+ printf("%s(%d, %s, %d)\n", probefunc, arg0, copyinstr(arg1, arg2 + 1), arg2);
+}
+```
+
+We can run the script as follows:
+
+```
+# chmod +x wrun.d
+# ./wrun.d
+write(1, 12:14:55 up 3:21, 3 users, load average: 0.14, 0.15, 0.18
+, 62)
+write(1, USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
+, 61)
+write(1, guest pts/0 :0.0 08:57 7.00s 0.17s 0.03s w
+, 58)
+...
+^C
+```
+
+### Timing Events on a System
+
+Determining the time that a system takes to perform different
+activities is a fundamental technique for analyzing its operation
+and determining where bottlenecks might be occurring.
+
+#### Example: Monitoring read() System Call Duration (readtrace.d)
+
+The following is an example of the D program, `readtrace.d`.
+
+```
+/* readtrace.d -- Display time spent in read() calls */
+
+syscall::read:entry
+{
+ self->t = timestamp; /* Initialize a thread-local variable */
+}
+
+syscall::read:return
+/self->t != 0/
+{
+ printf("%s (pid=%d) spent %d microseconds in read()\n",
+ execname, pid, ((timestamp - self->t)/1000)); /* Divide by 1000 for microseconds */
+
+ self->t = 0; /* Reset the variable */
+}
+```
+
+In the example, the `readtrace.d` program displays the command name,
+process ID, and call duration in microseconds whenever a process
+invokes the `read()` system call.
+The variable `self->t` is *thread-local*,
+meaning that it exists only within the scope of execution of a
+thread on the system.
+
+The program records the value of `timestamp` in `self->t` when
+the process calls `read()`, and subtracts this value from the
+value of `timestamp` when the call returns.
+The units of `timestamp` are nanoseconds,
+so you divide by 1000 to obtain a value in microseconds.
+
+The following is output from running this program:
+
+```
+# dtrace -q -s readtrace.d
+NetworkManager (pid=878) spent 10 microseconds in read()
+NetworkManager (pid=878) spent 9 microseconds in read()
+NetworkManager (pid=878) spent 2 microseconds in read()
+in:imjournal (pid=815) spent 63 microseconds in read()
+gdbus (pid=878) spent 7 microseconds in read()
+gdbus (pid=878) spent 66 microseconds in read()
+gdbus (pid=878) spent 63 microseconds in read()
+irqbalance (pid=816) spent 56 microseconds in read()
+irqbalance (pid=816) spent 113 microseconds in read()
+irqbalance (pid=816) spent 104 microseconds in read()
+irqbalance (pid=816) spent 91 microseconds in read()
+irqbalance (pid=816) spent 61 microseconds in read()
+irqbalance (pid=816) spent 63 microseconds in read()
+irqbalance (pid=816) spent 61 microseconds in read()
+irqbalance (pid=816) spent 61 microseconds in read()
+irqbalance (pid=816) spent 61 microseconds in read()
+irqbalance (pid=816) spent 61 microseconds in read()
+irqbalance (pid=816) spent 61 microseconds in read()
+irqbalance (pid=816) spent 61 microseconds in read()
+sshd (pid=10230) spent 8 microseconds in read()
+in:imjournal (pid=815) spent 6 microseconds in read()
+sshd (pid=10230) spent 7 microseconds in read()
+in:imjournal (pid=815) spent 5 microseconds in read()
+sshd (pid=10230) spent 7 microseconds in read()
+in:imjournal (pid=815) spent 6 microseconds in read()
+sshd (pid=10230) spent 7 microseconds in read()
+in:imjournal (pid=815) spent 5 microseconds in read()
+^C
+```
+
+#### Exercise: Timing System Calls
+
+Add a predicate to the `entry` probe in `readtrace.d` so that `dtrace`
+displays results for a disk space usage report that is selected
+by the name of its executable (`df`).
+
+#### Solution to Exercise: Timing System Calls
+
+The following example shows a modified version of the
+`readtrace.d` program that includes a predicate.
+
+```
+/* readtrace1.d -- Modified version of readtrace.d that includes a predicate */
+
+syscall::read:entry
+/execname == "df"/
+{
+ self->t = timestamp;
+}
+
+syscall::read:return
+/self->t != 0/
+{
+ printf("%s (pid=%d) spent %d microseconds in read()\n",
+ execname, pid, ((timestamp - self->t)/1000));
+
+ self->t = 0; /* Reset the variable */
+}
+```
+
+The predicate `/execname == "df"/` tests whether the
+`df` program is running when the probe fires.
+
+```
+# dtrace -q -s readtrace1.d
+df (pid=1666) spent 6 microseconds in read()
+df (pid=1666) spent 8 microseconds in read()
+df (pid=1666) spent 1 microseconds in read()
+df (pid=1666) spent 50 microseconds in read()
+df (pid=1666) spent 38 microseconds in read()
+df (pid=1666) spent 10 microseconds in read()
+df (pid=1666) spent 1 microseconds in read()
+^C
+```
+
+#### Exercise: Timing All System Calls for cp (calltrace.d)
+
+Using the `probefunc` variable and the `syscall:::entry` and
+`syscall:::return` probes, create a D program, `calltrace.d`,
+which times all system calls for the executable `cp`.
+
+#### Solution to Exercise: Timing All System Calls for cp (calltrace.d)
+
+```
+/* calltrace.d -- Time all system calls for cp */
+
+syscall:::entry
+/execname == "cp"/
+{
+ self->t = timestamp; /* Initialize a thread-local variable */
+}
+
+syscall:::return
+/self->t != 0/
+{
+ printf("%s (pid=%d) spent %d microseconds in %s()\n",
+ execname, pid, ((timestamp - self->t)/1000), probefunc);
+
+ self->t = 0; /* Reset the variable */
+}
+```
+
+Dropping the function name `read` from the probe specifications
+matches all instances of `entry` and `return` probes for `syscall`.
+The following is a check for system calls resulting from running
+the `cp` executable:
+
+```
+# dtrace -q -s calltrace.d
+cp (pid=2801) spent 4 microseconds in brk()
+cp (pid=2801) spent 5 microseconds in mmap()
+cp (pid=2801) spent 15 microseconds in access()
+cp (pid=2801) spent 7 microseconds in open()
+cp (pid=2801) spent 2 microseconds in newfstat()
+cp (pid=2801) spent 3 microseconds in mmap()
+cp (pid=2801) spent 1 microseconds in close()
+cp (pid=2801) spent 8 microseconds in open()
+cp (pid=2801) spent 3 microseconds in read()
+cp (pid=2801) spent 1 microseconds in newfstat()
+cp (pid=2801) spent 4 microseconds in mmap()
+cp (pid=2801) spent 12 microseconds in mprotect()
+ ...
+cp (pid=2801) spent 183 microseconds in open()
+cp (pid=2801) spent 1 microseconds in newfstat()
+cp (pid=2801) spent 1 microseconds in fadvise64()
+cp (pid=2801) spent 17251 microseconds in read()
+cp (pid=2801) spent 80 microseconds in write()
+cp (pid=2801) spent 58 microseconds in read()
+cp (pid=2801) spent 57 microseconds in close()
+cp (pid=2801) spent 85 microseconds in close()
+cp (pid=2801) spent 57 microseconds in lseek()
+cp (pid=2801) spent 56 microseconds in close()
+cp (pid=2801) spent 56 microseconds in close()
+cp (pid=2801) spent 56 microseconds in close()
+^C
+```
+
+### Tracing Parent and Child Processes
+
+When a process forks, it creates a child process that is effectively
+a copy of its parent process, but with a different process ID.
+For information about other differences, see the `fork(2)` manual page.
+The child process can either run independently from its parent process
+to perform some separate task.
+Or, a child process can execute a new program image that replaces
+the child's program image while retaining the same process ID.
+
+#### Example: Using proc Probes to Report Activity on a System (activity.d)
+
+The D program `activity.d` in the following example uses `proc`
+probes to report `fork()` and `exec()` activity on a system.
+
+```
+#pragma D option quiet
+
+/* activity.d -- Record fork() and exec() activity */
+
+proc:::create
+{
+ /* Extract PID of child process from the psinfo_t pointed to by args[0] */
+ childpid = args[0]->pr_pid;
+
+ time[childpid] = timestamp;
+ p_pid[childpid] = pid; /* Current process ID (parent PID of new child) */
+ p_name[childpid] = execname; /* Parent command name */
+ p_exec[childpid] = ""; /* Child has not yet been exec'ed */
+}
+
+proc:::exec
+/p_pid[pid] != 0/
+{
+ p_exec[pid] = args[0]; /* Child process path name */
+}
+
+proc:::exit
+/p_pid[pid] != 0 && p_exec[pid] != ""/
+{
+ printf("%s (%d) executed %s (%d) for %d microseconds\n",
+ p_name[pid], p_pid[pid], p_exec[pid], pid, (timestamp - time[pid])/1000);
+}
+
+proc:::exit
+/p_pid[pid] != 0 && p_exec[pid] == ""/
+{
+ printf("%s (%d) forked itself (as %d) for %d microseconds\n",
+ p_name[pid], p_pid[pid], pid, (timestamp - time[pid])/1000);
+}
+```
+
+In the example, the statement `#pragma D option quiet` has the same
+effect as specifying the `-q` option on the command line.
+
+The process ID of the child process (`childpid`), following a `fork()`,
+is determined by examining the `pr_pid` member of the `psinfo_t` data
+structure that is pointed to by the `args[0]` probe argument.
+For more information about the arguments to `proc` probes, see
+[Proc Provider](../userguide/reference/dtrace_providers_proc.md)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md).
+
+The program uses the value of the child process ID to initialize
+globally unique associative array entries, such as `p_pid[childpid]`.
+
+>Note:
+>
+>An *associative array* is similar to a normal array, in that it associates
+keys with values, but the keys can be of any type; they need not be integers.
+
+When you run the program, you should see output similar to the following as you
+use the `ssh` command to access the same system from another terminal window.
+You might want to try running different programs from this new terminal
+window to generate additional output:
+
+```
+# dtrace -s activity.d
+sshd (3966) forked itself (as 3967) for 3667020 microseconds
+bash (3971) forked itself (as 3972) for 1718 microseconds
+bash (3973) executed /usr/bin/hostname (3974) for 1169 microseconds
+grepconf.sh (3975) forked itself (as 3976) for 1333 microseconds
+bash (3977) forked itself (as 3978) for 967 microseconds
+bash (3977) executed /usr/bin/tput (3979) for 1355 microseconds
+bash (3980) executed /usr/bin/dircolors (3981) for 1212 microseconds
+sshd (3966) executed /usr/sbin/unix_chkpwd (3968) for 31444 microseconds
+sshd (3966) executed /usr/sbin/unix_chkpwd (3969) for 1653 microseconds
+bash (3970) forked itself (as 3971) for 2411 microseconds
+bash (3970) forked itself (as 3973) for 1830 microseconds
+bash (3970) executed /usr/libexec/grepconf.sh (3975) for 3696 microseconds
+bash (3970) forked itself (as 3977) for 3273 microseconds
+bash (3970) forked itself (as 3980) for 1928 microseconds
+bash (3970) executed /usr/bin/grep (3982) for 1570 microseconds
+^C
+```
+
+#### Exercise: Using a Predicate to Control the Execution of an Action
+
+Modify `activity.d` so that `dtrace` displays results for parent processes
+that are selected by their executable name, for example, `bash`,
+or by a program name that you specify as an argument to the `dtrace` command.
+
+#### Solution to Exercise: Using a Predicate to Control the Execution of an Action
+
+The only change that is required to specify the name of an executable
+is to add a predicate to the `proc::_:create` probe, for example:
+
+```
+/execname == "bash"/
+```
+
+A more generic version of the program sets the predicate check
+value from a passed-in command-line argument instead, for example:
+
+```
+/execname == $1/
+```
+
+#### Example: Recording fork() and exec() Activity for a Specified Program (activity1.d)
+
+The following example uses a predicate that is passed in from the command line.
+
+```
+#pragma D option quiet
+
+/* activity1.d -- Record fork() and exec() activity for a specified program */
+
+proc:::create
+/execname == $1/
+{
+ /* Extract PID of child process from the psinfo_t pointed to by args[0] */
+ childpid = args[0]->pr_pid;
+
+ time[childpid] = timestamp;
+ p_pid[childpid] = pid; /* Current process ID (parent PID of new child) */
+ p_name[childpid] = execname; /* Parent command name */
+ p_exec[childpid] = ""; /* Child has not yet been exec'ed */
+}
+
+proc:::exec
+/p_pid[pid] != 0/
+{
+ p_exec[pid] = args[0]; /* Child process path name */
+}
+
+proc:::exit
+/p_pid[pid] != 0 && p_exec[pid] != ""/
+{
+ printf("%s (%d) executed %s (%d) for %d microseconds\n",
+ p_name[pid], p_pid[pid], p_exec[pid], pid, (timestamp - time[pid])/1000);
+}
+
+proc:::exit
+/p_pid[pid] != 0 && p_exec[pid] == ""/
+{
+ printf("%s (%d) forked itself (as %d) for %d microseconds\n",
+ p_name[pid], p_pid[pid], pid, (timestamp - time[pid])/1000);
+}
+```
+
+As shown in the following example, you can now specify the name
+of the program to be traced as an argument to the `dtrace` command.
+Note that you need to escape the argument to protect the double quotes
+from the shell:
+
+```
+# dtrace -s activity.d '"bash"'
+bash (10367) executed /bin/ps (10368) for 10926 microseconds
+bash (10360) executed /usr/bin/tty (10361) for 3046 microseconds
+bash (10359) forked itself (as 10363) for 32005 microseconds
+bash (10366) executed /bin/basename (10369) for 1285 microseconds
+bash (10359) forked itself (as 10370) for 12373 microseconds
+bash (10360) executed /usr/bin/tput (10362) for 34409 microseconds
+bash (10363) executed /usr/bin/dircolors (10364) for 29527 microseconds
+bash (10359) executed /bin/grep (10365) for 21024 microseconds
+bash (10366) forked itself (as 10367) for 11749 microseconds
+bash (10359) forked itself (as 10360) for 41918 microseconds
+bash (10359) forked itself (as 10366) for 14197 microseconds
+bash (10370) executed /usr/bin/id (10371) for 11729 microseconds
+^C
+```
+
+### Simple Data Aggregations
+
+DTrace provides several functions for aggregating the data that individual
+probes gather.
+These functions include `avg()`, `count()`, `max()`, `min()`, `stddev()`,
+and `sum()`, which return the average (mean), count (number), maximum value,
+minimum value, standard deviation, and summation of the data being gathered,
+respectively.
+See
+[Aggregations](../userguide/reference/aggregation.md)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
+for descriptions of aggregation functions.
+
+DTrace indexes the results of an aggregation by using a tuple
+expression similar to what is used for an associative array:
+
+```
+@name[list_of_keys] = aggregating_function(args);
+```
+
+The name of the aggregation is prefixed with an `@` character.
+The keys describe the data that the aggregating function is collecting.
+If you do not specify a name for the aggregation, DTrace uses `@` as an
+anonymous aggregation name, which is usually sufficient for simple D programs.
+
+#### Example: Counting the Number of write() System Calls Invoked by Processes
+
+In the following example, the command counts the number of `write()`
+system calls that are invoked by processes, until you type `Ctrl-C`.
+
+```
+# dtrace -n 'syscall::write:entry { @["write() calls"] = count(); }'
+dtrace: description 'syscall:::' matched 1 probe
+^C
+
+ write() calls 6932
+```
+
+>Note:
+>
+>Rather than create a separate D script for this simple example,
+the probe and the action is specified on the `dtrace` command line.
+
+DTrace prints the result of the aggregation automatically.
+Alternatively, you can use the `printa()`
+function to format the result of the aggregation.
+
+#### Example: Counting the Number of read() and write() System Calls
+
+The following example counts the number of both
+`read()` and `write()` system calls.
+
+```
+# dtrace -n 'syscall::write:entry,syscall::read:entry { @[strjoin(probefunc,"() calls")] = count(); }'
+dtrace: description 'syscall::write:entry,syscall::read:entry' matched 2 probes
+^C
+
+ write() calls 150
+ read() calls 1555
+```
+
+#### Exercise: Counting System Calls Over a Fixed Period
+
+Write a D program named `countcalls.d` that uses a `tick` probe and `exit()`
+to stop collecting data after 100 seconds and display the number of `open()`,
+`read()` and `write()` calls.
+
+#### Solution to Exercise and Example: Counting Write, Read, and Open System Calls Over 100 Seconds (countcalls.d)
+
+```
+/* countcalls.d -- Count write, read, and open system calls over 100 seconds */
+
+profile:::tick-100sec
+{
+ exit(0);
+}
+
+syscall::write:entry, syscall::read:entry, syscall::open:entry
+{
+ @[strjoin(probefunc,"() calls")] = count();
+}
+```
+
+The action that is associated with the `tick-100s` probe means that
+`dtrace` exits after 100 seconds. By default, exit will print the
+results of the aggregation.
+
+```
+# dtrace -s countcalls.d
+dtrace: script 'countcalls.d' matched 4 probes
+CPU ID FUNCTION:NAME
+ 3 643 :tick-100sec
+
+ write() calls 1062
+ open() calls 1672
+ read() calls 29672
+```
+
+#### Example: Counting System Calls Invoked by a Process (countsyscalls.d)
+
+The D program `countsyscalls.d` shown in the following example counts
+the number of times a process that is specified by its process ID
+invokes different system calls.
+
+```
+#!/usr/sbin/dtrace -qs
+
+/* countsyscalls.d -- Count system calls invoked by a process */
+
+syscall:::entry
+/pid == $1/
+{
+ @num[probefunc] = count();
+}
+```
+
+After making the `syscalls.d` file executable,
+you can run it from the command line,
+specifying a process ID as its argument.
+
+The following example shows how you would monitor the use of the
+`emacs` program that was previously invoked.
+After the script is invoked, within `emacs` a couple files are opened,
+modified, and then saved before exiting the D script.
+
+Make the script executable:
+
+```
+# chmod +x countsyscalls.d
+```
+
+From another command line, type:
+
+```
+# emacs foobar.txt
+```
+
+Now, start the script and use the opened `emacs` window:
+
+```
+# ./countsyscalls.d $(pgrep -u root emacs)
+ ^C
+
+ chmod 1
+ exit_group 1
+ futex 1
+ getpgrp 1
+ lseek 1
+ lsetxattr 1
+ rename 1
+ fsync 2
+ lgetxattr 2
+ alarm 3
+ rt_sigaction 3
+ unlink 3
+ mmap 4
+ munmap 4
+ symlink 4
+ fcntl 6
+ newfstat 6
+ getgid 7
+ getuid 7
+ geteuid 8
+ openat 8
+ access 9
+ getegid 14
+ open 14
+ getdents 15
+ close 17
+ readlink 19
+ newlstat 33
+ newstat 155
+ read 216
+ timer_settime 231
+ write 314
+ pselect6 376
+ rt_sigreturn 393
+ ioctl 995
+ rt_sigprocmask 1261
+ clock_gettime 3495
+```
+
+In the preceding example, the `pgrep` command is used to determine the
+process ID of the `emacs` program that the `root` user is running.
+
+#### Exercise: Tracing Processes That Are Run by a User
+
+Create a program `countprogs.d` that counts and displays the number of
+times a user (specified by their user name) runs different programs.
+You can use the `id -u` command to obtain the ID that corresponds to a user name.
+
+#### Solution to Exercise and Example: Counting Programs Invoked by a Specified User (countprogs.d)
+
+```
+#!/usr/sbin/dtrace -qs
+
+/* countprogs.d -- Count programs invoked by a specified user */
+
+proc:::exec
+/uid == $1/
+{
+ @num[execname] = count();
+}
+```
+
+The predicate `/uid == $1/` compares the effective UID for each program
+that is run against the argument specified on the command line.
+You can use the `id -u` command to find out the ID of the guest user account, for example:
+
+```
+# chmod +x countprogs.d
+# ./countprogs.d $(id -u guest)
+^C
+
+less 1
+lesspipe.sh 1
+sh 1
+bash 9
+```
+
+You can use the same command for the `root` user, which is typically user `0`.
+For testing purposes, you might want to have the user account under a test
+login by using another window and then run some nominal programs.
+
+#### Example: Counting the Number of Times a Program Reads From Different Files in 10 Seconds (fdscount.d)
+
+The following D program counts the number of times
+a program reads from files in a ten-second period
+and displays at most the top five results.
+
+```
+/* fdscount.d -- Count read system calls for 10 seconds */
+
+syscall::read:entry
+/execname == ENAME/
+{
+ @[fds[arg0].fi_name] = count();
+}
+
+profile:::tick-10sec {
+ exit(0);
+}
+
+END
+{
+ trunc(@, 5);
+}
+```
+
+We can run the script as follows:
+
+```
+# dtrace -C -D ENAME='"df"' -qs fdscount.d
+
+ libc.so.6 1
+ locale.alias 2
+ mountinfo 5
+```
+
+You specify a C preprocessor directive to `dtrace` that sets
+the value of the `ENAME` variable, such as to `df`.
+You must use additional single quotes to escape the string quotes.
+
+Use the `fds[]` built-in array to determine which file
+corresponds to the file descriptor argument `arg0` to `read()`.
+The `fi_name` member of the `fileinfo_t` structure `fds[arg0]`
+contains the basename of the file.
+See
+[`fds`](../userguide/reference/dtrace_builtin_variable_reference.md#dt_ref_var_fds)
+and
+[`fileinfo_t`](../userguide/reference/dtrace_providers_io.md#dt_ref_iofile_prov)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
+for more information.
+
+The [`trunc()`](../userguide/reference/function_trunc.md) function
+in the `END` action removes all but the 5 most important keys.
+Here, there are fewer than 5 keys to start with.
+
+#### Exercise: Counting Context Switches on a System
+
+Create an executable D program named `cswpercpu.d` that displays
+a timestamp and prints the number of context switches per CPU and
+the total for all CPUs once per second, together with the CPU
+number or `"total"`.
+
+- Using the `BEGIN` probe, print a header for the display
+ with columns labelled `Timestamp`, `CPU`, and `Ncsw`.
+
+- Using the `sched:::on-cpu` probe to detect the end of a context switch,
+ use `count()` to increment the aggregation variable `@n`, once with the
+ key value set to the CPU number and once with the key value set to `"total"`.
+ Since `"total"` is a string and aggregation keys must have consistent types,
+ the CPU keys are also converted to strings, using `lltostr()`.
+
+ See
+ [Sched Provider](../userguide/reference/dtrace_providers_sched.md)
+ in the
+ [Oracle Linux: DTrace Reference Guide](../userguide/index.md)
+ for a description of the `sched:::on-cpu`
+ probe.
+
+- Using the `profile:::tick-1sec` probe, use `printf()` to
+ print the data and time, use `printa()` to print the key
+ (the CPU number string or `"total"`) and the aggregation value.
+ The date and time are available as the value of `walltimestamp` variable,
+ which you can print using the `%Y` conversion format
+
+- Use `clear()` to reset the aggregation variable `@n`.
+
+#### Solution to Exercise and Example: Counting Context Switches on a System
+
+The following example shows the executable D program `cswpercpu.d`.
+The program displays a timestamp and prints the number of context switches,
+per-CPU, and the total for all CPUs, once per second, together with the CPU
+number or `"total"`:
+
+```
+#!/usr/sbin/dtrace -qs
+
+/* cswpercpu.d -- Print number of context switches per CPU once per second */
+
+#pragma D option quiet
+
+dtrace:::BEGIN
+{
+ /* Print the header */
+ printf("%-25s %5s %15s", "Timestamp", "CPU", "Ncsw");
+}
+
+sched:::on-cpu
+{
+ /* Convert the cpu number to a string */
+ cpustr = lltostr(cpu);
+ /* Increment the counters */
+ @n[cpustr] = count();
+ @n["total"] = count();
+}
+
+profile:::tick-1sec
+{
+ /* Print the date and time before the first result */
+ printf("\n%-25Y ", walltimestamp);
+
+ /* Print the aggregated counts for each CPU and the total for all CPUs */
+ printa("%5s %@15d\n ", @n);
+
+ /* Reset the aggregation */
+ clear(@n);
+}
+```
+
+```
+# chmod +x cswpercpu.d
+# ./cswpercpu.d
+Timestamp CPU Ncsw
+2013 Nov 6 20:47:26 1 148
+ 0 155
+ 3 200
+ 2 272
+ total 775
+
+2013 Nov 6 20:47:27 1 348
+ 0 364
+ 3 364
+ 2 417
+ total 1493
+
+2013 Nov 6 20:47:28 3 47
+ 1 100
+ 0 121
+ 2 178
+ total 446
+ ^C
+```
+
+You might want to experiment with aggregating the total time that
+is spent context switching and the average time per context switch.
+For example, you can experiment by initializing a thread-local variable
+to the value of `timestamp` in the action to a `sched:::off-cpu` probe,
+and subtracting this value from the value of `timestamp`
+in the action to `sched:::on-cpu`.
+Use the `sum()` and `avg()` aggregation functions, respectively.
+
+### Working With More Complex Data Aggregations
+
+Use the `lquantize()` and `quantize()` functions to display
+linear and power-of-two frequency distributions of data.
+See
+[Aggregations](../userguide/reference/aggregation.md)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
+for a description of aggregation functions.
+
+#### Example: Displaying the Distribution of Read Sizes Resulting From a Command
+
+As shown in the following example, you can display the
+distribution of the sizes specified to `arg2` of `read()` calls
+that were invoked by all instances of `find` that are running.
+After running the script, start a search with `find`
+in another window, such as `find .` or `find /.`.
+
+```
+# dtrace -n 'syscall::read:entry /execname=="find"/{@dist["find"]=quantize(arg2);}'
+dtrace: description 'syscall::read:entry ' matched 1 probe
+^C
+
+ find
+ value ------------- Distribution ------------- count
+ 256 | 0
+ 512 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 6
+ 1024 | 0
+ 2048 | 0
+ 4096 |@@@@@@@@@@ 2
+ 8192 | 0
+```
+
+When the program is as simple as in this example,
+it is often convenient to run it from the command line.
+
+#### Example: Displaying the Distribution of I/O Throughput for Block Devices (diskact.d)
+
+In the following example, the `diskact.d` script uses `io`
+provider probes to display the distribution of I/O throughput
+for the block devices on the system.
+
+```
+#pragma D option quiet
+
+/* diskact.d -- Display the distribution of I/O throughput for block devices */
+
+io:::start
+{
+ start[args[0]->b_edev, args[0]->b_blkno] = timestamp;
+}
+
+io:::done
+/start[args[0]->b_edev, args[0]->b_blkno]/
+{
+ /*
+ You want to get an idea of throughput to this device in KB/sec
+ but you have values that are measured in bytes and nanoseconds.
+ You want to calculate the following:
+
+ bytes / 1024
+ ------------------------
+ nanoseconds / 1000000000
+
+ As DTrace uses integer arithmetic and the denominator is usually
+ between 0 and 1 for most I/O, the calculation as shown will lose
+ precision. So, restate the fraction as:
+
+ bytes 1000000000 bytes * 976562
+ ----------- * ------------- = --------------
+ nanoseconds 1024 nanoseconds
+
+ This is easy to calculate using integer arithmetic.
+ */
+ this->elapsed = timestamp - start[args[0]->b_edev, args[0]->b_blkno];
+ @[args[1]->dev_statname, args[1]->dev_pathname] =
+ quantize((args[0]->b_bcount * 976562) / this->elapsed);
+ start[args[0]->b_edev, args[0]->b_blkno] = 0;
+}
+
+END
+{
+ printa(" %s (%s)\n%@d\n", @);
+}
+```
+
+The `#pragma D option quiet` statement is used
+to suppress unwanted output and the `printa()`
+function is used to display the results of the aggregation.
+
+See
+[IO Provider](../userguide/reference/dtrace_providers_io.md)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
+for a description of the arguments to the
+`io:::start` and `io:::done` probes.
+
+See
+[printa](../userguide/reference/function_printa.md)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md)
+for a description of the function.
+
+After running the program for approximately a minute, type
+`Ctrl-C` to display the results:
+
+```
+# dtrace -s diskact.d
+ ^C
+
+xvda2 (<unknown>)
+
+ value ------------- Distribution ------------- count
+ -1 | 0
+ 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 3
+ 1 | 0
+
+xvdc (<unknown>)
+
+ value ------------- Distribution ------------- count
+ -1 | 0
+ 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 3
+ 1 | 0
+
+xvdc1 (<unknown>)
+
+ value ------------- Distribution ------------- count
+ -1 | 0
+ 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 3
+ 1 | 0
+
+ dm-0 (<unknown>)
+
+ value ------------- Distribution ------------- count
+ 256 | 0
+ 512 |@@ 1
+ 1024 |@@ 1
+ 2048 |@@@@@@ 3
+ 4096 |@@@@@@@@@@ 5
+ 8192 |@@@@@@@@@@@@@@@@@ 9
+ 16384 |@@@@ 2
+ 32768 | 0
+```
+
+#### Exercise: Displaying Read and Write I/O Throughput Separately
+
+Create a version of `diskact.d` that aggregates the results
+separately for reading from, and writing to, block devices.
+Use a `tick` probe to collect data for 10 seconds.
+
+- In the actions for `io:::start` and `io:::done`,
+ assign the value of `args[0]->b_flags & B_READ ? "READ" : "WRITE"`
+ to the clause-local variable `this->iodir`.
+
+- In the actions for `io:::start` and `io:::done`,
+ add `this->iodir` as a key to the `start[]` associative array.
+
+- In the action for `io:::done`,
+ add `this->iodir` as a key to the anonymous aggregation variable `@[]`.
+
+- Modify the format string for `printa()` to display the value of the `iodir` key.
+
+#### Solution to Exercise: Displaying Read and Write I/O Throughput Separately
+
+The following example shows a modified version of the `diskact.d` script,
+which displays separate results for read and write I/O:
+
+```
+#pragma D option quiet
+
+/* rwdiskact.d -- Modified version of diskact.d that displays
+ separate results for read and write I/O */
+
+profile:::tick-10sec
+{
+ exit(0);
+}
+
+io:::start
+{
+ this->iodir = args[0]->b_flags & B_READ ? "READ" : "WRITE";
+ start[args[0]->b_edev, args[0]->b_blkno, this->iodir] = timestamp;
+}
+
+io:::done
+/
+ (this->iodir = args[0]->b_flags & B_READ ? "READ" : "WRITE") != NULL
+ &&
+ start[args[0]->b_edev, args[0]->b_blkno, this->iodir] != 0
+/
+{
+ this->elapsed = timestamp - start[args[0]->b_edev, args[0]->b_blkno, this->iodir];
+ @[args[1]->dev_statname, args[1]->dev_pathname, this->iodir] =
+ quantize((args[0]->b_bcount * 976562) / this->elapsed);
+ start[args[0]->b_edev, args[0]->b_blkno, this->iodir] = 0;
+}
+
+END
+{
+ printa(" %s (%s) %s \n%@d\n", @);
+}
+```
+
+In the example, adding the `this->iodir` variable to the
+tuple in the aggregation variable enables DTrace to display
+separate aggregations for read and write I/O operations.
+
+```
+# dtrace -s rwdiskact.d
+
+^C
+ xvda2 (<unknown>) WRITE
+
+ value ------------- Distribution ------------- count
+ -1 | 0
+ 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
+ 1 | 0
+
+ xvdc (<unknown>) WRITE
+
+ value ------------- Distribution ------------- count
+ -1 | 0
+ 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
+ 1 | 0
+
+ xvdc1 (<unknown>) WRITE
+
+ value ------------- Distribution ------------- count
+ -1 | 0
+ 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
+ 1 | 0
+
+ nfs (<nfs>) READ
+
+ value ------------- Distribution ------------- count
+ -1 | 0
+ 0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 5
+ 1 | 0
+
+ dm-0 (<unknown>) WRITE
+
+ value ------------- Distribution ------------- count
+ 4096 | 0
+ 8192 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
+ 16384 | 0
+```
+
+#### Example: Displaying Cumulative Read and Write Activity Across a File System Device (fsact)
+
+The following example is a `bash` shell script that uses
+an embedded D program to display cumulative read and write
+block counts for a local file system according to their
+location on the file system's underlying block device.
+The `lquantize()` aggregation function is used to
+display the results linearly as tenths of the total
+number of blocks on the device.
+
+```
+#!/bin/bash
+
+# fsact -- Display cumulative read and write activity across a file system device
+#
+# Usage: fsact [<filesystem>]
+
+# If no file system is specified, assume /
+[ $# -eq 1 ] && FSNAME=$1 || FSNAME="/"
+[ ! -e $FSNAME ] && echo "$FSNAME not found" && exit 1
+
+# Determine the mountpoint, major and minor numbers, and file system size
+MNTPNT=$(df $FSNAME | gawk '{ getline; print $1; exit }')
+MAJOR=$(printf "%d\n" 0x$(stat -Lc "%t" $MNTPNT))
+MINOR=$(printf "%d\n" 0x$(stat -Lc "%T" $MNTPNT))
+FSSIZE=$(stat -fc "%b" $FSNAME)
+
+# Run the embedded D program
+dtrace -qs /dev/stdin << EOF
+io:::done
+/args[1]->dev_major == $MAJOR && args[1]->dev_minor == $MINOR/
+{
+ this->iodir = args[0]->b_flags & B_READ ? "READ" : "WRITE";
+ /* Normalize the block number as an integer in the range 0 to 10 */
+ blkno = (args[0]->b_blkno)*10/$FSSIZE;
+ /* Aggregate blkno linearly over the range 0 to 10 in steps of 1 */
+ @a[this->iodir] = lquantize(blkno,0,10,1)
+}
+
+tick-10sec
+{
+ printf("%Y\n",walltimestamp);
+ /* Display the results of the aggregation */
+ printa("%s\n%@d\n",@a);
+ /* To reset the aggregation every tick, uncomment the following line */
+ /* clear(@a); */
+}
+EOF
+```
+
+You embed the D program in a shell script so that you can
+set up the parameters that are needed, which are the major
+and minor numbers of the underlying device and the total
+size of the file system in file system blocks.
+You then access these parameters directly in the D code.
+
+>Note:
+>
+>An alternate way of passing values into the D program is to
+use C preprocessor directives, for example:
+
+```
+dtrace -C -D MAJ=$MAJOR -D MIN=$MINOR -D FSZ=$FSSIZE -qs /dev/stdin << EOF
+```
+
+You can then refer to the variables in the D program by their
+macro names instead of their shell names:
+
+```
+/args[1]->dev_major == MAJ && args[1]->dev_minor == MIN/
+
+blkno = (args[0]->b_blkno)*10/FSZ;
+```
+
+The following example shows output from running the `fsact`
+command after making the script executable,
+then running `cp -R` on a directory and `rm -rf` on the copied directory:
+
+```
+# chmod +x fsact
+# ./fsact
+2018 Feb 16 16:59:46
+READ
+
+ value ------------- Distribution ------------- count
+ < 0 | 0
+ 0 |@@@@@@@ 8
+ 1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 32
+ 2 | 0
+ 3 | 0
+ 4 | 0
+ 5 | 0
+ 6 | 0
+ 7 | 0
+ 8 | 0
+ 9 | 0
+ >= 10 |@@@@@@@ 8
+
+WRITE
+
+ value ------------- Distribution ------------- count
+ 9 | 0
+ >= 10 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 42 0
+
+^C
+```
+
+### Displaying System Call Errors
+
+The following information pertains to using the D program
+`errno.d` to display system call errors.
+
+#### Example: Displaying System Call Errors (`errno.d`)
+
+The following is an example of the D program, `errno.d`.
+In this example, the program displays the value of `errno`
+and the file name if an error occurs when using the `open()`
+system call to open a file.
+
+```
+#!/usr/sbin/dtrace -qs
+
+/* errno.d -- Display errno and the file name for failed open() calls */
+
+syscall::open:entry
+{
+ self->filename = copyinstr(arg0);
+}
+
+syscall::openat*:entry
+{
+ self->filename = copyinstr(arg1);
+}
+
+syscall::open*:return
+/arg0 < 0 && self->filename != NULL/
+{
+ printf("errno = %-2d file = %s\n", errno, self->filename);
+}
+
+syscall::open*:return
+{
+ self->filename = NULL;
+}
+```
+
+If an error occurs in the `open()` system call, the `return` probe
+sets the `arg0` argument to `-1` and the value of the built-in
+`errno` variable indicates the nature of the error.
+A predicate is used to test the value of `arg0`.
+Alternatively, you could test whether the value of `errno` is greater than zero.
+
+The dynamic variable `self->filename` should be freed (set to 0,
+or in this case `NULL`) as a habit to reduce memory leaks.
+
+When you have saved this script to a file and made the file executable,
+you can then run it to display information about any failures of the
+`open()` system call that occur on the system.
+After you have started the script, in a separate terminal window,
+you can run commands that result in an error, such as running the
+`ls` command to list a file that does not exist.
+Or, as in the following example, from another terminal the `cat`
+command has been issued on a directory, which results in an error:
+
+```
+# ./errno.d
+
+errno = 2 file = /usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo
+errno = 2 file = /usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo
+errno = 2 file = /usr/share/locale/en_US/LC_MESSAGES/libc.mo
+errno = 2 file = /usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo
+errno = 2 file = /usr/share/locale/en.utf8/LC_MESSAGES/libc.mo
+errno = 2 file = /usr/share/locale/en/LC_MESSAGES/libc.mo
+^C
+```
+
+#### Exercise: Displaying More Information About System Call Errors
+
+Adapt `errno.d` to display the name of the
+error instead of its number for any failed system call.
+
+- The numeric values of errors such as `EACCES` and `EEXIST`
+ are defined in `/usr/include/asm-generic/errno-base.h` and
+ `/usr/include/asm-generic/errno.h`.
+ DTrace defines inline names (which are effectively constants)
+ for the numeric error values in `/usr/lib64/dtrace/*/errno.d`.
+ Use an associative array named `error[]` to store the mapping
+ between the inline names and the error names that are defined
+ in `/usr/include/asm-generic/errno-base.h`.
+
+- Use `printf()` to display the user ID, the process ID, the
+ program name, the error name, and the name of the system call.
+
+- Use the `BEGIN` probe to print column headings.
+
+- Use the value of `errno` rather than `arg0` to test whether an
+ error from the range of mapped names has occurred in a system call.
+
+#### Solution to Exercise: Displaying More Information About System Call Errors
+
+The following is an example that shows a modified version of `errno.d`,
+which displays error names.
+
+##### Example: Modified Version of errno.d Displaying Error Names (displayerrno.d)
+
+```
+#!/usr/sbin/dtrace -qs
+
+/* displayerrno.d -- Modified version of errno.d that displays error names */
+
+BEGIN
+{
+ printf("%-4s %-6s %-10s %-10s %s\n", "UID", "PID", "Prog", "Error", "Func");
+
+ /* Assign error names to the associative array error[] */
+ error[EPERM] = "EPERM"; /* Operation not permitted */
+ error[ENOENT] = "ENOENT"; /* No such file or directory */
+ error[ESRCH] = "ESRCH"; /* No such process */
+ error[EINTR] = "EINTR"; /* Interrupted system call */
+ error[EIO] = "EIO"; /* I/O error */
+ error[ENXIO] = "ENXIO"; /* No such device or address */
+ error[E2BIG] = "E2BIG"; /* Argument list too long */
+ error[ENOEXEC] = "ENOEXEC"; /* Exec format error */
+ error[EBADF] = "EBADF"; /* Bad file number */
+ error[ECHILD] = "ECHILD"; /* No child processes */
+ error[EAGAIN] = "EAGAIN"; /* Try again or operation would block */
+ error[ENOMEM] = "ENOMEM"; /* Out of memory */
+ error[EACCES] = "EACCES"; /* Permission denied */
+ error[EFAULT] = "EFAULT"; /* Bad address */
+ error[ENOTBLK] = "ENOTBLK"; /* Block device required */
+ error[EBUSY] = "EBUSY"; /* Device or resource busy */
+ error[EEXIST] = "EEXIST"; /* File exists */
+ error[EXDEV] = "EXDEV"; /* Cross-device link */
+ error[ENODEV] = "ENODEV"; /* No such device */
+ error[ENOTDIR] = "ENOTDIR"; /* Not a directory */
+ error[EISDIR] = "EISDIR"; /* Is a directory */
+ error[EINVAL] = "EINVAL"; /* Invalid argument */
+ error[ENFILE] = "ENFILE"; /* File table overflow */
+ error[EMFILE] = "EMFILE"; /* Too many open files */
+ error[ENOTTY] = "ENOTTY"; /* Not a typewriter */
+ error[ETXTBSY] = "ETXTBSY"; /* Text file busy */
+ error[EFBIG] = "EFBIG"; /* File too large */
+ error[ENOSPC] = "ENOSPC"; /* No space left on device */
+ error[ESPIPE] = "ESPIPE"; /* Illegal seek */
+ error[EROFS] = "EROFS"; /* Read-only file system */
+ error[EMLINK] = "EMLINK"; /* Too many links */
+ error[EPIPE] = "EPIPE"; /* Broken pipe */
+ error[EDOM] = "EDOM"; /* Math argument out of domain of func */
+ error[ERANGE] = "ERANGE"; /* Math result not representable */
+}
+
+/* Specify any syscall return probe and test that the value of errno is in range */
+
+syscall:::return
+/errno > 0 && errno <= ERANGE/
+{
+ printf("%-4d %-6d %-10s %-10s %s()\n", uid, pid, execname, error[errno], probefunc);
+}
+```
+
+```
+# chmod +x displayerrno.d
+# ./displayerrno.d
+UID PID Prog Error Func
+500 3575 test EACCES open()
+500 3575 test EINTR clock_gettime()
+^C
+```
+
+You could modify this program so that it displays verbose
+information about the nature of the error, in addition to the
+name of the error.
diff --git a/doc/tutorial/3.TracingUserSpaceApplications.md b/doc/tutorial/3.TracingUserSpaceApplications.md
new file mode 100644
index 000000000..1d9bf69f2
--- /dev/null
+++ b/doc/tutorial/3.TracingUserSpaceApplications.md
@@ -0,0 +1,832 @@
+
+## Tracing User-Space Applications
+
+This chapter provides information about how to trace a user-space
+application and includes examples of D programs that you can use
+to investigate what is happening in an example user-space program.
+
+### Sample Application
+
+This section provides a sample application to be used in
+subsequent exercises and examples in this chapter. The example,
+which illustrates a simple program, favors brevity and probing
+opportunity rather than completeness or efficiency.
+
+>Note:
+>
+>The following simple program is provided for example purposes
+*only* and is not intended to efficiently solve a practical
+problem nor exhibit preferred coding methods.
+
+The sample program finds the lowest factor of a number, which you input.
+The program is comprised of the following four files:
+`makefile`, `primelib.h`, `primelib.c`, and `primain.c`,
+which are stored in the same working directory.
+
+#### Description and Format of the makefile File
+
+The following example shows the contents of the `makefile` file.
+
+>Note:
+>
+>A `makefile` must use tabs for indentation so
+that the `make` command can function properly.
+Also, be sure that tabs are retained if the
+file is copied and then used.
+
+```
+default: prime
+
+# compile the library primelib first
+primelib.o: primelib.c
+ gcc -c primelib.c
+
+# compile the main program
+primain.o: primain.c
+ gcc -c primain.c
+
+# link and create executable file "prime"
+prime: primelib.o primain.o
+ gcc primain.o primelib.o -o prime -lm
+
+clean:
+ -rm -f *.o
+ -rm -f prime
+```
+
+#### Description of the primelib.h Source File
+
+The following example shows the contents of the `primelib.h` file.
+
+```
+int findMaxCheck( int inValue );
+int seekFactorA( int input, int maxtry );
+int seekFactorB( int input );
+```
+
+#### Description of the primelib.c Source File
+
+The following example shows the contents of the `primelib.c` file.
+
+```
+#include <stdio.h>
+#include <math.h>
+
+/*
+ * utility functions which are called from the main source code
+ */
+
+// Find and return our highest value to check -- which is the square root
+int findMaxCheck( int inValue ) {
+ float sqRoot;
+ sqRoot = sqrt( inValue );
+ printf("Square root of %d is %lf\n", inValue, sqRoot);
+ return floor( sqRoot );
+}
+
+int debugFlag = 0;
+
+// Search for a factor to the input value, proving prime on return of zero
+int seekFactorA( int input, int maxtry ) {
+ int divisor, factor = 0;
+ for( divisor=2; divisor<=maxtry; ++divisor ) {
+ if( 0 == input%divisor ) {
+ factor = divisor;
+ break;
+ }
+ else if ( debugFlag != 0 )
+ printf( "modulo %d yields: %d\n", divisor, input%divisor );
+ }
+ return factor;
+}
+
+// Search for a factor to the input value, proving prime on return of zero
+// This is a different method than "A", using one argument
+int seekFactorB( int input ) {
+ int divisor, factor = 0;
+ if( 0 == input%2 ) return 2;
+ for( divisor=3; divisor<=input/2; divisor+=2 ) {
+ if( 0 == input%divisor ) {
+ factor = divisor;
+ break;
+ }
+ }
+ return factor;
+}
+```
+
+#### Description of the primain.c Source File
+
+The following example shows the contents of the `primain.c` file.
+
+```
+#include <stdio.h>
+#include "primelib.h"
+
+/*
+ * Nominal C program churning to provide a code base we might want to
+ * instrument with D
+ */
+
+// Search for a divisor -- thereby proving composite value of the input.
+int main() {
+ int targVal, divisor, factorA=0, factorB=0;
+
+ printf( "Enter a positive target integer to test for prime status: " );
+ scanf( "%d", &targVal );
+
+ // Check that the user input is valid
+ if( targVal < 2 ) {
+ printf( "Invalid input value -- exiting now\n" );
+ return -2;
+ }
+
+ // Search for a divisor using method and function A
+ int lastCheck;
+ lastCheck = findMaxCheck( targVal );
+ printf( "%d highest value to check as divisor\n", lastCheck );
+ factorA = seekFactorA( targVal, lastCheck );
+
+ // Search for a divisor using method and function B
+ factorB = seekFactorB( targVal );
+
+ // Warn if the methods give different results
+ if (factorA != factorB)
+ printf( "%d does not equal %d! How can this be?\n", factorA, factorB );
+
+ // Print results
+ if( !factorA )
+ printf( "%d is a prime number\n", targVal );
+ else
+ printf( "%d is not prime because there is a factor %d\n",
+ targVal, factorA );
+ return 0;
+}
+```
+
+#### Compiling the Program and Running the prime Executable
+
+With the four files previously described located in the
+same working directory, compile the program by using the
+`make` command as follows:
+
+```
+# make
+gcc -c primelib.c
+gcc -c primain.c
+gcc primain.o primelib.o -o prime -lm
+```
+
+Running the `make` command creates an executable named `prime`,
+which can be run to find the lowest prime value of the input,
+as shown in the following two examples:
+
+```
+# ./prime
+Enter a positive target integer to test for prime status: 5099
+Square root of 5099 is 71.407280
+71 highest value to check as divisor
+5099 is a prime number
+```
+```
+# ./prime
+Enter a positive target integer to test for prime status: 95099
+Square root of 95099 is 308.381256
+308 highest value to check as divisor
+95099 is not prime because there is a factor 61
+```
+
+### Using the `pid` Provider
+
+So far, we have built our application without
+taking DTrace at all into account.
+Nevertheless, we can still study execution of this application,
+using the `pid` provider to trace entry into and return from user functions.
+
+For example, from one window, start the application
+but do not yet provide it with any input,
+thereby leaving the application paused and waiting:
+
+```
+$ ./prime
+Enter a positive target integer to test for prime status:
+```
+
+In another window, we can list the probes supplied by the `pid` provider.
+Actually, there is a different such provider for each process,
+meaning that we must append the process ID to specify the process to trace.
+For example, we can run `pgrep prime` to get the process ID.
+In our case, it is 2889188.
+Then, we can list the `pid` probes associated with this process in either of two ways:
+
+```
+# dtrace -ln pid2889188:a.out::
+# dtrace -p 2889188 -ln pid'$target':a.out::
+```
+
+In the first case, we append the process ID
+explicitly to form the provider name `pid2889188`.
+In the second case, the provider name refers to the process ID symbolically,
+using the macro `$target`,
+and the traced process 2889188 is specified using a
+`dtrace` command-line option `-p`.
+This second form is useful when the probe specification is contained in
+a D script that we do not want to modify each time we use it.
+
+We specify a module of the application executable.
+It could be the load object of the executable, here referred to as `a.out`,
+or it could be a shared library.
+The probe function and name are omitted, equivalent to the wildcard `*`.
+
+In our case, this probe specification results in 187 probes.
+We show a few of them here:
+
+```
+ ID PROVIDER MODULE FUNCTION NAME
+ 5433 pid2889188 prime seekFactorB 56
+ 5431 pid2889188 prime seekFactorB 55
+ 5429 pid2889188 prime seekFactorB 52
+ [...]
+ 5375 pid2889188 prime seekFactorB 4
+ 5373 pid2889188 prime seekFactorB 1
+ 5371 pid2889188 prime seekFactorB 0
+ 5370 pid2889188 prime seekFactorB entry
+ 5368 pid2889188 prime seekFactorB return
+ 5366 pid2889188 prime seekFactorA 68
+ 5364 pid2889188 prime seekFactorA 67
+ 5362 pid2889188 prime seekFactorA 64
+ [...]
+ 5302 pid2889188 prime seekFactorA 4
+ 5300 pid2889188 prime seekFactorA 1
+ 5298 pid2889188 prime seekFactorA 0
+ 5297 pid2889188 prime seekFactorA entry
+ 5295 pid2889188 prime seekFactorA return
+ 5293 pid2889188 prime findMaxCheck 70
+ 5291 pid2889188 prime findMaxCheck 6f
+ 5289 pid2889188 prime findMaxCheck 6b
+ [...]
+ 5243 pid2889188 prime findMaxCheck 4
+ 5241 pid2889188 prime findMaxCheck 1
+ 5239 pid2889188 prime findMaxCheck 0
+ 5238 pid2889188 prime findMaxCheck entry
+ 5236 pid2889188 prime findMaxCheck return
+ 5234 pid2889188 prime main ef
+ 5232 pid2889188 prime main ee
+ 5230 pid2889188 prime main e9
+ [...]
+ 5110 pid2889188 prime main 4
+ 5108 pid2889188 prime main 1
+ 5106 pid2889188 prime main 0
+ 5105 pid2889188 prime main entry
+ 5103 pid2889188 prime main return
+ [...]
+```
+
+The module, which we specified by the nickname `a.out`,
+is listed by its executable name `prime`.
+The functions of this module appear: `seekFactorB`, `seekFactorA`, and so on.
+For each function, there is a probe for entry, return, and each instruction offset.
+
+This listing shows us what probes we might specify to trace this
+user program without having added any DTrace-specific instrumentation.
+For more information on how to use the `pid` provider to trace
+user function entry and return or on specific instruction offsets,
+see
+[Pid Provider](../userguide/reference/dtrace_providers_pid.md)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md).
+
+### Adding USDT Probes to an Application
+
+In this section, we practice adding USDT probes to an application.
+For background information and other details, see
+[Adding USDT Probes to Application Code](../userguide/reference/dtrace-ref-StaticallyDefinedTracingofUserApplications.md#dt_ref_usdt_probe_add_prov)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md).
+
+To get started, you will need to create a `.d` file, as described in
+[Defining USDT Providers and Probes](../userguide/reference/dtrace-ref-StaticallyDefinedTracingofUserApplications.md#dt_ref_usdtprobes_prov)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md).
+
+>Note:
+>
+>This `.d` file is not a script that is run in the same
+way that is shown in previous examples in this tutorial,
+but is rather the `.d` source file that you use when
+compiling and linking your application.
+
+In this `.d` file, you define the probes you will
+place in the `primain.c` source file.
+These probes mark the sequence of operations that are
+used after the user entry is completed and checked:
+
+| Description | Probe |
+| :----------- | :----------- |
+| User entry complete and checked | `userentry( int )` |
+| Return from `findMaxCheck()` | `maxcheckval( int, int )` |
+| Return from `seekFactorA()` | `factorreturnA( int, int )` |
+| Return from `seekFactorB()` | `factorreturnB( int, int )` |
+| Immediately prior to the program exiting | `final()` |
+
+#### Exercise: Creating a dprime.d File
+
+Declare the above probes in a file `dprime.d` and store the
+file in the same working directory as the other source files.
+
+#### Solution to Exercise: Creating a dprime.d File
+
+```
+provider primeget
+{
+ probe query__userentry( int );
+ probe query__maxcheckval( int, int );
+ probe query__factorreturnA( int, int );
+ probe query__factorreturnB( int, int );
+ probe query__final();
+};
+```
+
+#### Example: Creating a .h File From a dprime.d File
+
+The next step is to create a `.h` header file to use in your
+C source program from the `dprime.d` file, as shown here:
+
+```
+# dtrace -h -s dprime.d
+```
+
+The `dprime.h` file that is created contains a reference to each
+of the probe points that are defined in the `dprime.d` file.
+
+Next, in the application source file, `primain.c`, we add
+`#include "dprime.h"` and the appropriate probe macros at the proper
+locations.
+
+In the resulting `primain.c` file, the probe macros are easy
+to recognize, as they appear in uppercase letters:
+
+```
+#include <stdio.h>
+#include "primelib.h"
+#include "dprime.h"
+
+/*
+ * Nominal C program churning to provide a code base we might want to
+ * instrument with D
+*/
+
+// Search for a divisor -- thereby proving composite value of the input.
+int main() {
+ int targVal, divisor, factorA=0, factorB=0;
+
+ printf( "Enter a positive target integer to test for prime status: " );
+ scanf( "%d", &targVal );
+
+ // Check that the user input is valid
+ if( targVal < 2 ) {
+ printf( "Invalid input value -- exiting now\n" );
+ return -2;
+ }
+ if (PRIMEGET_QUERY_USERENTRY_ENABLED())
+ PRIMEGET_QUERY_USERENTRY(targVal);
+
+ // Search for a divisor using method and function A
+ int lastCheck;
+ lastCheck = findMaxCheck( targVal );
+ printf( "%d highest value to check as divisor\n", lastCheck );
+ if (PRIMEGET_QUERY_MAXCHECKVAL_ENABLED())
+ PRIMEGET_QUERY_MAXCHECKVAL(lastCheck, targVal);
+
+ factorA = seekFactorA( targVal, lastCheck );
+ if (PRIMEGET_QUERY_FACTORRETURNA_ENABLED())
+ PRIMEGET_QUERY_FACTORRETURNA(factorA, targVal);
+
+ // Search for a divisor using method and function B
+ factorB = seekFactorB( targVal );
+ if (PRIMEGET_QUERY_FACTORRETURNB_ENABLED())
+ PRIMEGET_QUERY_FACTORRETURNB(factorB, targVal);
+
+ // Warn if the methods give different results
+ if (factorA != factorB)
+ printf( "%d does not equal %d! How can this be?\n", factorA, factorB );
+
+ // Print results
+ if( !factorA )
+ printf( "%d is a prime number\n", targVal );
+ else
+ printf( "%d is not prime because there is a factor %d\n",
+ targVal, factorA );
+ if (PRIMEGET_QUERY_FINAL_ENABLED())
+ PRIMEGET_QUERY_FINAL();
+
+ return 0;
+}
+```
+
+Any `*_ENABLED()` probe will translate into a truth value if
+the associated probe is enabled (some consumer is using it),
+and a false value if the associated probe is not enabled.
+
+Next, you will need to modify the `makefile` file.
+For step-by-step instructions, See
+[Building Applications With USDT Probes](../userguide/reference/dtrace-ref-StaticallyDefinedTracingofUserApplications.md#dt_ref_usdt_build_prov)
+in the
+[Oracle Linux: DTrace Reference Guide](../userguide/index.md).
+
+#### Exercise: Directing makefile to Re-Create the dprime.h File
+
+Add a target that instructs `dtrace` to re-create the `dprime.h` file
+in the event that changes are subsequently made to the `dprime.d` file.
+This step ensures that you do not have to manually run the
+`dtrace -h -s dprime.d` command if any changes are made.
+
+This exercise also has you direct `dtrace` to create a `prime.o` file.
+
+#### Solution to Exercise: Directing makefile to Re-Create the dprime.h File
+
+```
+default: prime
+
+# re-create new dprime.h if dprime.d file has been changed
+dprime.h: dprime.d
+ dtrace -h -s dprime.d
+
+# compile the library primelib first
+primelib.o: primelib.c
+ gcc -c primelib.c
+
+# compile the main program
+primain.o: primain.c dprime.h
+ gcc -c -I/usr/lib64/dtrace/include primain.c
+
+# have dtrace post-process the object files
+prime.o: dprime.d primelib.o primain.o
+ dtrace -G -s dprime.d primelib.o primain.o -o prime.o
+
+# link and create executable file "prime"
+prime: prime.o
+ gcc -Wl,--export-dynamic,--strip-all -o prime prime.o primelib.o primain.o -lm
+
+clean:
+ -rm -f *.o
+ -rm -f prime
+ -rm -f dprime.h
+```
+
+Notice that the `primain.c` file now includes the generated
+`dprime.h` file, which includes `<sys/usdt.h>`.
+Therefore, when we compile `gcc -c primain.c`,
+we include a `-I` option to find `<sys/usdt.h>`.
+The path to use on your system is given by:
+
+```
+pkg-config --cflags dtrace_sdt
+```
+
+which is `/usr/lib64/dtrace/include` in the example above.
+In the DTrace source code directory, the file is in `uts/common`.
+
+In the link stage, the `-Wl,--export-dynamic` link options to `gcc`
+are required for symbol lookup in a stripped executable at runtime,
+for example, when you use the D function `ustack()`.
+
+#### Example: Testing the Program
+
+After creating a fresh build, test that the executable is still
+working as expected:
+
+```
+# make clean
+rm -f *.o
+rm -f prime
+rm -f dprime.h
+# make
+gcc -c primelib.c
+dtrace -h -s dprime.d
+gcc -c primain.c
+dtrace -G -s dprime.d primelib.o primain.o -o prime.o
+gcc -Wl,--export-dynamic,--strip-all -o prime prime.o primelib.o primain.o dprime.h -lm
+```
+
+```
+# ./prime
+Enter a positive target integer to test for prime status: 6799
+Square root of 6799 is 82.456047
+82 highest value to check as divisor
+6799 is not prime because there is a factor 13
+```
+
+### Using USDT Probes
+
+This section provides some practice in the nominal use of the USDT
+probes that were created in
+[Adding USDT Probes to an Application](#adding-usdt-probes-to-an-application).
+
+Initially, the probes are not visible because the application is
+not running with the probes, as shown in the following output:
+
+```
+# dtrace -l -P 'prime*'
+ ID PROVIDER MODULE FUNCTION NAME
+dtrace: failed to match prime*:::: No probe matches description
+```
+
+Start the application, but do not enter any value yet:
+
+```
+# ./prime
+Enter a positive target integer to test for prime status:
+```
+
+From another command line, issue a probe listing:
+
+```
+# dtrace -l -P 'prime*'
+ ID PROVIDER MODULE FUNCTION NAME
+ 2475 primeget26556 prime main query-factorreturnA
+ 2476 primeget26556 prime main query-factorreturnB
+ 2477 primeget26556 prime main query-final
+ 2478 primeget26556 prime main query-maxcheckval
+ 2479 primeget26556 prime main query-userentry
+```
+
+Note that the provider name is a combination of the defined `provider primeget`,
+from the `dprime.d` file, and the PID of the running application `prime`.
+The output of the following command displays the PID of prime:
+
+```
+# ps aux | grep prime
+root 26556 0.0 0.0 7404 1692 pts/0 S+ 21:50 0:00 ./prime
+```
+
+At this point, we can kill the application or provide an input
+value so that the application runs to completion.
+
+#### Example: Using simpleTimeProbe.d to Show the Elapsed Time Between Two Probes
+
+The following example shows how you would create a simple script
+that measures the time elapsed between the first probe and the
+second probe (`query-userentry` to `query-maxcheckval`).
+
+```
+/* simpleTimeProbe.d */
+
+/* Show how much time elapses between two probes */
+
+primeget*:::query-userentry
+{
+ self->t = timestamp; /* Initialize a thread-local variable with the time */
+}
+
+primeget*:::query-maxcheckval
+/self->t != 0/
+{
+ this->timeNow = timestamp;
+ /* Divide by 1000 for microseconds */
+ printf("%s (pid=%d) spent %d microseconds between userentry & maxcheckval\n",
+ execname, pid, ((this->timeNow - self->t)/1000));
+
+ self->t = 0; /* Reset the variable */
+}
+```
+
+Since `timeNow` is used only within this clause,
+we use `this->` to denote its clause-local scope.
+
+Once again, start the execution of the target application:
+
+```
+# ./prime
+Enter a positive target integer to test for prime status:
+```
+
+Then, run the DTrace script from another window:
+
+```
+# dtrace -q -s simpleTimeProbe.d
+```
+
+As the application is running, the output of the script is also
+running in parallel:
+
+```
+# ./prime
+Enter a positive target integer to test for prime status: 7921
+Square root of 7921 is 89.000000
+89 highest value to check as divisor
+7921 is not prime because there is a factor 89
+# ./prime
+Enter a positive target integer to test for prime status: 995099
+Square root of 995099 is 997.546509
+997 highest value to check as divisor
+995099 is not prime because there is a factor 7
+# ./prime
+Enter a positive target integer to test for prime status: 7921
+Square root of 7921 is 89.000000
+89 highest value to check as divisor
+7921 is not prime because there is a factor 89
+```
+
+On the command line where the script is being run, you should
+see output similar to the following:
+
+```
+# dtrace -q -s simpleTimeProbe.d
+prime (pid=2328) spent 45 microseconds between userentry & maxcheckval
+prime (pid=2330) spent 41 microseconds between userentry & maxcheckval
+prime (pid=2331) spent 89 microseconds between userentry & maxcheckval
+^C
+```
+
+Another way of running the D script is with the `-Z` option.
+Previously, we started the application first so that some
+`primeget*` probes would be found.
+With the `-Z` option, `dtrace` will start up even if
+zero matching probes are found.
+Then, it will wait patiently until such probes are discovered
+and those probes fire. That is, in one window, run:
+
+```
+# dtrace -q -Z -s simpleTimeProbe.d
+```
+
+Note the addition of the `-Z` option.
+In other window, run the application:
+
+```
+# ./prime
+Enter a positive target integer to test for prime status: 7921
+Square root of 7921 is 89.000000
+89 highest value to check as divisor
+7921 is not prime because there is a factor 89
+#
+```
+
+The first window should show the `dtrace` output
+and you can terminate the script.
+
+```
+# dtrace -q -Z -s simpleTimeProbe.d
+prime (pid=4334) spent 85 microseconds between userentry & maxcheckval
+^C
+```
+
+And yet one more way of launching both the D script and the
+application, from the same window, is to use the `-c` option
+to `dtrace` to launch a command to trace:
+
+```
+# dtrace -q -c ./prime -s simpleTimeProbe.d
+Enter a positive target integer to test for prime status: 579
+Square root of 579 is 24.062418
+24 highest value to check as divisor
+579 is not prime because there is a factor 3
+prime (pid=2884516) spent 98 microseconds between userentry & maxcheckval
+```
+
+Here, `dtrace` is launched, it starts the command `./prime`,
+and when `dtrace` or the application terminates, so does the other.
+
+#### Example: Using timeTweenprobes.d to Show the Elapsed Time Between Each Probe
+
+You can broaden the script to monitor all of the following
+probes in the application:
+
+- `query-userentry`
+- `query-maxcheckval`
+- `query-factorreturnA`
+- `query-factorreturnB`
+- `query-final`
+
+```
+/* timeTweenProbes.d */
+
+/* show how much time elapses between each probe */
+
+BEGIN
+{
+ iterationCount = 0;
+}
+
+primeget*:::query-userentry
+{
+ printf("%s (pid=%d) running\n", execname, pid);
+ self->t = timestamp; /* Initialize a thread-local variable with time */
+}
+
+primeget*:::query-maxcheckval
+/self->t != 0/
+{
+ this->timeNow = timestamp;
+ printf(" maxcheckval spent %d microseconds since userentry\n",
+ ((this->timeNow - self->t)/1000)); /* Divide by 1000 for microseconds */
+ self->t = this->timeNow; /* set the time to recent sample */
+}
+
+primeget*:::query-factorreturnA
+/self->t != 0/
+{
+ this->timeNow = timestamp;
+ printf(" factorreturnA spent %d microseconds since maxcheckval\n",
+ ((this->timeNow - self->t)/1000)); /* Divide by 1000 for microseconds */
+ self->t = this->timeNow; /* set the time to recent sample */
+}
+
+primeget*:::query-factorreturnB
+/self->t != 0/
+{
+ this->timeNow = timestamp;
+ printf(" factorreturnB spent %d microseconds since factorreturnA\n",
+ ((this->timeNow - self->t)/1000)); /* Divide by 1000 for microseconds */
+ self->t = this->timeNow; /* set the time to recent sample */
+}
+
+primeget*:::query-final
+/self->t != 0/
+{
+ printf(" prime spent %d microseconds from factorreturnB until ending\n",
+ ((timestamp - self->t)/1000));
+ self->t = 0; /* Reset the variable */
+ iterationCount++;
+}
+
+END
+{
+ trace(iterationCount);
+}
+```
+
+Again, start the execution of the target application first,
+without yet providing input.
+Then from another window, start the D script.
+Back in the first window, you can run the application multiple times.
+Finally, in the second window, you can terminate the D script.
+The first window might look something like this:
+
+```
+# ./prime
+Enter a positive target integer to test for prime status: 995099
+Square root of 995099 is 997.546509
+997 highest value to check as divisor
+995099 is not prime because there is a factor 7
+# ./prime
+Enter a positive target integer to test for prime status: 7921
+Square root of 7921 is 89.000000
+89 highest value to check as divisor
+7921 is not prime because there is a factor 89
+# ./prime
+Enter a positive target integer to test for prime status: 95099
+Square root of 95099 is 308.381256
+308 highest value to check as divisor
+95099 is not prime because there is a factor 61
+# ./prime
+Enter a positive target integer to test for prime status: 95099
+Square root of 95099 is 308.381256
+308 highest value to check as divisor
+95099 is not prime because there is a factor 61
+# ./prime
+Enter a positive target integer to test for prime status: 5099
+Square root of 5099 is 71.407280
+71 highest value to check as divisor
+5099 is a prime number
+```
+
+The corresponding output from the script is similar to the
+following:
+
+```
+# dtrace -q -s ./timeTweenProbes.d
+prime (pid=2437) running
+ maxcheckval spent 96 microseconds since userentry
+ factorreturnA spent 9 microseconds since maxcheckval
+ factorreturnB spent 6 microseconds since factorreturnA
+ prime spent 9 microseconds from factorreturnB until ending
+prime (pid=2439) running
+ maxcheckval spent 45 microseconds since userentry
+ factorreturnA spent 10 microseconds since maxcheckval
+ factorreturnB spent 7 microseconds since factorreturnA
+ prime spent 9 microseconds from factorreturnB until ending
+prime (pid=2440) running
+ maxcheckval spent 43 microseconds since userentry
+ factorreturnA spent 11 microseconds since maxcheckval
+ factorreturnB spent 8 microseconds since factorreturnA
+ prime spent 10 microseconds from factorreturnB until ending
+prime (pid=2441) running
+ maxcheckval spent 53 microseconds since userentry
+ factorreturnA spent 10 microseconds since maxcheckval
+ factorreturnB spent 7 microseconds since factorreturnA
+ prime spent 10 microseconds from factorreturnB until ending
+prime (pid=2442) running
+ maxcheckval spent 40 microseconds since userentry
+ factorreturnA spent 9 microseconds since maxcheckval
+ factorreturnB spent 48 microseconds since factorreturnA
+ prime spent 10 microseconds from factorreturnB until ending
+
+^C
+5
+```
+
+As is observed in the previous example, there is now a set of
+DTrace features that can be used with the probes that were
+created.
diff --git a/doc/tutorial/4.GoingFurtherWithDTrace.md b/doc/tutorial/4.GoingFurtherWithDTrace.md
new file mode 100644
index 000000000..e267db907
--- /dev/null
+++ b/doc/tutorial/4.GoingFurtherWithDTrace.md
@@ -0,0 +1,14 @@
+
+## Going Further With DTrace
+
+For more information about using DTrace on Linux, see the
+[DTrace User Guide](../userguide/index.md).
+
+There are [DTrace trainings videos](https://oracle-samples.github.io/oltrain/tracks/ol/dtrace).
+
+The latest DTrace development work and source code for Linux is
+available at <https://github.com/oracle/dtrace-utils/>.
+
+Or, there are [DTrace packages built for Oracle Linux](https://www.oracle.com/linux/downloads/linux-dtrace.html).
+There are
+[Oracle Linux: DTrace Release Notes](https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-relnotes/)
diff --git a/doc/tutorial/index.md b/doc/tutorial/index.md
new file mode 100644
index 000000000..60d5dd8b4
--- /dev/null
+++ b/doc/tutorial/index.md
@@ -0,0 +1,15 @@
+# DTrace Tutorial
+
+- [Introducing DTrace](1.IntroducingDTrace.md)
+- [Tracing Operating System Behavior](2.TracingOperatingSystemBehavior.md)
+- [Tracing User Space Applications](3.TracingUserSpaceApplications.md)
+- [Going Further With DTrace](4.GoingFurtherWithDTrace.md)
+
+<!--
+These are from:
+https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-tutorial/dtrace-tutorial-Preface.html
+https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-tutorial/dtrace-tutorial-IntroducingDTrace.html
+https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-tutorial/dtrace-tutorial-TracingOperatingSystemBehavior.html
+https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-tutorial/dtrace-tutorial-TracingUserSpaceApplications.html
+https://docs.oracle.com/en/operating-systems/oracle-linux/dtrace-tutorial/dtrace-tutorial-GoingFurtherWithDTrace.html
+-->
--
2.47.3
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.