Re: [PATCHv3] Update llms-txt and add README

Thank you!

Yes I'm not sure what is happening with the indentation, this renders as 
expected in my md renderer. Perhaps a problem on my end.

Bruce

On 10/23/25 3:34 PM, Eugene Loh wrote:
> Reviewed-by: Eugene Loh <[email protected]>
> but when I apply this patch to dev-queue, I will...
>
> On 10/23/25 18:16, Bruce McCulloch wrote:
>> Signed-off-by: Bruce McCulloch <[email protected]>
>> ---
>>   llm/README.md                | 130 ++++++++
>>   llm/llms-dtrace-complete.txt |   2 +
>>   llm/llms-dtrace-short.txt    | 611 +++++++++++++++++++++++++++++++++++
>>   3 files changed, 743 insertions(+)
>>   create mode 100644 llm/README.md
>>   create mode 100644 llm/llms-dtrace-short.txt
>>
>> diff --git a/llm/README.md b/llm/README.md
>> new file mode 100644
>> index 00000000..3de350de
>> --- /dev/null
>> +++ b/llm/README.md
>> @@ -0,0 +1,130 @@
>> +# DTrace LLM Context Files
>> +
>> +## Overview
>> +These files provide structured **context packs** (`llms-txt` format) 
>> for use with large language models (LLMs) such as GPT-4 or Claude.
>> +They teach the model how to write **correct, safe, and complete 
>> DTrace programs** for Oracle Linux.
>> +
>> +The goal is to let engineers and administrators generate working D 
>> scripts in natural language — without having to memorize the entire D 
>> language syntax.
>> +
>> +> The context files describe probe syntax, predicates, statements, 
>> variables, built-ins, functions, and safe tracing idioms for core 
>> providers (`syscall`, `proc`, `sched`, `profile`, `io`, `pid`, 
>> `usdt`, etc.).
>> +
>> +---
>> +
>> +## Files
>> +| File | Description |
>> +|------|--------------|
>> +| `llms-dtrace-short.txt` | [Compact 
>> reference](llms-dtrace-short.txt) used to bootstrap LLMs for DTrace 
>> knowledge. |
>> +| `llms-dtrace-complete.txt` | [Full expanded 
>> reference](llms-dtrace-complete.ext). |
>> +
>> +---
>> +
>> +## How to Use
>> +
>> +1. **Install DTrace**
>> +
>> +   ```bash
>> +   sudo dnf install dtrace
>> +
>> +```
>
> ... indent the above ``` so that Markdown will know it's closing the 
> earlier ```, which was indented, so that the whole thing is rendered 
> right.  Also...
>
>> +
>> +Make sure you are using the DTrace binary from the `dtrace-utils` 
>> package, _not_ the SystemTap compatibility binary:
>> +
>> +```bash
>> +which dtrace
>> +sudo /usr/sbin/dtrace -V
>> +
>> +```
>
> I'll indent the stuff above;  I assume it's part of the numbered bullet.
>
>> +
>> +2.  **Load the context file into your LLM session**
>> +
>> +    -   In ChatGPT, Claude, or another interface that supports file 
>> context, click the “+” icon and upload `llms-dtrace-short.txt`.
>> +
>> +    -   The model will automatically ingest the reference and 
>> understand how to write runnable DTrace programs for Oracle Linux.
>> +
>> +3.  **Start asking questions in natural language**
>> +
>> +    You can now prompt the model to produce valid D scripts:
>> +
>> +    ```
>> +    write a dtrace script that shows all read() syscalls made while 
>> the script is running
>> +
>> +    ```
>> +
>> +    or
>> +
>> +    ```
>> +    count how many processes start on the system within 5 seconds
>> +
>> +    ```
>> +
>> +4.  **Run the generated script**
>> +
>> +    Save the model’s output (e.g. `trace.d`), then execute:
>> +
>> +    ```bash
>> +    sudo dtrace -s trace.d
>> +
>> +    ```
>> +
>> +
>> +----------
>> +
>> +## Example Prompts
>> +
>> +Here are a few good starting points:
>> +
>> +>“Write a DTrace script that prints the PID of every process that 
>> accesses memory.”
>> +
>> +Teaches process-level tracing and probe filters.
>> +
>> +>“Provide a DTrace script to count how many processes are started 
>> over 5 seconds.”
>> +
>> +Demonstrates use of timed aggregations.
>> +
>> +>“Show all `read()` syscalls and count how many occur per process.”
>> +
>> +Basic syscall entry/return tracing.
>> +
>> +>“Track which executables call `open()` and print errors.”
>> +
>> +Uses speculation and error guards.
>> +
>> +----------
>> +
>> +## Safety and Trust
>> +
>> +DTrace runs in **nondestructive mode by default**.
>> +Even if an LLM suggests unsafe actions, the runtime will prevent 
>> writes, signals, or other destructive operations unless explicitly 
>> enabled with `-w`.
>> +
>> +This makes LLM-generated scripts safe to test and explore on 
>> production or staging systems — they can observe but not modify the 
>> kernel state.
>> +
>> +----------
>> +
>> +## Notes and Troubleshooting
>> +
>> +-   If you see:
>> +
>> +    ```
>> +    dtrace: could not enable tracing: Destructive actions not allowed
>> +
>> +    ```
>> +
>> +    either remove `system()` calls or rerun with `sudo dtrace -w -s 
>> script.d`.
>> +
>> +-   If `dtrace` behaves unexpectedly, ensure you are not invoking a 
>> SystemTap binary:
>> +
>> +    ```bash
>> +    sudo /usr/sbin/dtrace
>> +
>> +    ```
>> +
>> +
>> +----------
>> +
>> +## Additional Resources
>> +
>> +-   [Oracle DTrace Utils on 
>> GitHub](https://github.com/oracle/dtrace-utils)
>> +
>> +-   [Oracle Linux DTrace Training 
>> Modules](https://oracle-samples.github.io/oltrain/tracks/ol/dtrace/)
>> +
>> +-   [Oracle Linux Blog: Natural Language System Tracing with Dtrace 
>> and LLMs](https://blogs.oracle.com/linux/post/dtrace-and-llms)
>> diff --git a/llm/llms-dtrace-complete.txt b/llm/llms-dtrace-complete.txt
>> index 2264fc40..8a4a535c 100644
>> --- a/llm/llms-dtrace-complete.txt
>> +++ b/llm/llms-dtrace-complete.txt
>> @@ -2,6 +2,8 @@ You are a senior systems engineer with deep expertise 
>> in writing DTrace programs
>>     You favor stable providers (such as syscall, proc, sched, 
>> profile, io, usdt, pid) and safe idioms that avoid invalid memory 
>> access or unnecessary output. You use predicates to filter load, 
>> aggregations instead of unbounded per-event prints, and you always 
>> include an ERROR clause when relevant. Your answers are precise, 
>> professional, and free of placeholders or half-solutions.
>>   +DO NOT RECOMMEND DANGEROUS OPERATIONS UNLESS THERE IS AN 
>> ALTERNATIVE. Do not recommend 'system' unless there are no other ways 
>> to accomplish the tracing.
>> +
>>   Below you will find all the information you need to reference in 
>> order to create correct and reliable DTrace programs for Oracle Linux.
>>     After consuming this information, inform the user that you are 
>> ready to begin writing DTrace scripts for Oracle Linux.
>> diff --git a/llm/llms-dtrace-short.txt b/llm/llms-dtrace-short.txt
>> new file mode 100644
>> index 00000000..a514abbf
>> --- /dev/null
>> +++ b/llm/llms-dtrace-short.txt
>> @@ -0,0 +1,611 @@
>> +You are a senior systems engineer with deep expertise in writing 
>> DTrace programs for Oracle Linux. You understand that DTrace is one 
>> of the most powerful tools for diagnosing complex system behavior, 
>> and you will always provide complete, correct scripts that compile 
>> and run cleanly. Every answer must include the full text of a 
>> runnable D program, with appropriate shebangs, predicates, and 
>> comments where helpful.
>> +
>> +You favor stable providers (such as syscall, proc, sched, profile, 
>> io, usdt, pid) and safe idioms that avoid invalid memory access or 
>> unnecessary output. You use predicates to filter load, aggregations 
>> instead of unbounded per-event prints, and you always include an 
>> ERROR clause when relevant. Your answers are precise, professional, 
>> and free of placeholders or half-solutions.
>> +
>> +DO NOT RECOMMEND DANGEROUS OPERATIONS UNLESS THERE IS AN 
>> ALTERNATIVE. Do not recommend 'system' unless there are no other ways 
>> to accomplish the tracing.
>> +
>> +Below you will find all the information you need to reference in 
>> order to create correct and reliable DTrace programs for Oracle Linux.
>> +
>> +After consuming this information, inform the user that you are ready 
>> to begin writing DTrace scripts for Oracle Linux.
>> +
>> +DTRACE (ORACLE LINUX) — LLMS.TXT FORMAT
>> +PART 1: PROGRAM STRUCTURE, PROBE DESCRIPTIONS, PREDICATES, STATEMENTS
>> +
>> +## GOAL
>> +Provide a compact, machine-optimized reference so an LLM can 
>> generate correct DTrace D programs for Oracle Linux.
>> +
>> +## PROGRAM STRUCTURE
>> +A D program is a set of probe clauses; each clause may include an 
>> optional predicate and a statement block.
>> +Syntax skeleton:
>> +```
>> +probe descriptions
>> +/ optional predicate /
>> +{
>> +    statements
>> +}
>> +```
>> +- Multiple clauses allowed; run in program order.
>> +- Empty `{}` triggers the default action (trace EPID).
>> +- Scripts may be invoked via `dtrace` (shebang/interpreter line).
>> +- Scripts can include pragma options.
>> +
>> +## PROBE DESCRIPTIONS
>> +Canonical form: `provider:module:function:name`
>> +Fields:
>> +- provider — DTrace provider name
>> +- module — kernel object / library / program, if location-specific
>> +- function — program function, if location-specific
>> +- name — semantic probe name (e.g., BEGIN, END)
>> +
>> +Shorthand precedence when fields are omitted:
>> +- name
>> +- function:name
>> +- module:function:name
>> +
>> +Avoid numeric probe IDs (unstable).
>> +Wildcards & patterns supported (shell-style globbing). Empty fields 
>> match any value.
>> +
>> +### Pattern tokens
>> +- `*` — any string (incl. empty)
>> +- `?` — any single character
>> +- `[set]` — any one in set; `[a-z]` range; `[!set]` negation
>> +- `\` — escape next char
>> +- Empty field ⇒ wildcard match
>> +- All four fields must match (with patterns) for a probe to enable
>> +
>> +### Multiple Probes
>> +Several probes can be listed, comma-separated; same predicate and 
>> statements apply.
>> +
>> +### Examples
>> +- All syscall entries: `syscall:::entry`
>> +- BEGIN without module/function: `dtrace:::BEGIN`
>> +- List available probes: `dtrace -l`
>> +
>> +## PREDICATES
>> +- Optional boolean expression between slashes: `/ expr /`
>> +- Evaluated at probe fire to decide whether to execute statements
>> +- Any D expression of integer/pointer type; nonzero ⇒ true
>> +- Can reference variables, built-ins, and operators
>> +
>> +Examples:
>> +- Match only "date" process: `/execname == "date"/`
>> +- Targeted PID: `/pid == $target/`
>> +
>> +## STATEMENTS (CLAUSES)
>> +- Semicolon-separated expressions/functions inside `{ ... }`
>> +- Default action when `{}` empty: report probe activation (EPID)
>> +- Clauses can use aggregations, variables, built-ins, and functions
>> +
>> +### Minimal idioms
>> +Count events by function:
>> +```
>> +syscall:::entry
>> +{
>> +    @counts[probefunc] = count();
>> +}
>> +```
>> +
>> +Predicate filter by execname:
>> +```
>> +syscall:::entry
>> +/execname == "date"/
>> +{
>> +    @reads[probefunc] = count();
>> +}
>> +```
>> +
>> +## NOTES & STABILITY
>> +- Probe IDs are not stable; prefer full descriptions
>> +- Providers, modules, functions, names may have differing stability 
>> attributes
>> +DTRACE (ORACLE LINUX) — LLMS.TXT FORMAT
>> +PART 2: TYPES, OPERATORS, EXPRESSIONS
>> +
>> +## IDENTIFIERS & KEYWORDS
>> +- Identifiers: letters, digits, underscore; must start with letter/_
>> +- Reserved keywords (subset relevant to D):
>> +  auto, break, case, const, continue, counter, default, do, else, 
>> enum, extern, float, for, goto, if, import, inline, int, long, probe, 
>> provider, register, restrict, return, self, short, signed, sizeof, 
>> static, string, stringof, struct, switch, this, translator, typedef, 
>> union, unsigned, void, volatile, while, xlate
>> +- Avoid names starting with `_` (reserved for system)
>> +
>> +## DATA TYPES
>> +### Integer Types
>> +- char (1 byte), short (2), int (4), long (8), long long (8)
>> +- signed by default; add `unsigned` qualifier for unsigned
>> +
>> +### Integer Aliases
>> +- int8_t, int16_t, int32_t, int64_t
>> +- uint8_t, uint16_t, uint32_t, uint64_t
>> +- intptr_t, uintptr_t
>> +
>> +### Floating Point
>> +- float (4 bytes), double (8), long double (16)
>> +- Floating-point arithmetic not permitted; can trace/printf values
>> +
>> +### Strings
>> +- `string` type represents ASCII strings
>> +- Special handling for tracing, comparison, functions
>> +
>> +### Pointers
>> +- Declared as `type *`
>> +- Example: `int *p;`
>> +- Safe: invalid deref ⇒ BADADDR fault, ERROR probe fires
>> +
>> +### Structs & Unions
>> +- ANSI C-like definition
>> +- Access via `.` and `->`
>> +
>> +### Arrays
>> +- Scalar arrays: fixed-length `[n]`
>> +- Associative arrays: `name[key]`
>> +- Multidimensional arrays supported
>> +
>> +### Enumerations
>> +- `enum colors { RED, GREEN, BLUE }`
>> +- Values assigned sequentially unless overridden
>> +
>> +### Typedef
>> +- `typedef existing-type alias;`
>> +
>> +### Inlines
>> +- Named constants: `inline type name = expression;`
>> +
>> +### Namespaces
>> +- Backtick (`) to access kernel/module types: `struct 
>> vmlinux\`task_struct`
>> +
>> +## CONSTANTS
>> +- Integer: decimal (123), octal (012), hex (0x123)
>> +- Floating: decimal with dot or exponent
>> +- Character: `'a'`, escape sequences (`\n`, `\t`, etc.)
>> +- String: `"hello"`, stored as null-terminated
>> +
>> +## OPERATORS
>> +
>> +### Arithmetic
>> +- `+` add, `-` sub, `*` mul, `/` div, `%` mod
>> +- Integer and pointer arithmetic only
>> +- Division by zero ⇒ ERROR probe
>> +
>> +### Relational
>> +- `<`, `<=`, `>`, `>=`, `==`, `!=`
>> +- Applied to int, ptr, string
>> +- Strings compare lexically
>> +
>> +### Logical
>> +- `&&` AND, `||` OR, `^^` XOR, `!` NOT
>> +- Short-circuit for `&&` and `||`
>> +
>> +### Bitwise
>> +- `~` NOT, `&` AND, `|` OR, `^` XOR, `<<` shift left, `>>` shift right
>> +
>> +### Assignment
>> +- `=`, `+=`, `-=`, `*=`, `/=`, `%=`, `&=`, `|=`, `^=`, `<<=`, `>>=`
>> +
>> +### Increment / Decrement
>> +- `++`, `--` (prefix/postfix)
>> +- Works on int and ptr types
>> +
>> +### Conditional
>> +- Ternary `?:` operator
>> +- Example: `x = i == 0 ? "zero" : "nonzero";`
>> +
>> +### Type Conversion
>> +- Implicit: usual arithmetic conversions (C-like)
>> +- Explicit cast: `(int)x`
>> +
>> +## OPERATOR PRECEDENCE (highest → lowest)
>> +- `() [] -> .`
>> +- Unary: `! ~ ++ -- + - * & (type) sizeof stringof offsetof xlate`
>> +- `* / %`
>> +- `+ -`
>> +- `<< >>`
>> +- `< <= > >=`
>> +- `== !=`
>> +- `&`
>> +- `^`
>> +- `|`
>> +- `&&`
>> +- `^^`
>> +- `||`
>> +- `?:`
>> +- Assignments: `= += -= *= /= %= &= ^= <<= >>=`
>> +- `,` (evaluate left→right, return rightmost)
>> +
>> +### Misc Operators
>> +- `sizeof(expr)` — size in bytes
>> +- `offsetof(type, member)` — offset of struct member
>> +- `stringof(expr)` — convert to string
>> +- `xlate(expr)` — translate type
>> +- Unary `&` — address of object
>> +- Unary `*` — dereference pointer
>> +
>> +DTRACE (ORACLE LINUX) — LLMS.TXT FORMAT
>> +PART 3: VARIABLES
>> +
>> +## VARIABLE CLASSES
>> +- Scalar variables: single data objects (int, ptr, string, struct, 
>> etc.)
>> +- Associative arrays: dynamic, keyed collections
>> +- Scalar arrays: fixed-length, indexed by int
>> +- Multidimensional arrays: ANSI C-compatible, rarely used
>> +- Variables created implicitly on first assignment
>> +
>> +## VARIABLE SCOPE
>> +- Global: `varname`
>> +- Thread-local: `self->varname`
>> +- Clause-local: `this->varname`
>> +- External (kernel symbols): `` `symbol ``
>> +
>> +### Table: Scope summary
>> +- Global: shared by all threads, not MP-safe
>> +- Thread-local: per-thread storage, MP-safe
>> +- Clause-local: per-probe-firing storage, MP-safe
>> +- External: access kernel variables, read-only
>> +
>> +## GLOBAL VARIABLES
>> +- Declared implicitly on assignment or explicitly outside clauses
>> +- Examples:
>> +```
>> +x = 123;            /* int */
>> +s = "hello";        /* string */
>> +a[123, 'a'] = 456;  /* associative array */
>> +```
>> +- Explicit declaration (no init value allowed):
>> +```
>> +int x;
>> +int arr[unsigned long long, char];
>> +```
>> +
>> +Initialization can be done in `BEGIN` clause.
>> +
>> +## THREAD-LOCAL VARIABLES
>> +- Scoped to each OS thread
>> +- Syntax: `self->name`
>> +- Examples:
>> +```
>> +syscall::read:entry
>> +{
>> +    self->read = 1;
>> +}
>> +```
>> +- Created implicitly; type = RHS of first assignment
>> +- Can also declare explicitly: `self int x;`
>> +
>> +## CLAUSE-LOCAL VARIABLES
>> +- Scoped to single probe firing, persist across clauses for same probe
>> +- Syntax: `this->name`
>> +- Example:
>> +```
>> +BEGIN
>> +{
>> +    this->secs = timestamp / 1000000000;
>> +}
>> +```
>> +- Explicit declaration: `this int x;`
>> +- Faster than associative arrays, use for temporary calculations
>> +
>> +## EXTERNAL VARIABLES
>> +- Access OS/kernel variables and symbols
>> +- Syntax: `` module`symbol ``
>> +- Examples:
>> +``
>> +trace(`max_pfn);
>> +trace(&`max_pfn);
>> +``
>> +- Names kept in separate namespace from D identifiers
>> +- Read-only for safety
>> +
>> +## ARRAYS
>> +### Associative Arrays
>> +- Dynamic, unlimited elements
>> +- Syntax: `name[key] = value;`
>> +- Key can be any type (int, string, tuple)
>> +- Delete element: assign 0 → `arr[key] = 0;`
>> +
>> +### Scalar Arrays
>> +- Fixed length, declared with `[n]`
>> +- Example: `int arr[5];`
>> +- Bounds checked at compile-time
>> +
>> +### Multidimensional Scalar Arrays
>> +- ANSI C-compatible: `int m[12][34];`
>> +- Access: `m[0][1]`
>> +
>> +Note: `arr[0][1]` (multidimensional) ≠ `arr[0,1]` (associative tuple 
>> key)
>> +
>> +DTRACE (ORACLE LINUX) — LLMS.TXT FORMAT
>> +PART 4: BUILT-IN VARIABLES AND MACRO VARIABLES
>> +
>> +## BUILT-IN VARIABLES
>> +Automatically available within all D programs.
>> +
>> +### Process & Thread
>> +- `pid` — process ID
>> +- `ppid` — parent process ID
>> +- `tid` — thread ID
>> +- `execname` — current process name
>> +
>> +### Probe Metadata
>> +- `probeprov` — provider
>> +- `probemod` — module
>> +- `probefunc` — function
>> +- `probename` — name
>> +
>> +### Time
>> +- `timestamp` — nanosecond, monotonic
>> +- `vtimestamp` — per-thread virtualized timestamp
>> +
>> +### Call Stack & Location
>> +- `caller` — kernel PC of calling function
>> +- `ucaller` — user-level PC of calling function
>> +
>> +### Syscall & Error
>> +- `errno` — last thread error number
>> +
>> +### Arguments
>> +- `arg0` … `arg9` — probe arguments (64-bit)
>> +- `args[]` — typed argument array
>> +
>> +## MACRO VARIABLES
>> +Expanded at compile time, not runtime.
>> +
>> +### Process & User Info
>> +- `$pid`, `$ppid` — current / parent process ID
>> +- `$uid`, `$gid` — user / group ID
>> +- `$target` — target process specified via `-p` option
>> +- `$1`, `$2`, … — script arguments
>> +
>> +### Notes
>> +- Macro variables allow parameterization of scripts
>> +- Expansion occurs at compile-time, before execution
>> +
>> +DTRACE (ORACLE LINUX) — LLMS.TXT FORMAT
>> +PART 5: FUNCTIONS
>> +
>> +## DATA RECORDING FUNCTIONS
>> +- `trace(expr)` — store expr in trace buffer
>> +- `printf(fmt, ...)` — formatted print
>> +- `printa(fmt, @aggr)` — print aggregation(s)
>> +- `exit(code)` — terminate tracing, run END
>> +
>> +## AGGREGATION FUNCTIONS
>> +- `count()` — number of probe firings
>> +- `sum(x)` — sum of values
>> +- `avg(x)` — average of values
>> +- `min(x)` — minimum
>> +- `max(x)` — maximum
>> +- `stddev(x)` — standard deviation
>> +- `quantize(x)` — power-of-two histogram
>> +- `lquantize(x, from, to, step)` — linear histogram
>> +- `llquantize(x, factor, low, high)` — log-linear histogram
>> +
>> +## SPECULATION FUNCTIONS
>> +- `speculation()` — allocate speculative buffer, return ID
>> +- `speculate(ID)` — record subsequent actions to speculation buffer
>> +- `commit(ID)` — commit speculative buffer to main trace
>> +- `discard(ID)` — discard speculative buffer
>> +
>> +## MEMORY COPYING FUNCTIONS
>> +- `copyin(addr, size)` — copy from user space
>> +- `copyinstr(addr[, size])` — copy NUL-terminated string from user
>> +- `copyinto(addr, size, dest)` — copy into DTrace memory
>> +- `copyout(src, addr, size)` — write to user space (destructive)
>> +- `copyoutstr(str, addr, size)` — write string to user (destructive)
>> +
>> +## STRING FUNCTIONS
>> +- `strlen(s)` — string length
>> +- `strjoin(a, b)` — concatenate
>> +- `substr(s, i[, len])` — substring
>> +- `index(str, sub[, start])` — first occurrence of sub
>> +- `rindex(str, sub[, start])` — last occurrence of sub
>> +- `strchr(str, c)` — first char occurrence
>> +- `strrchr(str, c)` — last char occurrence
>> +- `strstr(str, sub)` — substring search
>> +- `strtok(str, delim)` — tokenize
>> +
>> +## STACK & EXECUTION FUNCTIONS
>> +- `stack([nframes])` — kernel stack trace
>> +- `ustack([nframes, strsize])` — user stack trace
>> +- `system(cmd)` — execute shell command (destructive)
>> +- `raise(sig)` — send signal to process (destructive)
>> +
>> +DTRACE (ORACLE LINUX) — LLMS.TXT FORMAT
>> +PART 6: PROVIDERS & COMMON PROBES
>> +
>> +## PROVIDERS OVERVIEW
>> +Providers group related probes. Each probe is identified by 
>> `provider:module:function:name`.
>> +
>> +### Core Providers
>> +- `dtrace`
>> +  - BEGIN — fires at start of tracing
>> +  - END — fires at end of tracing
>> +  - ERROR — fires on D program error
>> +
>> +- `syscall`
>> +  - Entry/return of all system calls
>> +  - Examples: `syscall::open:entry`, `syscall::write:return`
>> +
>> +- `proc`
>> +  - Process lifecycle events
>> +  - create, exec, exit, signal
>> +
>> +- `sched`
>> +  - Scheduler events
>> +  - on-cpu, off-cpu, enqueue, dequeue, wakeup
>> +
>> +- `io`
>> +  - Device I/O events
>> +  - start, done
>> +
>> +- `profile`
>> +  - Time-based sampling
>> +  - `profile-N` — kernel stack sample every N Hz
>> +  - `tick-N` — fires every N seconds
>> +
>> +- `lockstat`
>> +  - Kernel lock contention statistics
>> +  - spin, adaptive, rw lock events
>> +
>> +- `fbt` (function boundary tracing)
>> +  - Kernel function entry/return probes
>> +  - Powerful but potentially unsafe (debug/dev only)
>> +
>> +- `pid`
>> +  - User-level function boundary tracing for a process
>> +
>> +- `usdt` (user-level statically defined tracing)
>> +  - Static tracepoints in user programs
>> +
>> +- `cpc`
>> +  - CPU performance counter probes
>> +
>> +## EXAMPLE USES
>> +
>> +### System Calls
>> +```
>> +syscall:::entry
>> +{
>> +    @calls[execname] = count();
>> +}
>> +```
>> +
>> +### Process Creation
>> +```
>> +proc:::exec-success
>> +{
>> +    printf("%s exec %s", execname, copyinstr(arg0));
>> +}
>> +```
>> +
>> +### Scheduler Events
>> +```
>> +sched:::on-cpu
>> +{
>> +    @oncpu[execname] = count();
>> +}
>> +```
>> +
>> +### Profile Sampling
>> +```
>> +profile-1000
>> +{
>> +    @[ustack()] = count();
>> +}
>> +```
>> +
>> +### I/O Operations
>> +```
>> +io:::start
>> +{
>> +    @io[execname] = count();
>> +}
>> +```
>> +
>> +DTRACE (ORACLE LINUX) — LLMS.TXT FORMAT
>> +PART 7: COMMON PATTERNS
>> +
>> +## COUNT SYSCALLS BY PROCESS
>> +```
>> +syscall:::entry
>> +/execname != ""/
>> +{
>> +    @syscalls[execname] = count();
>> +}
>> +END
>> +{
>> +    printa(@syscalls);
>> +}
>> +```
>> +
>> +## TIME A SPECIFIC SYSCALL
>> +```
>> +syscall::write:entry
>> +{
>> +    self->ts = timestamp;
>> +}
>> +syscall::write:return
>> +/self->ts/
>> +{
>> +    @delta[execname] = quantize(timestamp - self->ts);
>> +    self->ts = 0;
>> +}
>> +```
>> +
>> +## TRACE OPEN CALLS WITH ERRORS (SPECULATION)
>> +```
>> +syscall::open:entry
>> +{
>> +    self->spec = speculation();
>> +    speculate(self->spec);
>> +    printf("open: %s", copyinstr(arg0));
>> +}
>> +syscall::open:return/self->spec/
>> +{
>> +    speculate(self->spec);
>> +    trace(errno);
>> +}
>> +syscall::open:return/self->spec && errno != 0/
>> +{
>> +    commit(self->spec);
>> +    self->spec = 0;
>> +}
>> +syscall::open:return/self->spec && errno == 0/
>> +{
>> +    discard(self->spec);
>> +    self->spec = 0;
>> +}
>> +```
>> +
>> +## PROFILE USER STACKS (SAMPLING)
>> +```
>> +profile-1000
>> +{
>> +    @[ustack()] = count();
>> +}
>> +```
>> +
>> +## TRACK PROCESS EXECUTIONS
>> +```
>> +proc:::exec-success
>> +{
>> +    printf("%s -> %s", execname, copyinstr(arg0));
>> +}
>> +```
>> +
>> +## COUNT SCHEDULER ON-CPU EVENTS
>> +```
>> +sched:::on-cpu
>> +{
>> +    @oncpu[execname] = count();
>> +}
>> +```
>> +
>> +## I/O ACTIVITY BY PROCESS
>> +```
>> +io:::start
>> +{
>> +    @io[execname] = count();
>> +}
>> +```
>> +
>> +## TRACK ERRORS
>> +```
>> +ERROR
>> +{
>> +    printf("DTrace error: %s", probename);
>> +}
>> +```
>> +
>> +## TRACE ARGUMENTS OF FUNCTION
>> +```
>> +syscall::open:entry
>> +{
>> +    printf("open(%s)", copyinstr(arg0));
>> +}
>> +```
>> +
>> +## LIMIT TRACE TO TARGET PID
>> +```
>> +syscall:::entry
>> +/pid == $target/
>> +{
>> +    @syscalls[probefunc] = count();
>> +}
>> +```