Re: sun4v trap table

On Thu, 9 Oct 2025 18:53:48 +0000 (UTC)
Eduardo Horvath <[email protected]> wrote:

> 
> Don't look at the Solaris, Illuminos, or Linux kernels.  They use a 
> completely different trap and address space model from NetBSD.
> 

OK I suspected something similar. I tried tracing NetBSD trap handlers
to understand how they work, but it doesn't seem to work in ddb. I can
set breakpoints on various *spill* trap handlers but they are never
triggered when calling a recursive user space test program which creates
a deep (32 nested functions) window stack.

# nm /netbsd | grep spill
00000000010113c0 t checkalignspill
0000000001001280 t kspill4
0000000001009280 t kspill4_sun4vt0
000000000100d280 t kspill4_sun4vt1
0000000001001200 t kspill8
0000000001009200 t kspill8_sun4vt0
000000000100d200 t kspill8_sun4vt1
0000000001005200 t nucleus_kspill
0000000001005000 t nucleus_uspill
0000000001005400 t nucleus_uspillk
00000000010112a8 t pcbspill
0000000001011388 t pcbspill_fail
000000000101129c t pcbspill_normals
00000000010112a4 t pcbspill_others
0000000001001200 t user_kspill
0000000001001000 t user_uspill
0000000001001400 t user_uspillk
0000000001001080 t uspill4
0000000001009080 t uspill4_sun4vt0
0000000001001000 t uspill8
0000000001009000 t uspill8_sun4vt0
0000000001001480 t uspillk4
0000000001009480 t uspillk4_sun4vt0
0000000001001400 t uspillk8
0000000001009400 t uspillk8_sun4vt0
0000000001010328 t winfixspill

Maybe ddb (which itself relies on traps) cannot stop at breakpoints in
trap handlers?

> Remember, sun4u can use all 4 trap levels.  sun4v can only use 2 because 
> the other 2 are reserved for the hypervisor.  On sun4v the hypervisor and 
> LDOMs are always active, whether configured or not.
> 

I'm confused about how the transition between Trap Levels (TLs)
happens. Hardware documentation gives this table for sun4v TLs usage:

TL Exec Mode        Usage
---------------------------------------------------------------------------
0  Nonprivileged    Normal execution
1  Privileged       System calls; interrupt handlers; instruction emulation
2  Privileged       Window spill/fill handler
3  Hyperprivileged  Real address TLB miss handler
4  Hyperprivileged  Reserved for error handling
5  Hyperprivileged  RED_state handler

So let's assume a user process makes a function call which results in a
window spill trap. The user process is running at TL=0, but the
documentation states that TLs are incremented sequentially, so how does
it get to TL=2 as stated in the table above? It would be running at TL=1.

When I look at NetBSD sun4u spill trap handlers, these are virtually
the same for TL=0 and TL>0. I assume the "supervisor mode" is a synonym
for the "nucleus mode".

/* Traps from TL=0 -- traps from user mode */
TABLE(uspill):
  SPILL64(uspill8,ASI_AIUS)   ! 0x080 spill_0_normal -- used to save user windows in user mode
  SPILL32(uspill4,ASI_AIUS)   ! 0x084 spill_1_normal
  SPILLBOTH(uspill8,uspill4,ASI_AIUS)  ! 0x088 spill_2_normal
  UTRAP(0x08c); TA32          ! 0x08c spill_3_normal
TABLE(kspill):
  SPILL64(kspill8,ASI_N)      ! 0x090 spill_4_normal -- used to save supervisor windows
  SPILL32(kspill4,ASI_N)      ! 0x094 spill_5_normal
  SPILLBOTH(kspill8,kspill4,ASI_N) ! 0x098 spill_6_normal
  UTRAP(0x09c); TA32          ! 0x09c spill_7_normal
TABLE(uspillk):
  SPILL64(uspillk8,ASI_AIUS)  ! 0x0a0 spill_0_other -- used to save user windows in supervisor mode
  SPILL32(uspillk4,ASI_AIUS)  ! 0x0a4 spill_1_other
  SPILLBOTH(uspillk8,uspillk4,ASI_AIUS) ! 0x0a8 spill_2_other
...
/* Traps from TL>0 -- traps from supervisor mode */
TABLE(uspill):
  SPILL64(1,ASI_AIUS)         ! 0x080 spill_0_normal -- used to save user windows
  SPILL32(2,ASI_AIUS)         ! 0x084 spill_1_normal
  SPILLBOTH(1b,2b,ASI_AIUS)   ! 0x088 spill_2_normal
  UTRAP(0x08c); TA32          ! 0x08c spill_3_normal
TABLE(kspill):
  SPILL64(1,ASI_N)            ! 0x090 spill_4_normal -- used to save supervisor windows
  SPILL32(2,ASI_N)            ! 0x094 spill_5_normal
  SPILLBOTH(1b,2b,ASI_N)      ! 0x098 spill_6_normal
  UTRAP(0x09c); TA32          ! 0x09c spill_7_normal
TABLE(uspillk):
  SPILL64(1,ASI_AIUS)         ! 0x0a0 spill_0_other -- used to save user windows in nucleus mode
  SPILL32(2,ASI_AIUS)         ! 0x0a4 spill_1_other
  SPILLBOTH(1b,2b,ASI_AIUS)   ! 0x0a8 spill_2_other
...

So how do they differ between various TLs?

When TL=0:
  uspill:  spill windows when in user address space?

  kspill:  spill windows when in kernel address space?

  uspillk: spill windows for user space process when in kernel space?
           When would this happen? User process making a syscall which
           then results in that syscall running in kernel address space
           but needing to spill windows for user process?

When TL>1:
  These are the same, so when would these traps occur? Are these nested
  traps, e.g. a user process causes a trap (syscall, pagefault, etc)
  which then causes another trap related to window spill?