14-CURRENT Corrupting SPI Flash on Panic?

Daniel Morante <[email protected]>
Newsgroups gmane.os.freebsd.devel.arm
Message-ID <[email protected]> 
Sometimes (not always) when there is a kernel panic the hardware will no 
longer POST after resetting.  I would have to power down the system AND 
remove all power cables.  Then after reconnecting power its back to a 
bootable state.  I first noticed this happening on 
14-CURRENT-3ede04c78c7.  Snapshots prior to that didn't exhibit this 
behavior. The most recent version I've experienced it on is 
14-CURRENT-45c23c2608e.  This is a ARM64 server (Gigabyte R281-T91).  At 
first I thought it could be bad memory.  I replaced the memory and saw 
no changes.  I sent the server back to the vendor for testing/repair.  
It came back as OK/passing all tests.

Here is the serial over LAN capture of the most recent kernel panic that 
"broke" my server (also attached):
http://venus.morante.net/downloads/unibia/screenshots/freebsd/R281-T91/14-CURRENT-45c23c2608e/panic1.txt

Rebooting the server is no longer possible, the UEFI firmware gets stuck 
here (also attached):
http://venus.morante.net/downloads/unibia/screenshots/freebsd/R281-T91/14-CURRENT-45c23c2608e/stuckboot.txt

This is what a "good" boot should start out like (also attached):
http://venus.morante.net/downloads/unibia/screenshots/freebsd/R281-T91/14-CURRENT-45c23c2608e/goodboot.txt

It seems as if the kernel panic is corrupting the firmware or 
overwriting memory somewhere (sorry I don't fully understand UEFI yet) 
with 1's
panic1.txt (text/plain, 10.1 KB) 
CPU252: Cavium ThunderX2 r1p2 affinity:  1 28  3
CPU253: Cavium ThunderX2 r1p2 affinity:  1 29  3
CPU254: Cavium ThunderX2 r1p2 affinity:  1 30  3
CPU255: Cavium ThunderX2 r1p2 affinity:  1 31  3
Release APs...done
RAS CONTROLLER: Fatal unrecoverable error detected

        *** NBU Error ***


  MPIDR= 0x81000000
  CTX_X0= ffffa0000bd30600
  CTX_X1= ffff000000f38430
  CTX_X2= ffff000000772098
  CTX_X3= 286
  CTX_X4= 0
  CTX_X5= ffff00011db95290
  CTX_X6= f001ee10
  CTX_X7= 0
  CTX_X8= 1
  CTX_X9= 0
  CTX_X10= ffff000000e5ee78
  CTX_X11= 1
  CTX_X12= 1
  CTX_X13= ffffa00018149080
  CTX_X14= c1
  CTX_X15= 2af8
  CTX_X16= c1
  CTX_X17= 3
  CTX_X18= ffff000000f35b80
  CTX_X19= 1
  CTX_X20= ffff000000e5ee58
  CTX_X21= ffff000000b99000
  CTX_X22= ffff000000e5ee78
  CTX_X23= ffffa0000bc769c0
  CTX_X24= ff00fffe
  CTX_X25= 43000a10
  CTX_X26= ffff000000f2d088
  CTX_X27= ffff00005574e000
  CTX_X28= ffff0000007720c4
  CTX_X29= ffff00011db95290
  CTX_X30= ffff0000007720c4
  CTX_X31= ffffa0000bd30ad0
  CTX_SCR_EL3= 735
  CTX_RUNTIME_SP= 6e63dc0
  CTX_SPSR_EL3= 404002c5
  CTX_ELR_EL3= ffff000000772124
 Node 0 NBU 0 Error report :
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820044c
      NBU_REG_BAR_ADDRESS_ERROR_REG1 : 0x3ffffdc0
      NBU_REG_BAR_ADDRESS_ERROR_REG2 : 0x00000000
      Physical Address : 0x3ffffdc0
      
NBU BAR Error : Decoded info :
        Agent info : CPU
            Core ID : 1
            Thread ID : 0
        Requ: type : 4 : Write Back
 Node 0 NBU 1 Error report :
 NBU BAR Error
      NBUtimeout stoDRiSg_cpusR
                               REG0 : 0x0820044c
fault  Ntr_, G_Br: 0DfRfSS00109d_7000 : 0x00000000
      Physical Address : 0x3ffffdb0

cpuid = 39      
NBU BAR Error : Decoded info :
        Agent info : CPU
    
 ime = 2re ID : 1
            Thread ID : 0
        Requ: type : 4 : Writ
tack backtrace: Node 0 NBU 2 Error report :
 NBU BAR Error
at     Ndb_tracB_RelDDRESS_ERROR_REG1 : 0x3ffffd70
      NBU_REG_BAR_ADDRESS_ERROR_REG2 : 0x00000000
    Pxysic:lr: paesi   0x3ffffd70
      
NBU BAR Error : Decoded info :
        Agent info : CPU
           tCo  d pid 0 t d 10 51    Thread ID : 0
        Requ: type : 4 : Write Back
 Node 0 NBU 3 Error report :
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820041c
      NBU_
St pReADa ffff00R     k1b_ nx4r+0f440
   : undefinedB     Df9020E1fOR_REG2 : 0x00000000
      Physical Address : 0x3ffffd30
      
NBU BAR Error : Decoded info :
        Agent info : CPU
            Core ID : 1
      
       b>x2:hfeff I0 :08
 b2     Re(u: txpcve 1 :ece_p
c_title + 33397) Node 0 NBU 4 Error report :
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820044c
      NBU_REG_BAR_ADDRESS_ERROR_REG1 : 0x
ff  0 0N087R645BAR_AD (ESS_sRROu__tGble +x200dd)00
      Physical Address : 0x3ffff8b0
      
NBU BAR Error : Decoded info :
        Agent info : CPU
            Core ID : 1
            Thre
         Re u: type : 4 : Write Back
 Node 0 NBU 5 Error report :
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820044c
      NBU_REG_BAR_ADDRESS_E
                           ROx5:Ef1 f 0031df14700
                                                       NBU_REG_BAR_ADDRESS_ERROR_REG2 : 0x00000000
      Physical Address : 0x3ffff8f0
      
NBU BAR Error : Decoded info :
        Agent info : CPU
    0        Thread ID : 0
        Requ: type : 4 : Write Back
 Node 0 NBU 6 Error report :
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820044c
  x7   B    G  A   DD0ESS_ERROR_REG1 : 0x3ffffc40
      NBU_REG_BAR_ADDRESS_ERROR_REG2 : 0x00000000
      Physical Address : 0x3ffffc40
      
NBU BAR Error : Decoded info :
  x8:      t i f    CPU
            Core ID : 1
            Thread ID : 0
        Requ: type : 4 : Write Back
 Node 0 NBU 7 Error report :
 NBU BAR Error
c_ r e Nsk_+EG8)AR_ADDRESS_ERROR_REG1 : 0x3ffffc30
      NBU_REG_BAR_ADDRESS_ERROR_REG2 : 0x00000000
      Physical Address : 0x3ffffc30
      
NBU BAR Error : Decoded info :
      
      )      Thread ID : 0
        Requ: type : 4 : Write Back
 Node 0 NBU 8 Error report :
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820044c
      NBU_REG_BAR_ADD
 x11R   _     : 0x3ff f8e0
      NBU_REG_BAR_ADDRESS_ERROR_REG2 : 0x00000000
      Physical Address : 0x3fffffe0
      
NBU BAR Error : Decoded info :
        Agent info : CPU
            Core ID : 

  :         T600ad ID : 0
        Requ: type : 4 : Write Back
 Node 0 NBU 9 Error report :
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820044c
      NBU_REG_BAR_ADDRESS_ERROR_REG1 
       2Bf8REG_BAR_ADDRESS_ERROR_REG2 : 0x00000000
      Physical Address : 0x3ffffbf0
      
NBU BAR Error : Decoded info :
        Agent info : CPU
            Core ID : 1
         
 xhr:a   D : 0 
                    2  Requ: type : 4 : Write Back
 Node 0 NBU 10 Error report :
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820044c
      NBU_REG_BAR_ADDRESS_ERROR_REG1 : 0x3fffff40
   15:     EG  A _A2aRESS_ERROR_REG2 : 0x00000000
      Physical Address : 0x3fffff40
      
NBU BAR Error : Decoded info :
        Agent info : CPU
            Core ID : 1
            Thread ID : 0
 x  :   R q : ty   :e4 : Write Back
 Node 0 NBU 11 Error report :
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820044c
      NBU_REG_BAR_ADDRESS_ERROR_REG1 : 0x3fffff30
      NBU_REG_BAR_AD
                     ES7:ERR    EG  : 0  0000000
      Physical Address : 0x3fffff30
      
NBU BAR Error : Decoded info :
        Agent info : CPU
            Core ID : 1
            Thread ID : 0
        Requ: type :
 x 8r ffffa0k
5b0bfe0 Node 0 NBU 12 Error report :
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820044c
      NBU_REG_BAR_ADDRESS_ERROR_REG1 : 0x3ffffef0
      NBU_REG_BAR_ADDRESS_ERROR_REG2 : 0x00000000

                                                 x     P  s   l A dress : 0x3ffffef0
      
NBU BAR Error : Decoded info :
        Agent info : CPU
            Core ID : 1
            Thread ID : 0
        Requ: type : 4 : Write Back

 x20:                1 Node 0 NBU 13 Error report :
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820044c
      NBU_REG_BAR_ADDRESS_ERROR_REG1 : 0x3ffffeb0
      NBU_REG_BAR_ADDRESS_ERROR_REG2 x 1:00  0 0  
                                                        0hysical Address : 0x3ffffeb0
      
NBU BAR Error : Decoded info :
        Agent info : CPU
            Core ID : 1
            Thread ID : 0
        Requ: 
0 Node 0 NBU 14 Error report :ca90
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820044c
      NBU_REG_BAR_ADDRESS_ERROR_REG1 : 0x3ffffe70
R G23:RfADDRESS_ERRO48a1G2 : 0x(0o0ex0cve.  x  v_psocat Ald + 2 86)x3ffffe70
      
NBU BAR Error : Decoded info :
        Agent info : CPU
            Core ID : 1
            Thread ID : 0
        Requ: type : 4 : Write Back

 x24: ffff0000008e7041 (digits + 194da) Node 0 NBU 15 Error report :
 NBU BAR Error
      NBU_REG_BAR_ADDRESS_ERROR_REG0 : 0x0820044c
      NBU_REG_BAR_ADDRESS_ERROR_REG1 : 0x3ffffe10
      NBU_REG_BAR_A
                   DxE5S_E R R_REG     x10000000
      Physical Address : 0x3ffffe10
      
NBU BAR Error : Decoded info :
        Agent info : CPU
            Core ID : 1
            Thread ID : 

    f f 0001:5t0def: 4 : Write Back

Current NBU DRAM BAR setting:
Node0 BAR0 Base 00004000 Limit 00007FFC chan_xlation 00004008 node_xlation 00000000
Node0 BAR1 Base 00080001 Limit
4Node0 BAR2 Base 00880001 Limit 00FFCFFC chan_xlation 007FD008 node_xlation 00000000
Node0 BAR3 Base 00FFD001 Limit 00FFFFFC chan_xlation 00FFD008 node_xlation 00000002

ox20:Bfff 0a0e008806451 Limit(0amFCFFtus_anbxlat o3 d87FD008 node_xlation 00000002
Node0 BAR5 Base FFFFF000 Limit 00000000 chan_xlation 00000000 node_xlation 00000000
Node0 BAR6 Base FFFFF000 Limit 00000000 chan_xlation 000000 x 9:de_fl0tio1 000f0000
Node0 BAR7 Base FFFFF000 Limit 00000000 chan_xlation 00000000 node_xlation 00000000
0 0s0 BAR8 Base FFFFF000 Limit 00000000 chan_xlation 00000000 node_xlation 0000
    :odeffB009 5a0efF0FFF000 Limit 00000000 chan_xlation 00000000 node_xlation 00000000
Node0 BAR10 Base FFFFF000 Limit 00000000 chan_xlation 00000000 node_xlation 00000000
N
e0lB:Rffff0se0F07FF000 Limit 00o00l1h_cyac_xla5i)n 00000000 node_xlation 00000000
Node0 BAR12 Base FFFFF000 Limit 00000000 chan_xlation 00000000 node_xlation 00000000
Node0 BAR13 Base FFFFF000 Limit  0lr0 fffc0a0_xlati7b 0000000 (uode__intno+ 10)00000
Node0 BAR14 Base FFFFF000 Limit 00000000 chan_xlation 00000000 node_xlation 00000000
Node0 BAR15 Base FFFFF000 Limit 00000000 chan_xlatisr  000 000 80400xc5tion 00000000
Node1 BAR0 Base 00004000 Limit 00007FFC chan_xlation 00004008 node_xlation 00000000
Node1 BAR1 Base 00080001 Limit 
                                fFrFF  c   _x   i   0007C008 node_xlation 00000000
Node1 BAR2 Base 00880001 Limit 00FFCFFC chan_xlation 007FD008 node_xlation 00000000
epa0ic: U1 Lnmltd00FFFFFC cnal xatt on r0FFD008 node_xlation 00000002
Node1 BAR4 Base 08800001 Limit 08FFCFFC chan_xlation 087FD008 node_xlation 00000002
Node1 BAR5 Base FFFFF000 Limit 00000
0 cian=x1ation 00000000 node_xlation 00000000
Node1 BAR6 Base FFFFF000 Limit 00000000 chan_xlation 00000000 node_xlation 00000000
Node1 BAR7 Base FFFF
t0mL mit 00000000 chan_xlation 00000000 node_xlation 00000000
Node1 BAR8 Base FFFFF000 Limit 00000000 chan_xlation 00000000 node_xlation 00000000
Node1
ADB:BsteckFFFc000aLi:it 00000000 chan_xlation 00000000 node_xlation 00000000
Node1 BAR10 Base FFFFF000 Limit 00000000 chan_xlation 00000000 node_xlation 00000000
Node1 BAR_trBcs_ eFfF) a0 Limid 00r0c0_selfan_xlation 00000000 node_xlation 00000000
Node1 BAR12 Base FFFFF000 Limit 00000000 chan_xlation 00000000 node_xlation 00000000
Node1 BAR13 Base FFFF
                     00Uptime: 20000000 chan_xlation 00000000 node_xlation 00000000
Node1 BAR14 Base FFFFF000 Limit 00000000 chan_xlation 00000000 node_xlation 00000000
5 Base FFFFF000 Limit 00000000 chan_xlation 00000000 node_xlation 00000000

db> bt   
Tracing pid 0 tid 100512 td 0xffffa00008093000
db_trace_self() at db_trace_self
db> reset
stuckboot.txt (text/plain, 2.3 KB) 
Rom...
CRC: len=0xf080, cal=0x27ff5de9, img=0x27ff5de9, match!

Loading from boot device SPI NOR 
Header:
  000|0x23ffdc0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  010|0x23ffdd0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  020|0x23ffde0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  030|0x23ffdf0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  040|0x23ffe00:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  050|0x23ffe10:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  060|0x23ffe20:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  070|0x23ffe30:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  080|0x23ffe40:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  090|0x23ffe50:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  0A0|0x23ffe60:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  0B0|0x23ffe70:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  0C0|0x23ffe80:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  0D0|0x23ffe90:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  0E0|0x23ffea0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  0F0|0x23ffeb0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  100|0x23ffec0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  110|0x23ffed0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  120|0x23ffee0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  130|0x23ffef0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  140|0x23fff00:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  150|0x23fff10:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  160|0x23fff20:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  170|0x23fff30:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  180|0x23fff40:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  190|0x23fff50:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  1A0|0x23fff60:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  1B0|0x23fff70:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  1C0|0x23fff80:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  1D0|0x23fff90:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  1E0|0x23fffa0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  1F0|0x23fffb0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF
goodboot.txt (text/plain, 2.9 KB) 
Rom...
CRC: len=0xf080, cal=0x27ff5de9, img=0x27ff5de9, match!

Loading from boot device SPI NOR 
Header:
  000|0x23ffdc0:   01 02 FF FF 4C 46 43 53  00 04 00 00 10 FB 05 00 
  010|0x23ffdd0:   01 04 FF FF 4C 46 43 53  00 00 06 00 D8 92 00 00 
  020|0x23ffde0:   01 03 FF FF 4C 46 43 53  00 00 07 00 15 8E 00 00 
  030|0x23ffdf0:   01 05 FF FF 4C 46 43 53  00 00 08 00 00 00 0B 01 
  040|0x23ffe00:   01 06 FF FF 4C 46 43 53  00 00 13 01 FD 17 00 00 
  050|0x23ffe10:   01 14 FF FF 4C 46 43 53  00 00 14 01 8C 3E 00 00 
  060|0x23ffe20:   7F FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  070|0x23ffe30:   7F FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  080|0x23ffe40:   7F FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  090|0x23ffe50:   7F FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  0A0|0x23ffe60:   7F FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  0B0|0x23ffe70:   01 16 FF FF 4C 46 43 53  00 00 15 01 00 00 0A 00 
  0C0|0x23ffe80:   01 07 FF FF 4C 46 43 53  00 00 1F 01 E0 3D 1B 00 
  0D0|0x23ffe90:   7F FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  0E0|0x23ffea0:   7F FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  0F0|0x23ffeb0:   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
  100|0x23ffec0:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  110|0x23ffed0:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  120|0x23ffee0:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  130|0x23ffef0:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  140|0x23fff00:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  150|0x23fff10:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  160|0x23fff20:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  170|0x23fff30:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  180|0x23fff40:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  190|0x23fff50:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  1A0|0x23fff60:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  1B0|0x23fff70:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  1C0|0x23fff80:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  1D0|0x23fff90:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  1E0|0x23fffa0:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
  1F0|0x23fffb0:   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 

Loading Normal Image from table offset 400 
CRC: len=0x5fb10, cal=0x1352ae58, img=0x1352ae58, match!

 Jump to Boot1 @0x2200000

=====================
 Cavium CN99XX BOOT1
=====================
 Version : TX2-FW-Release-7.3-build_01
 Built at 16:20:32 on Aug 13 2020
=====================

SPI NOR device inform: node=0, cs=0, khz=0
  Chip Selection : 0
     Device Name : 
    Manufacturer : 0x20, MICRON
        ID Codes : 0xba20, 0x1044
  Bytes Per Page : 0x100
 Page Per Sector : 0x100
   Total Sectors : 0x400
OTP: Freq: core 2199, mem 2199, socn 666, socs 1199
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.