Re: LSI SAS3008 on 13-RC5 and 14-CURRENT

Yikes... let me try to reformat the message display:

> I done extensive testing with many different configurations. I 
> apologize in advance for the length of this 😉.
>
> I have an arm64 system with an onboard SAS controller (Broadcom® 
> SAS3008) and expander (Broadcom® SAS35x36R). This system also has an 
> onboard SATA controller that is currently unusable in any version of 
> FreeBSD ARM64 
> (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254651).  As a 
> result I am limited to using the SAS3008.
>
> I've noticed strange behavior on 13-RC3/4/5 and on the latest 
> 14-CURRENT snapshot (2021-04-01).In the controller firmware I can 
> create a logical volume (RAID) or use the individual drives 
> (non-RAID/pass thru). In RAID mode there is a only logical volume and 
> no individual drives.
>
> I use the DVD (or disc1) ISO image as the installation media.  I load 
> the kernel module (kldload mpr) using a shell and then proceed to 
> install FreeBSD using either ZFS or UFS on root.
>
> With 13.0 the drives or logical volumes are detected correctly. On 
> 14.0 no logical volumes are detected.
>
> mpr0: <Avago Technologies (LSI) SAS3008> mem 
> 0x42140000-0x4214ffff,0x42100000-0x4213ffff at device 0.0 numa-domain 
> 0 on pci14
> mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd
> mpr0: IOCCapabilities: 
> 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a> 
> enclosureHandle<0x0002> slot 0
> mpr0: At enclosure level 0 and connector name ( )
> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000b> 
> enclosureHandle<0x0002> slot 1
> mpr0: At enclosure level 0 and connector name ( )
> mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> 
> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 16
> mpr0: At enclosure level 0 and connector name ( )
>
> This volume only shows up on 13.0 and not on 14.0:
>
> da1 at mpr0 bus 0 scbus2 target 0 lun 0
> da1: <LSI Logical Volume 3000> Fixed Direct Access SPC-4 SCSI device
> da1: Serial Number 2692521697153322079
> da1: 150.000MB/s transfers
> da1: Command Queueing enabled
> da1: 226974MB (464842752 512 byte sectors)
>
> In Non-RAID mode individual drives will be shown in both 13.0 and 14.0.
>
> I can proceed through the OS installation, but it's painfully slow.  

> Towards the end (at which point I assume it's writing bootcode... 
> before asking to set the root password), the following messages begin 
> to appear in the console:
>
> mpr0: IOC Fault 0x40002622, Resetting
> mpr0: Reinitializing controller
> mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd
> mpr0: IOCCapabilities: 
> 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a> 
> enclosureHandle<0x0002> slot 0
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000b> 
> enclosureHandle<0x0002> slot 1
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> 
> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 16
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: IOC Fault 0x40002622, Resetting
> mpr0: Reinitializing controller
> mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd
> mpr0: IOCCapabilities: 
> 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a> 
> enclosureHandle<0x0002> slot 0
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000b> 
> enclosureHandle<0x0002> slot 1
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> 
> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 16
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: IOC Fault 0x40002622, Resetting
> mpr0: Reinitializing controller
> mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd
> mpr0: IOCCapabilities: 
> 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a> 
> enclosureHandle<0x0002> slot 0
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000b> 
> enclosureHandle<0x0002> slot 1
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> 
> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 16
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: IOC Fault 0x40002622, Resetting
> mpr0: Reinitializing controller
> mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd
> mpr0: IOCCapabilities: 
> 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
> (da1:mpr0:0:0:0): Invalidating pack
> g_vfs_done():da1p1[WRITE(offset=512, length=4096)]error = 6
> g_vfs_done():da1p1[WRITE(offset=33792, length=4096)]error = 6
> da1 at mpr0 bus 0 scbus2 target 0 lun 0
> da1: <LSI Logical Volume 3000>  s/n 2692521697153322079 detached
> g_vfs_done():da1p1[WRITE(offset=99840, length=16384)]error = 6
> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a> 
> enclosureHandle<0x0002> slot 0
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000b> 
> enclosureHandle<0x0002> slot 1
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> 
> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 16
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: IOC Fault 0x40002622, Resetting
> mpr0: Reinitializing controller
> (da1:mpr0:0:0:0): WRITE(10). CDB: 2a 00 00 00 0a cb 00 08 00 00
> (da1:mpr0:0:0:0): CAM status: SCSI Bus Reset Sent/Received
> (da1:mpr0:0:0:0): Error 5, Periph was invalidated
> g_vfs_done():da1p1[WRITE(offset=1394176, length=1048576)]error = 5
> (da1:mpr0:0:0:0): WRITE(6). CDB: 0a 00 01 0b 20 00
> (da1:mpr0:0:0:0): CAM status: SCSI Bus Reset Sent/Received
> (da1:mpr0:0:0:0): Error 5, Periph was invalidated
> g_vfs_done():da1p1[WRITE(offset=116224, length=16384)]error = 5
> fsync: giving up on dirty (error = 5) (da1:mpr0:0:0:0): WRITE(10). 
> CDB: 2a 00 01 d7 0d 08 00 00 08 00
> 0xffffa0002797b5b8: (da1:mpr0:0:0:0): CAM status: SCSI Bus Reset 
> Sent/Received
> type VCHR
> (da1:mpr0:0:0:0): Error 5, Periph was invalidated
>     usecount 1, writecount 0, refcount 6 seqc users 0(da1:mpr0:0:0:0): 
> WRITE(10). CDB: 2a 00 01 52 b0 a8 00 00 08 00
>  rdev 0xffffa0000af73400
> (da1:mpr0:0:0:0): CAM status: SCSI Bus Reset Sent/Received
>     hold count flags ()
> mpr0: Firmware: 15.00.00.00, Driver: 23.00.00.00-fbsd
>     flags ()mpr0: IOCCapabilities: 
> 6985c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR,MSIXIndex,FastPath,RDPQArray>
>
> (da1:mpr0:0:0:0): Error 5, Periph was invalidated
>     v_object 0xffffa00027a6f528 ref 0 pages 27 cleanbuf 2 dirtybuf 1
>     lock type devfs: EXCL by thread 0xffffa0000aac3000 (pid 271, 
> syncer, tid 101634)
>  with exclusive waiters pending
>         dev da1p1
> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000a> 
> enclosureHandle<0x0002> slot 0
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: Found device <81<SataDev>,End Device> <6.0Gbps> handle<0x000b> 
> enclosureHandle<0x0002> slot 1
> mpr0: At enclosure level 0 and connector name (    )
> mpr0: Found device <4411<SmpInit,SspTarg,SepDev>,End Device> 
> <12.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 16
> mpr0: At enclosure level 0 and connector name (    )
>
> The above continues indefinitely and the system eventually locks up 
> (no kernel panic).  I've tried it countless times and it always ends 
> the same.  Regardless of how I configure the controller (RAID 1, 0, or 
> pass thru).
>
> To rule out a bad controller I test with Ubuntu 20.10 for arm64. No 
> issues, everything is detected, SAS, onboard SATA, and Ethernet.  Not 
> surprising since Gigabyte has Ubuntu on the compatibility list for 
> this system.  All hardware is confirmed to be functional.
>
> This is were things get strange...
>
> I go back to test FreeBSD 13.0 and I no longer have issues with the 
> SAS controller (onboard SATA still not working).  It seems that 
> installing/booting Ubuntu did something to 'fix' the problem?  Maybe 
> it loaded some type of firmware or places some persistent data in 
> UEFI? (I'm not sure how that works). Regardless I was able to install 
> FreeBSD on to the SAS controller, and I can boot it.
>
> Has anyone seen this happen before? Should I file another bug report?
>
> 14.0 btw, still can't see logical volumes.  Should that be a separate 
> bug report?
>
>