aac(4) vs. Dell PERC 3/Di _and_ HP NetRAID-4M in same host

I've got a Dell PowerEdge 2650 here with that I've added an HP
NetRAID-4M card to for additional external storage.

Has anyone here successfully used a similar configuration with NetBSD
(or FreeBSD or OpenBSD, or even GNU/Linux)?  Any kind of system with
both Dell PERC and HP NetRAID in the same box?

This one boots up with netbsd-4 saying, in part, the following:

aac0 at pci2 dev 8 function 1: Dell PERC 3/Di
aac0: interrupting at ioapic1 pin 14 (irq 11)
aac0: i960RX at 100MHz, 128MB mem (118MB cache), optional battery present
ld0 at aac0 unit 0: RAID 5
ld0: 135 GB, 17700 cyl, 255 head, 63 sec, 512 bytes/sect x 284365824 sectors

aac1 at pci4 dev 6 function 0: HP NetRAID-4M
aac1: interrupting at ioapic1 pin 0 (irq 7)
aac1: StrongARM SA110 at 233MHz, 144MB mem (128MB cache), required battery present
ld1 at aac1 unit 0: RAID 5
ld1: 95450 MB, 12168 cyl, 255 head, 63 sec, 512 bytes/sect x 195482496 sectors


ld0 works just fine.  I was able to install onto it from sysinst, and
I've done a full source build on it right to the binary sets without
trouble, albiet a bit slower than it could be, apparently.

However ld1 behaves very bizarrely.  I can write to it at what seems to
be full blast.  I can read from it too, but slower than it should be, I
think (a lot slower than it writes!), and all I ever get back are zero
value bytes.  There are no errors, not a peep from anything, just lots
of blocks of zeros, no matter what's been "written" previously.

The NetRAID controller itself says the container is A-OK.

In fact I've tried two different HP NetRAID-4M controllers.  They each
have slightly different firmware, and they came from totally different
parts of the world.  However both are happy to say the container status
is A-OK, and neither give any errors.

(well there are status updates if the array is rebuilding during and
after NetBSD boots, but those seem normal -- the array status in the
controller interface still ends up being "OK") 


FreeBSD-6.2 seems to have similar problems with the NetRAID controller,
though its behaviour is slightly different.  With FreeBSD's devfs a
device file is created automatically in /dev for each container, but one
is not created for aac1.  I can still write to the raw device, but again
if I read from it I only ever get zeros.  I haven't done an actual
install of FreeBSD yet, but the kernel on the CD has the driver in it
and I'm able to play with the likes of dd from the "live" tools on the
CD.  Since there's no node for the container I can't really disklabel it.


I'd like to try OpenBSD's aac(4) variant, especially since it hooks into
the SCSI layer instead of using ld(4), but at the same time has most of
the improvements and fixes from FreeBSD.  Unfortunately their INSTALL
kernel (on their "cd40.iso" image) doesn't have the aac(4) driver in it.

(in fact if I can get this hardware to work right then I'd really like
to port OpenBSD's aac(4) back over to NetBSD!)


I've been wondering if there might be some issue with the controller
BIOS.  The BIOS is _not_ installed for the NetRAID controller.  Maybe
I'll try reconfiguring it a little later tonight so that its BIOS is
installed at boot.

The only thing I have done in the system BIOS is change the interrupt
for the NetRAID card to avoid sharing with another high-speed device.


I've got two more HP NetRAID controllers of lesser capacity that I can
also try....

I suppose I could also try simpler/multiple container configurations to
see if that makes any difference.  I could even write some known data
and then boot with a plain SCSI controller to see if that data can be
found anywhere on any of the disks, thus maybe sorting out whether the
error is with writing or with reading, though I strongly suspect the
problem is with reading.


There's also some possible strangeness with the PCI bus probing.  This
kernel is GENERIC.MPACPI (adding "options ACPI_SCANPCI" didn't seem to
make any difference in the behaviour).  The PERC seems to be seen on two
separate buses (and maybe 3), but it's only recognized and matched by
the aac(4) driver on the last one:

pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
...
Dell Computer PERC 3/Di (undefined subclass 0x00) at pci0 dev 4 function 1 not configured
...
pchb5 at pci0 dev 16 function 2
pchb5: ServerWorks CIOB-X2 PCI-X bridge (rev. 0x03)
pci2 at pchb5 bus 4
pci2: i/o space, memory space enabled
ppb0 at pci2 dev 8 function 0: Intel product 0x0309 (rev. 0x01)
pci3 at ppb0 bus 5
pci3: i/o space, memory space enabled
Adaptec (2nd PCI Vendor ID) AIC-7899F RAID (SCSI mass storage, revision 0x01) at pci3 dev 6 function 0 not configured
Adaptec (2nd PCI Vendor ID) AIC-7899F RAID (SCSI mass storage, revision 0x01) at pci3 dev 6 function 1 not configured
aac0 at pci2 dev 8 function 1: Dell PERC 3/Di
aac0: interrupting at ioapic1 pin 14 (irq 11)
aac0: i960RX at 100MHz, 128MB mem (118MB cache), optional battery present


The NetRAID controller, which is plugged into one of the three PCI-X
slots in the machine, is only seen once, alone on its bus:

pchb6 at pci0 dev 17 function 0
pchb6: ServerWorks CIOB-X2 PCI-X bridge (rev. 0x03)
pci4 at pchb6 bus 1
pci4: i/o space, memory space enabled
aac1 at pci4 dev 6 function 0: HP NetRAID-4M
aac1: interrupting at ioapic1 pin 0 (irq 7)
aac1: StrongARM SA110 at 233MHz, 144MB mem (128MB cache), required battery present


There's other weirdness with unrecognized bridges and such.  The whole
dmesg is below.


-- 
						Greg A. Woods

H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack <[email protected]>
Planix, Inc. <[email protected]>       Secrets of the Weird <[email protected]>


Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 4.0_BETA2 (GENERIC.MPACPI) #0: Thu Feb 15 12:15:32 EST 2007
        woods@always:/build/woods/always/netbsd-4-i386-i386-ppro-obj/building/work/woods/m-NetBSD-4/sys/arch/i386/compile/GENERIC.MPACPI
total memory = 3839 MB
avail memory = 3762 MB
timecounter: Timecounters tick every 10.000 msec
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
BIOS32 rev. 0 found at 0xffe90
mainbus0 (root)
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel Xeon (686-class), 2385.55 MHz, id 0xf27
cpu0: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu0: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu0: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu0: features2 400<CID>
cpu0: "Intel(R) Xeon(TM) CPU 2.40GHz"
cpu0: I-cache 12K uOp cache 8-way, D-cache 8 KB 64B/line 4-way
cpu0: L2 cache 512 KB 64B/line 8-way
cpu0: ITLB 4K/4M: 64 entries
cpu0: DTLB 4K/4M: 64 entries
cpu0: enabling thermal monitor 1 ... enabled.
cpu0: calibrating local timer
cpu0: apic clock running at 99 MHz
cpu0: 16 page colors
cpu1 at mainbus0: apid 2 (application processor)
cpu1: starting
cpu1: Intel Xeon (686-class), 2385.45 MHz, id 0xf27
cpu1: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu1: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu1: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu1: features2 400<CID>
cpu1: "Intel(R) Xeon(TM) CPU 2.40GHz"
cpu1: I-cache 12K uOp cache 8-way, D-cache 8 KB 64B/line 4-way
cpu1: L2 cache 512 KB 64B/line 8-way
cpu1: ITLB 4K/4M: 64 entries
cpu1: DTLB 4K/4M: 64 entries
cpu1: using thermal monitor 1
cpu2 at mainbus0: apid 1 (application processor)
cpu2: starting
cpu2: Intel Xeon (686-class), 2385.45 MHz, id 0xf27
cpu2: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu2: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu2: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu2: features2 400<CID>
cpu2: "Intel(R) Xeon(TM) CPU 2.40GHz"
cpu2: I-cache 12K uOp cache 8-way, D-cache 8 KB 64B/line 4-way
cpu2: L2 cache 512 KB 64B/line 8-way
cpu2: ITLB 4K/4M: 64 entries
cpu2: DTLB 4K/4M: 64 entries
cpu2: using thermal monitor 1
cpu3 at mainbus0: apid 3 (application processor)
cpu3: starting
cpu3: Intel Xeon (686-class), 2385.46 MHz, id 0xf27
cpu3: features bfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR>
cpu3: features bfebfbff<PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX>
cpu3: features bfebfbff<FXSR,SSE,SSE2,SS,HTT,TM,SBF>
cpu3: features2 400<CID>
cpu3: "Intel(R) Xeon(TM) CPU 2.40GHz"
cpu3: I-cache 12K uOp cache 8-way, D-cache 8 KB 64B/line 4-way
cpu3: L2 cache 512 KB 64B/line 8-way
cpu3: ITLB 4K/4M: 64 entries
cpu3: DTLB 4K/4M: 64 entries
cpu3: using thermal monitor 1
ioapic0 at mainbus0 apid 4 (I/O APIC)
ioapic0: pa 0xfec00000, version 11, 16 pins
ioapic0: misconfigured as apic 0
ioapic0: remapped to apic 4
ioapic1 at mainbus0 apid 5 (I/O APIC)
ioapic1: pa 0xfec01000, version 11, 16 pins
ioapic1: misconfigured as apic 0
ioapic1: remapped to apic 5
ioapic2 at mainbus0 apid 6 (I/O APIC)
ioapic2: pa 0xfec02000, version 11, 16 pins
ioapic2: misconfigured as apic 0
ioapic2: remapped to apic 6
acpi0 at mainbus0: Advanced Configuration and Power Interface
acpi0: using Intel ACPI CA subsystem version 20060217
acpi0: X/RSDT: OemId <DELL  ,PE2650  ,00000001>, AslId <MSFT,0100000a>
acpi0: SCI interrupting at int 9
acpi0: fixed-feature power button present
timecounter: Timecounter "ACPI-Fast" frequency 3579545 Hz quality 1000
ACPI-Fast 32-bit timer
LUSB: ACPI: Found matching pin for 0.15.INTA at func 2: 5
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
ACPI Object Type 'Processor' (0x0c) at acpi0 not configured
PNP0A03 [PCI/PCI-X Host Bridge] at acpi0 not configured
PNP0200 [AT DMA Controller] at acpi0 not configured
npx0 at acpi0 (PNP0C04)
npx0: io 0xf0-0xff irq 13
npx0: reported by CPUID; using exception 16
PNP0000 [AT Interrupt Controller] at acpi0 not configured
pcppi1 at acpi0 (PNP0800)
pcppi1: io 0x61
pcppi1: children must have an explicit unit
midi0 at pcppi1: PC speaker (CPU-intensive output)
sysbeep0 at pcppi1
attimer1 at acpi0 (PNP0100): AT Timer
attimer1: io 0x40-0x5f irq 0
fdc0 at acpi0 (PNP0700)
fdc0: io 0x3f0-0x3f5,0x3f7 irq 6 drq 2
pckbc0 at acpi0 (PNP0303): kbd port
pckbc0: io 0x60,0x64 irq 1
pckbc1 at acpi0 (PNP0F13): aux port
pckbc1: irq 12
com0 at acpi0 (PNP0501-1)
com0: io 0x3f8-0x3ff irq 4
com0: ns16550a, working fifo
com0: console
com1 at acpi0 (PNP0501-2)
com1: io 0x2f8-0x2ff irq 3
com1: ns16550a, working fifo
PNP0B00 [AT Real-Time Clock] at acpi0 not configured
PNP0C01 [System Board] at acpi0 not configured
PNP0C01 [System Board] at acpi0 not configured
PNP0A03 [PCI/PCI-X Host Bridge] at acpi0 not configured
PNP0A03 [PCI/PCI-X Host Bridge] at acpi0 not configured
PNP0A03 [PCI/PCI-X Host Bridge] at acpi0 not configured
PNP0A03 [PCI/PCI-X Host Bridge] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
PNP0C0F [PCI interrupt link device] at acpi0 not configured
pcppi1: attached to attimer1
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0 mux 1
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0
pchb0: ServerWorks CNB30-HE PCI bridge (rev. 0x13)
pchb1 at pci0 dev 0 function 1
pchb1: ServerWorks CNB30-HE PCI bridge (rev. 0x00)
pchb2 at pci0 dev 0 function 2
pchb2: ServerWorks product 0x0000 (rev. 0x00)
pchb2: unknown ServerWorks chip ID 0x0000; trying to attach PCI buses behind it
Dell Computer product 0x000c (undefined subclass 0x00) at pci0 dev 4 function 0 not configured
Dell Computer PERC 3/Di (undefined subclass 0x00) at pci0 dev 4 function 1 not configured
Dell Computer product 0x000d (undefined subclass 0x00) at pci0 dev 4 function 2 not configured
vga0 at pci0 dev 14 function 0: ATI Technologies Rage XL (rev. 0x27)
wsdisplay0 at vga0 kbdmux 1
wsmux1: connecting to wsdisplay0
wskbd0: connecting to wsdisplay0
pchb3 at pci0 dev 15 function 0
pchb3: ServerWorks CSB5 southbridge (rev. 0x93)
rccide0 at pci0 dev 15 function 1
rccide0: ServerWorks CSB5 IDE Controller (rev. 0x93)
rccide0: bus-master DMA support present
rccide0: primary channel configured to compatibility mode
rccide0: primary channel interrupting at ioapic0 pin 14 (irq 14)
atabus0 at rccide0 channel 0
rccide0: secondary channel wired to compatibility mode
rccide0: secondary channel interrupting at ioapic0 pin 15 (irq 15)
atabus1 at rccide0 channel 1
ohci0 at pci0 dev 15 function 2: ServerWorks OSB4/CSB5 USB Host Controller (rev. 0x05)
ohci0: interrupting at ioapic0 pin 5 (irq 5)
ohci0: OHCI version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0
uhub0: ServerWorks OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
pcib0 at pci0 dev 15 function 3
pcib0: ServerWorks CSB5 ISA/LPC bridge (rev. 0x00)
pchb4 at pci0 dev 16 function 0
pchb4: ServerWorks CIOB-X2 PCI-X bridge (rev. 0x03)
pci1 at pchb4 bus 3
pci1: i/o space, memory space enabled
bge0 at pci1 dev 6 function 0: Broadcom BCM5701 Gigabit Ethernet
bge0: interrupting at ioapic1 pin 12 (irq 10)
bge0: ASIC BCM5701 B5 (0x0105), Ethernet address 00:06:5b:ed:e2:d1
brgphy0 at bge0 phy 1: BCM5701 1000BASE-T media interface, rev. 0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge1 at pci1 dev 8 function 0: Broadcom BCM5701 Gigabit Ethernet
bge1: interrupting at ioapic1 pin 13 (irq 7)
bge1: ASIC BCM5701 B5 (0x0105), Ethernet address 00:06:5b:ed:e2:d2
brgphy1 at bge1 phy 1: BCM5701 1000BASE-T media interface, rev. 0
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
pchb5 at pci0 dev 16 function 2
pchb5: ServerWorks CIOB-X2 PCI-X bridge (rev. 0x03)
pci2 at pchb5 bus 4
pci2: i/o space, memory space enabled
ppb0 at pci2 dev 8 function 0: Intel product 0x0309 (rev. 0x01)
pci3 at ppb0 bus 5
pci3: i/o space, memory space enabled
Adaptec (2nd PCI Vendor ID) AIC-7899F RAID (SCSI mass storage, revision 0x01) at pci3 dev 6 function 0 not configured
Adaptec (2nd PCI Vendor ID) AIC-7899F RAID (SCSI mass storage, revision 0x01) at pci3 dev 6 function 1 not configured
aac0 at pci2 dev 8 function 1: Dell PERC 3/Di
aac0: interrupting at ioapic1 pin 14 (irq 11)
aac0: i960RX at 100MHz, 128MB mem (118MB cache), optional battery present
aac0: Kernel 2.7-1 [Build 3170], Monitor 2.7-1 [Build 3170], S/N A441D3
aac0: Controller supports: 75c<WCACHE,DATA64,HOSTTIME,WINDOW4GB,SOFTERR,NORECOND,SGMAP64>
ld0 at aac0 unit 0: RAID 5
ld0: 135 GB, 17700 cyl, 255 head, 63 sec, 512 bytes/sect x 284365824 sectors
pchb6 at pci0 dev 17 function 0
pchb6: ServerWorks CIOB-X2 PCI-X bridge (rev. 0x03)
pci4 at pchb6 bus 1
pci4: i/o space, memory space enabled
aac1 at pci4 dev 6 function 0: HP NetRAID-4M
aac1: interrupting at ioapic1 pin 0 (irq 7)
aac1: StrongARM SA110 at 233MHz, 144MB mem (128MB cache), required battery present
aac1: Kernel 2.4-1 [Build 4607], Monitor 2.4-1 [Build 4607], S/N 8A1A4F
aac1: Controller supports: 17f<SNAPSHOT,CLUSTERS,WCACHE,DATA64,HOSTTIME,RAID50,WINDOW4GB,SOFTERR>
ld1 at aac1 unit 0: RAID 5
ld1: 95450 MB, 12168 cyl, 255 head, 63 sec, 512 bytes/sect x 195482496 sectors
pchb7 at pci0 dev 17 function 2
pchb7: ServerWorks CIOB-X2 PCI-X bridge (rev. 0x03)
pci5 at pchb7 bus 2
pci5: i/o space, memory space enabled
isa0 at pcib0
isapnp0 at isa0 port 0x279: ISA Plug 'n Play device support
isapnp0: no ISA Plug 'n Play devices found
ioapic0: enabling
ioapic1: enabling
ioapic2: enabling
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
Kernelized RAIDframe activated
atapibus0 at atabus0: 2 targets
cd0 at atapibus0 drive 0: <SAMSUNG CD-ROM SN-124, , N102> cdrom removable
cd0: 32-bit data port
cd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33)
cd0(rccide0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 2 (Ultra/33) (using DMA)
boot device: ld0
root on ld0a dumps on ld0b
root file system type: ffs
cpu2: CPU 1 running
cpu1: CPU 2 running
cpu3: CPU 3 running
wsdisplay0: screen 0 added (80x50, vt100 emulation)
wsdisplay0: screen 1 added (80x50, vt100 emulation)
wsdisplay0: screen 2 added (80x50, vt100 emulation)
wsdisplay0: screen 3 added (80x50, vt100 emulation)
wsdisplay0: screen 4 added (80x50, vt100 emulation)
wsdisplay0: screen 5 added (80x50, vt100 emulation)
wsdisplay0: screen 6 added (80x25, vt100 emulation)
wsdisplay0: screen 7 added (80x25, vt100 emulation)