Re: SMP on IBM eseriesand amd64

Thank you to your answer, now I know that it has no sense to compile
some new kernels and spend my time :)

Our machine is used as firewall, so we really need to pin irqs to both
cpu and to utilise both CPUs. 

In this situation, we do not achieve even 100mbit throughput :( Do you
think it is normal? Is there any other optimalization ? Is ther
possibility to use first cpu for kernel and interrupts and second for
applications? Now only one cpu is utilized.. 

Thank you!
Lukas

On Po, 2006-06-12 at 09:04 +0200, Niklas Hallqvist wrote:
> Hi!
> 
> Unfortunately, you seem to have an application where OpenBSD is an 
> inferior choice.
> The design-goals behind SMP was to be able to utilize extra CPUs without 
> adding too
> much complexity (believe me, it is complex enough anyway).  This is why 
> we chose the
> so-called biglock approach, i.e. only one CPU can execute kernel code at 
> any given time.
> Interrupts are kernel code, thus only one interrupt can be served 
> concurrently.  Furthermore,
> at this point only the boot processor can execute interrupts (except 
> timer which are cpu-local
> and thus can occur on all CPUs), which means high-interrupt pressure 
> will clog one singe CPU.
> This is not optimal, but for many purposes it works quite OK.  If you 
> have several processes
> eating most of its CPU consumption in userland, this is indeed quite 
> good.  An example is
> compilation, one CPU will run the compiler, another the preprocessor, 
> both are expensive
> CPU consumers.  Web applications can also utilize extra CPUs with 
> concurrent requests where
> non-trivial time is spent in the application code (userland).
> 
> It is not good for mostly-kernel or single-process applications, like 
> routing, filtering or
> database servers (unless the database implementation is designed with 
> multiple worker processes,
> and the queries are non-trivial, i.e. complexity is in the query 
> optimization, not in just shoveling
> data to the clients).
> 
> We have always had intentions to improve this, but intentions is not 
> enough.  Capable people must
> have motivation to do the work.  What kind of motivation depends on 
> exactly who is doing the work.
> 
> What kind of application is this?  Pure routing/filtering?  I suspect so 
> since you don't get any action
> from the secondary CPU.
> 
> Sorry for the sad answer,
> Niklas
> 
> Lukas Macura wrote:
> > Hello all,
> >
> > please we have little problem with openbsd runing on $SUBJ.
> > I think there could be better irq routing. But I don't know how to
> > achieve this. In BIOS, I cannot change IRQ for anything. Every device in
> > this server is on same IRQ. I really don't know why but I cannot change
> > this. When I boot linux on this machine, IRQ routing is OK, probably
> > bacause linux know how to change IRQ of devices.
> >
> > I do not understand what is difference between int and irq in dmesg.
> > Sorry, I am not expert for this. But reality is, that on OpenBSD, only
> > one CPU is used. Probably because all interrupts are routed thru this
> > CPU. Second CPU is still idle. 
> >
> > Please can somebody help me what to do to utilise second CPU? We need
> > more bandwidth and we want to use one NIC/one IRQ. So it shuold bring
> > better CPU utilisation. Am I right?
> >
> > Thanks to all,
> > Lukas Macura
> >
> > Here is dmesg:
> > OpenBSD 3.8-current (GENERIC.MP) #6: Thu Nov  3 17:32:14 CET 2005
> >     [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > real mem = 1073319936 (1048164K)
> > avail mem = 908783616 (887484K)
> > using 22937 buffers containing 107540480 bytes (105020K) of memory
> > mainbus0 (root)
> > mainbus0: scanning 0x9d400 to 0x9d7f0 for MP signature
> > mainbus0: MP floating pointer found in extended bios data area at
> > 0x9d540
> > mainbus0: MP config table at 0x9e520, 356 bytes long
> > mainbus0: Intel MP Specification (Version 1.4) (IBM ENSW X336 SMP    )
> > cpu0 at mainbus0: apid 0 (boot processor)
> > cpu0: Intel(R) Xeon(TM) CPU 3.20GHz, 3200.71 MHz
> > cpu0:
> > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,NXE,LONG
> > cpu0: 2MB 64b/line 8-way L2 cache
> > cpu0: calibrating local timer
> > cpu0: apic clock running at 200006987Hz
> > cpu0: kstack at 0xffff80006585c000 for 20480 bytes
> > cpu0: idle pcb at 0xffff80006585c000, idle sp at 0xffff800065860ff0
> > cpu1 at mainbus0: apid 6 (application processor)
> > cpu1: Intel(R) Xeon(TM) CPU 3.20GHz, 3200.12 MHz
> > cpu1:
> > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,NXE,LONG
> > cpu1: 2MB 64b/line 8-way L2 cache
> > cpu1: kstack at 0xffff800065861000 for 20480 bytes
> > cpu1: idle pcb at 0xffff800065861000, idle sp at 0xffff800065865ff0
> > mpbios: bus 0 is type PCI
> > mpbios: bus 1 is type PCI
> > mpbios: bus 2 is type PCI
> > mpbios: bus 3 is type PCI
> > mpbios: bus 4 is type PCI
> > mpbios: bus 5 is type PCI
> > mpbios: bus 6 is type PCI
> > mpbios: bus 7 is type PCI
> > mpbios: bus 8 is type ISA
> > ioapic0 at mainbus0 apid 14: pa 0xffff800001ba7f24, virtual wire mode,
> > version 20, 24 pins
> > ioapic1 at mainbus0 apid 13: pa 0xffff800001ba7e24, virtual wire mode,
> > version 20, 24 pins
> > ioapic2 at mainbus0 apid 12: pa 0xffff800001ba7d24, virtual wire mode,
> > version 20, 24 pins
> > ioapic0: int1 attached to isa0 irq 1 (type 0x0 flags 0x0)
> > ioapic0: int2 attached to isa0 irq 0 (type 0x0 flags 0x0)
> > ioapic0: int6 attached to isa0 irq 6 (type 0x0 flags 0x0)
> > ioapic0: int8 attached to isa0 irq 8 (type 0x0 flags 0x5)
> > ioapic0: int9 attached to isa0 irq 9 (type 0x0 flags 0x0)
> > ioapic0: int12 attached to isa0 irq 12 (type 0x0 flags 0x0)
> > ioapic0: int13 attached to isa0 irq 13 (type 0x0 flags 0x0)
> > ioapic0: int14 attached to isa0 irq 14 (type 0x0 flags 0x0)
> > ioapic0: int15 attached to isa0 irq 15 (type 0x0 flags 0x0)
> > mpbios: can't find ioapic 0
> > ioapic0: int16 attached to pci0 device 29 INT_A (type 0x0 flags 0x0)
> > ioapic0: int19 attached to pci0 device 29 INT_B (type 0x0 flags 0x0)
> > ioapic0: int23 attached to pci0 device 29 INT_D (type 0x0 flags 0x0)
> > ioapic0: int17 attached to pci0 device 31 INT_B (type 0x0 flags 0x0)
> > ioapic0: int16 attached to pci1 device 1 INT_A (type 0x0 flags 0x0)
> > mpbios: can't find ioapic 0
> > ioapic1: int4 attached to pci4 device 1 INT_A (type 0x0 flags 0x0)
> > ioapic2: int0 attached to pci5 device 1 INT_A (type 0x0 flags 0x0)
> > ioapic0: int16 attached to pci6 device 0 INT_A (type 0x0 flags 0x0)
> > ioapic0: int16 attached to pci7 device 0 INT_A (type 0x0 flags 0x0)
> > local apic: int1 attached to NMI (type 0x1 flags 0x0)
> > local apic: int0 attached to ExtINT (type 0x3 flags 0x0)
> > mainbus0: MP WARNING: 348 bytes of extended entries not examined
> > pci0 at mainbus0 bus 0: configuration mode 1
> > pchb0 at pci0 dev 0 function 0 "Intel E7710 SMCH" rev 0x0c
> > "Intel E7710 MCH ERR" rev 0x0c at pci0 dev 0 function 1 not configured
> > ppb0 at pci0 dev 2 function 0 "Intel E7710 MCH PCIE" rev 0x0c
> > pci1 at ppb0 bus 2
> > ppb1 at pci0 dev 4 function 0 "Intel E7710 MCH PCIE" rev 0x0c
> > pci2 at ppb1 bus 3
> > ppb2 at pci2 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
> > pci3 at ppb2 bus 4
> > mpt0 at pci3 dev 1 function 0 "Symbios Logic 53c1030" rev 0x08: apic 13
> > int 4 (irq 10)
> > mpt0: sending FW Upload request to IOC (size: 36, img size: 69956)
> > mpt0: IM support: 4
> > scsibus0 at mpt0: 16 targets
> > sd0 at scsibus0 targ 0 lun 0: <LSILOGIC, 1030 IM IM, 1000> SCSI2
> > 0/direct fixed
> > sd0: 139898MB, 139898 cyl, 16 head, 128 sec, 512 bytes/sec, 286511104
> > sec total
> > mpt0: target 0 Asynchronous at 0MHz width 8bit offset 0 QAS 0 DT 0 IU 0
> > ppb3 at pci2 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
> > pci4 at ppb3 bus 5
> > em0 at pci4 dev 1 function 0 "Intel PRO/1000MT (82545GM)" rev 0x04: apic
> > 12 int 0 (irq 10), address 00:0e:0c:9c:07:13
> > ppb4 at pci0 dev 6 function 0 "Intel E7710 MCH PCIE" rev 0x0c
> > pci5 at ppb4 bus 6
> > bge0 at pci5 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
> > (0x4101): apic 14 int 16 (irq 10) address 00:14:5e:0b:3e:ea
> > brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
> > ppb5 at pci0 dev 7 function 0 "Intel E7710 MCH PCIE" rev 0x0c
> > pci6 at ppb5 bus 7
> > bge1 at pci6 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
> > (0x4101): apic 14 int 16 (irq 10) address 00:14:5e:0b:3e:eb
> > brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
> > vendor "Intel", unknown product 0x359b (class system subclass
> > miscellaneous, rev 0x0c) at pci0 dev 8 function 0 not configured
> > uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: apic 14
> > int 16 (irq 10)
> > usb0 at uhci0: USB revision 1.0
> > uhub0 at usb0
> > uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1
> > uhub0: 2 ports with 2 removable, self powered
> > uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: apic 14
> > int 19 (irq 7)
> > usb1 at uhci1: USB revision 1.0
> > uhub1 at usb1
> > uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1
> > uhub1: 2 ports with 2 removable, self powered
> > ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB" rev 0x02: apic 14
> > int 23 (irq 5)
> > usb2 at ehci0: USB revision 2.0
> > uhub2 at usb2
> > uhub2: Intel EHCI root hub, rev 2.00/1.00, addr 1
> > uhub2: 4 ports with 4 removable, self powered
> > ppb6 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xc2
> > pci7 at ppb6 bus 1
> > vga1 at pci7 dev 1 function 0 "ATI Radeon VE QY" rev 0x00
> > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> > wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> > pcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
> > pciide0 at pci0 dev 31 function 2 "Intel 82801EB SATA" rev 0x02: DMA,
> > channel 0 configured to compatibility, channel 1 configured to
> > compatibility
> > atapiscsi0 at pciide0 channel 0 drive 0
> > scsibus1 at atapiscsi0: 2 targets
> > cd0 at scsibus1 targ 0 lun 0: <HL-DT-ST, DVD-ROM GDR8083N, 0L02> SCSI0
> > 5/cdrom removable
> > cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
> > "Intel 82801EB/ER SMBus" rev 0x02 at pci0 dev 31 function 3 not
> > configured
> > isa0 at pcib0
> > isadma0 at isa0
> > pckbc0 at isa0 port 0x60/5
> > pckbd0 at pckbc0 (kbd slot)
> > pckbc0: using irq 1 for kbd slot
> > wskbd0 at pckbd0: console keyboard, using wsdisplay0
> > pmsi0 at pckbc0 (aux slot)
> > pckbc0: using irq 12 for aux slot
> > wsmouse0 at pmsi0 mux 0
> > pcppi0 at isa0 port 0x61
> > spkr0 at pcppi0
> > sysbeep0 at pcppi0
> > cpu0: prelint0 0x700 0x0
> > cpu0: prelint1 0x400 0x0
> > cpu0: timer0 0x300c0 0x0
> > cpu0: pcint0 0x10000 0x0
> > cpu0: lint0 0x10700 0x0
> > cpu0: lint1 0x400 0x0
> > cpu0: err0 0x10000 0x0
> > ioapic2: int0 0xa070 0x0
> > ioapic1: int4 0xa060 0x0
> > ioapic0: int16 0xa061 0x0
> > ioapic0: int19 0xa062 0x0
> > ioapic0: int23 0xa063 0x0
> > dkcsum: sd0 matches BIOS drive 0x80
> > root on sd0a
> > rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
> > cpu1: prelint0 0x10000 0x0
> > cpu1: prelint1 0x10000 0x0
> > cpu1: timer0 0x200c0 0x0
> > cpu1: pcint0 0x10000 0x0
> > cpu1: lint0 0x10700 0x0
> > cpu1: lint1 0x400 0x0
> > cpu1: err0 0x10000 0x0