device number issues

Dermot Paikkos <[email protected]> Thu, 25 Oct 2012 09:28:08 +0000
Newsgroups org.kernel.vger.linux-admin
Message-ID <D76685D16701D1409564F052D7A5D95102B996C2@CASTOR.sciencephoto.local>
Hi,

Kernel: 2.6.18-194.26.1.el5
Disk controller: HP MSA 1500

I have an irritating problem with two servers that are attached by fibre to our SAN. Any server that has a connection to the SAN is reporting errors with disk device files. It might be easier to show than explain. If I do pvscan I see this:

pvscan
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda: read failed after 0 of 4096 at 1500244410368: Input/output error
  /dev/sda: read failed after 0 of 4096 at 1500244492288: Input/output error
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda: read failed after 0 of 4096 at 4096: Input/output error
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 1500244410368: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 1500244492288: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 2000341368832: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 2000341483520: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
  /dev/sdc: read failed after 0 of 4096 at 0: Input/output error
  PV /dev/sdd1   VG images   lvm2 [1.36 TB / 0    free]
  PV /dev/sde1   VG images   lvm2 [1.36 TB / 0    free]
  PV /dev/sdf1   VG images   lvm2 [1.82 TB / 9.52 GB free]
  Total: 3 [4.55 TB] / in use: 3 [4.55 TB] / in no VG: 0 [0   ]

The number of errored devices;  sda,sdb,sdc, is equal to the number of ok one; sdd, sde and sdf. These volumes from the SAN are masked so that only this server can see them. I have the same problem with another server that uses 12 volumes from the same. I get a lot of noise from dmesg and to the console at boot time. 

Does anyone have any advices on how I might remedy this? 
Thanks,
Dermot