Re: SRP timeout

--===============7720495619535297709==
Content-Type: multipart/alternative; boundary=001485eefd78fa715c048b45dacc

--001485eefd78fa715c048b45dacc
Content-Type: text/plain; charset=ISO-8859-1

Michael,
Do you have an idea? What can be the problem with the arbel driver?

Itay

On Mon, Jul 12, 2010 at 3:18 AM, M Lowe <[email protected]> wrote:

> I have been able to log the debug messages now however I see no errors
> that would indicate where the problem is.
>
> Just to recap quickly, the problem is that san-booting over InfiniBand
> using SRP doesn't work and just times out. The timeout occurs while
> waiting for a response to the SRP login request. I'm fairly certain the
> problem lies within gPXE because I can access the SRP target just fine
> through a local installation of Windows. In addition, on the SRP target
> side I have traced through the ib_srpt module and found that a login
> response is generated and sent (or at least posted to the mthca module
> work queue).
>
> On the gPXE side I've found that I'm not receiving the SRP_LOGIN_RSP
> packet even at the InfiniBand protocol level (net/infiniband.c). So far
> I have been able to determine the packet is lost at some point in the
> Arbel driver (drivers/infiniband/arbel.c) before arbel_complete().This
> would indicate the problem exists within the Arbel driver and explains
> why SRP sanboot worked with the Hermon driver. Despite compiling with
> DEBUG=arbel:3 I get no errors indicating there are any problems or
> dropped packets.
>
> Here is the output from autoboot with
> DEBUG=srp,ipoib,arp,infiniband,ib_cm,ib_cmrc,ib_mcast,ib_mi,ib_packet,ib
> _pathrec,ib_sma,ib_smc,ib_srp
>
> Note: I have added some debug messages to help illustrate the flow of
> packets. At the beginning of ipoib_complete_recv, ib_complete_recv, and
> ib_mi_complete_recv I have added "RX" debug messages.
>
> Booting from root path
> "ib_srp::::fe800000000000000002c9020022e5e5::0002c9020022e5e4::0002c9020
> 022e5e4:0002c9020022e5e4"
> SRP 0xbb134 using
> ib_srp::::fe800000000000000002c9020022e5e5::0002c9020022e5e4::0002c90200
> 22e5e4:0002c9020022e5e4
> SRP attached successfully
> IBDEV 0xb9a84 creating completion queue
> IBDEV 0xb9a84 created 8-entry completion queue 0xbb4c4 (0xbb214) with
> CQN 0x83
> IBDEV 0xb9a84 creating queue pair
> IBDEV 0xb9a84 created queue pair 0xbb4f4 (0xbb5c4) with QPN 0x550403
> IBDEV 0xb9a84 QPN 0x550403 has 4 send entries at [0xbb5a0,0xbb5b0)
> IBDEV 0xb9a84 QPN 0x550403 has 2 receive entries at [0xbb5b0,0xbb5b8)
> CMRC 0xbb1b4 using QPN 550403
> SRP 0xbb134 TX login request tag 0000000000000001
> CM 0xbbb64 created for IBDEV 0xb9a84 QPN 550403
> CM 0xbbb64 connecting to fe800000:00000000:0002c902:0022e5e5
> 0002c902:0022e5e4
> MI 0xba564 TX TID 6750584500000003 (03,02,01,0035) status 0000
> infiniband RX
> MI 0xba564 RX
> MI 0xba564 RX TID 6750584500000003 (03,02,81,0035) status 0000
> IBDEV 0xb9a84 path to fe800000:00000000:0002c902:0022e5e5 is 0007 sl 0
> rate 6
> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000
> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000
> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000
> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000
> infiniband RX
> IPoIB 0xb9ccc RX
> ARP cache add: IP 10.20.76.1 => IPoIB
> 80000404:fe800000:00000000:0002c902:0022e5e5
> ARP reply: IP 10.20.76.45 => IPoIB
> 00550402:fe800000:00000000:0002c902:00243035
> IPoIB peer 4 has MAC 80000404:fe800000:00000000:0002c902:0022e5e5
> MI 0xba564 TX TID 6750584500000005 (03,02,01,0035) status 0000
> infiniband RX
> MI 0xba564 RX
> MI 0xba564 RX TID 6750584500000005 (03,02,81,0035) status 0000
> MI 0xba564 RX TID 6750584500000005 handling via transaction handler
> IBDEV 0xb9a84 path to fe800000:00000000:0002c902:0022e5e5 is 0007 sl 0
> rate 6
> infiniband RX
> IPoIB 0xb9ccc RX
> ARP cache update: IP 10.20.76.1 => IPoIB
> 80000404:fe800000:00000000:0002c902:0022e5e5
> ARP reply: IP 10.20.76.45 => IPoIB
> 00550402:fe800000:00000000:0002c902:00243035
> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000
> MI 0xba564 abandoning TID 6750584500000004
> CM 0xbbb64 connection request failed: Connection timed out (0x4c206035)
> CMRC 0xbb1b4 disconnected: Connection timed out (0x4c206035)
> SRP 0xbb134 socket closed: Connection timed out (0x4c206035)
>
>
>
> From: Itay Gazit [mailto:[email protected]]
> Sent: Friday, June 25, 2010 11:47 AM
> To: Stefan Hajnoczi; M Lowe
> Cc: [email protected]; gpxe; Michael Brown
> Subject: Re: [Etherboot-discuss] SRP timeout
>
> Hi Matthew,
> Stefan is right, you should reduce the DEBUG messages depth to find the
> fail cause.
> I have tried SRP boot only with Hermon driver (ConnectX) and it worked
> for me.
> Regards,
> Itay
>

--001485eefd78fa715c048b45dacc
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Michael,</div>
<div>Do you have an idea? What can be the problem with the arbel driver?</d=
iv>
<div>=A0</div>
<div>Itay<br><br></div>
<div class=3D"gmail_quote">On Mon, Jul 12, 2010 at 3:18 AM, M Lowe <span di=
r=3D"ltr">&lt;<a href=3D"mailto:[email protected]">[email protected]</a>&gt;</span>=
 wrote:<br>
<blockquote style=3D"BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex=
; PADDING-LEFT: 1ex" class=3D"gmail_quote">I have been able to log the debu=
g messages now however I see no errors<br>that would indicate where the pro=
blem is.<br>
<br>Just to recap quickly, the problem is that san-booting over InfiniBand<=
br>using SRP doesn&#39;t work and just times out. The timeout occurs while<=
br>waiting for a response to the SRP login request. I&#39;m fairly certain =
the<br>
problem lies within gPXE because I can access the SRP target just fine<br>t=
hrough a local installation of Windows. In addition, on the SRP target<br>s=
ide I have traced through the ib_srpt module and found that a login<br>
response is generated and sent (or at least posted to the mthca module<br>w=
ork queue).<br><br>On the gPXE side I&#39;ve found that I&#39;m not receivi=
ng the SRP_LOGIN_RSP<br>packet even at the InfiniBand protocol level (net/i=
nfiniband.c). So far<br>
I have been able to determine the packet is lost at some point in the<br>Ar=
bel driver (drivers/infiniband/arbel.c) before arbel_complete().This<br>wou=
ld indicate the problem exists within the Arbel driver and explains<br>
why SRP sanboot worked with the Hermon driver. Despite compiling with<br>DE=
BUG=3Darbel:3 I get no errors indicating there are any problems or<br>dropp=
ed packets.<br><br>Here is the output from autoboot with<br>DEBUG=3Dsrp,ipo=
ib,arp,infiniband,ib_cm,ib_cmrc,ib_mcast,ib_mi,ib_packet,ib<br>
_pathrec,ib_sma,ib_smc,ib_srp<br><br>Note: I have added some debug messages=
 to help illustrate the flow of<br>packets. At the beginning of ipoib_compl=
ete_recv, ib_complete_recv, and<br>ib_mi_complete_recv I have added &quot;R=
X&quot; debug messages.<br>
<br>Booting from root path<br>&quot;ib_srp::::fe800000000000000002c9020022e=
5e5::0002c9020022e5e4::0002c9020<br>022e5e4:0002c9020022e5e4&quot;<br>SRP 0=
xbb134 using<br>ib_srp::::fe800000000000000002c9020022e5e5::0002c9020022e5e=
4::0002c90200<br>
22e5e4:0002c9020022e5e4<br>SRP attached successfully<br>IBDEV 0xb9a84 creat=
ing completion queue<br>IBDEV 0xb9a84 created 8-entry completion queue 0xbb=
4c4 (0xbb214) with<br>CQN 0x83<br>IBDEV 0xb9a84 creating queue pair<br>
IBDEV 0xb9a84 created queue pair 0xbb4f4 (0xbb5c4) with QPN 0x550403<br>IBD=
EV 0xb9a84 QPN 0x550403 has 4 send entries at [0xbb5a0,0xbb5b0)<br>IBDEV 0x=
b9a84 QPN 0x550403 has 2 receive entries at [0xbb5b0,0xbb5b8)<br>CMRC 0xbb1=
b4 using QPN 550403<br>
SRP 0xbb134 TX login request tag 0000000000000001<br>CM 0xbbb64 created for=
 IBDEV 0xb9a84 QPN 550403<br>CM 0xbbb64 connecting to fe800000:00000000:000=
2c902:0022e5e5<br>0002c902:0022e5e4<br>MI 0xba564 TX TID 6750584500000003 (=
03,02,01,0035) status 0000<br>
infiniband RX<br>MI 0xba564 RX<br>MI 0xba564 RX TID 6750584500000003 (03,02=
,81,0035) status 0000<br>IBDEV 0xb9a84 path to fe800000:00000000:0002c902:0=
022e5e5 is 0007 sl 0<br>rate 6<br>MI 0xba564 TX TID 6750584500000004 (07,02=
,03,0010) status 0000<br>
MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000<br>MI 0xba56=
4 TX TID 6750584500000004 (07,02,03,0010) status 0000<br>MI 0xba564 TX TID =
6750584500000004 (07,02,03,0010) status 0000<br>infiniband RX<br>IPoIB 0xb9=
ccc RX<br>
ARP cache add: IP 10.20.76.1 =3D&gt; IPoIB<br>80000404:fe800000:00000000:00=
02c902:0022e5e5<br>ARP reply: IP 10.20.76.45 =3D&gt; IPoIB<br>00550402:fe80=
0000:00000000:0002c902:00243035<br>IPoIB peer 4 has MAC 80000404:fe800000:0=
0000000:0002c902:0022e5e5<br>
MI 0xba564 TX TID 6750584500000005 (03,02,01,0035) status 0000<br>infiniban=
d RX<br>MI 0xba564 RX<br>MI 0xba564 RX TID 6750584500000005 (03,02,81,0035)=
 status 0000<br>MI 0xba564 RX TID 6750584500000005 handling via transaction=
 handler<br>
IBDEV 0xb9a84 path to fe800000:00000000:0002c902:0022e5e5 is 0007 sl 0<br>r=
ate 6<br>infiniband RX<br>IPoIB 0xb9ccc RX<br>ARP cache update: IP 10.20.76=
.1 =3D&gt; IPoIB<br>80000404:fe800000:00000000:0002c902:0022e5e5<br>ARP rep=
ly: IP 10.20.76.45 =3D&gt; IPoIB<br>
00550402:fe800000:00000000:0002c902:00243035<br>MI 0xba564 TX TID 675058450=
0000004 (07,02,03,0010) status 0000<br>MI 0xba564 abandoning TID 6750584500=
000004<br>CM 0xbbb64 connection request failed: Connection timed out (0x4c2=
06035)<br>
CMRC 0xbb1b4 disconnected: Connection timed out (0x4c206035)<br>SRP 0xbb134=
 socket closed: Connection timed out (0x4c206035)<br><br><br><br>From: Itay=
 Gazit [mailto:<a href=3D"mailto:[email protected]">[email protected]</=
a>]<br>
Sent: Friday, June 25, 2010 11:47 AM<br>To: Stefan Hajnoczi; M Lowe<br>Cc: =
<a href=3D"mailto:[email protected]">etherboot-discus=
[email protected]</a>; gpxe; Michael Brown<br>Subject: Re: [Etherboot=
-discuss] SRP timeout<br>

<div>
<div></div>
<div class=3D"h5"><br>Hi Matthew,<br>Stefan is right, you should reduce the=
 DEBUG messages depth to find the<br>fail cause.<br>I have tried SRP boot o=
nly with Hermon driver (ConnectX) and it worked<br>for me.<br>Regards,<br>
Itay<br></div></div></blockquote></div><br></div>

--001485eefd78fa715c048b45dacc--


--===============7720495619535297709==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
--===============7720495619535297709==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Etherboot-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/etherboot-discuss

--===============7720495619535297709==--