Re: [Etherboot-discuss] SRP timeout

--===============0677286110==
Content-Type: multipart/alternative; boundary=001485f8d0f6f5b3be048c201295

--001485f8d0f6f5b3be048c201295
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Viswa,
Michael is trying to add the RC support for the Arbel.

Michael,
I suspect something earlier went wrong before posting the WQE. Can you
verify the all MAP_FA process suceed.

Regards,

Itay
On Fri, Jul 23, 2010 at 8:03 AM, Viswanath Krishnamurthy <
[email protected]> wrote:

> I just looked into arbel code. It does not implement RC qp type at all (I=
t
> implements UD only). This means SRP will not work with arbel card since S=
RP
> uses RC/RDMA for data transfer.
>
> Your best bet at this moment is to us Hermon card (unless you want to add
> RC QP capability into arbel driver :-) )
>
> Itay,
>
> Can you reconfirm this from the arbel code?
>
> Thx,
> Viswa
>
>
>   On Wed, Jul 21, 2010 at 9:19 PM, M Lowe <[email protected]> wrote:
>
>>    Even ideas on how I can debug this issue further would help. I don=92=
t
>> mind putting in the leg work at all, but a lot of this code is over my h=
ead.
>>
>>
>>
>>
>>
>>
>> *From:* Itay Gazit [mailto:[email protected]]
>> *Sent:* Tuesday, July 13, 2010 8:41 AM
>> *To:* M Lowe; Michael Brown
>> *Cc:* Stefan Hajnoczi; [email protected]; gpxe
>>
>> *Subject:* Re: [Etherboot-discuss] SRP timeout
>>
>>
>>
>> Michael,
>>
>> Do you have an idea? What can be the problem with the arbel driver?
>>
>>
>>
>> Itay
>>
>> On Mon, Jul 12, 2010 at 3:18 AM, M Lowe <[email protected]> wrote:
>>
>> I have been able to log the debug messages now however I see no errors
>> that would indicate where the problem is.
>>
>> Just to recap quickly, the problem is that san-booting over InfiniBand
>> using SRP doesn't work and just times out. The timeout occurs while
>> waiting for a response to the SRP login request. I'm fairly certain the
>> problem lies within gPXE because I can access the SRP target just fine
>> through a local installation of Windows. In addition, on the SRP target
>> side I have traced through the ib_srpt module and found that a login
>> response is generated and sent (or at least posted to the mthca module
>> work queue).
>>
>> On the gPXE side I've found that I'm not receiving the SRP_LOGIN_RSP
>> packet even at the InfiniBand protocol level (net/infiniband.c). So far
>> I have been able to determine the packet is lost at some point in the
>> Arbel driver (drivers/infiniband/arbel.c) before arbel_complete().This
>> would indicate the problem exists within the Arbel driver and explains
>> why SRP sanboot worked with the Hermon driver. Despite compiling with
>> DEBUG=3Darbel:3 I get no errors indicating there are any problems or
>> dropped packets.
>>
>> Here is the output from autoboot with
>> DEBUG=3Dsrp,ipoib,arp,infiniband,ib_cm,ib_cmrc,ib_mcast,ib_mi,ib_packet,=
ib
>> _pathrec,ib_sma,ib_smc,ib_srp
>>
>> Note: I have added some debug messages to help illustrate the flow of
>> packets. At the beginning of ipoib_complete_recv, ib_complete_recv, and
>> ib_mi_complete_recv I have added "RX" debug messages.
>>
>> Booting from root path
>> "ib_srp::::fe800000000000000002c9020022e5e5::0002c9020022e5e4::0002c9020
>> 022e5e4:0002c9020022e5e4"
>> SRP 0xbb134 using
>> ib_srp::::fe800000000000000002c9020022e5e5::0002c9020022e5e4::0002c90200
>> 22e5e4:0002c9020022e5e4
>> SRP attached successfully
>> IBDEV 0xb9a84 creating completion queue
>> IBDEV 0xb9a84 created 8-entry completion queue 0xbb4c4 (0xbb214) with
>> CQN 0x83
>> IBDEV 0xb9a84 creating queue pair
>> IBDEV 0xb9a84 created queue pair 0xbb4f4 (0xbb5c4) with QPN 0x550403
>> IBDEV 0xb9a84 QPN 0x550403 has 4 send entries at [0xbb5a0,0xbb5b0)
>> IBDEV 0xb9a84 QPN 0x550403 has 2 receive entries at [0xbb5b0,0xbb5b8)
>> CMRC 0xbb1b4 using QPN 550403
>> SRP 0xbb134 TX login request tag 0000000000000001
>> CM 0xbbb64 created for IBDEV 0xb9a84 QPN 550403
>> CM 0xbbb64 connecting to fe800000:00000000:0002c902:0022e5e5
>> 0002c902:0022e5e4
>> MI 0xba564 TX TID 6750584500000003 (03,02,01,0035) status 0000
>> infiniband RX
>> MI 0xba564 RX
>> MI 0xba564 RX TID 6750584500000003 (03,02,81,0035) status 0000
>> IBDEV 0xb9a84 path to fe800000:00000000:0002c902:0022e5e5 is 0007 sl 0
>> rate 6
>> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000
>> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000
>> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000
>> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000
>> infiniband RX
>> IPoIB 0xb9ccc RX
>> ARP cache add: IP 10.20.76.1 =3D> IPoIB
>> 80000404:fe800000:00000000:0002c902:0022e5e5
>> ARP reply: IP 10.20.76.45 =3D> IPoIB
>> 00550402:fe800000:00000000:0002c902:00243035
>> IPoIB peer 4 has MAC 80000404:fe800000:00000000:0002c902:0022e5e5
>> MI 0xba564 TX TID 6750584500000005 (03,02,01,0035) status 0000
>> infiniband RX
>> MI 0xba564 RX
>> MI 0xba564 RX TID 6750584500000005 (03,02,81,0035) status 0000
>> MI 0xba564 RX TID 6750584500000005 handling via transaction handler
>> IBDEV 0xb9a84 path to fe800000:00000000:0002c902:0022e5e5 is 0007 sl 0
>> rate 6
>> infiniband RX
>> IPoIB 0xb9ccc RX
>> ARP cache update: IP 10.20.76.1 =3D> IPoIB
>> 80000404:fe800000:00000000:0002c902:0022e5e5
>> ARP reply: IP 10.20.76.45 =3D> IPoIB
>> 00550402:fe800000:00000000:0002c902:00243035
>> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000
>> MI 0xba564 abandoning TID 6750584500000004
>> CM 0xbbb64 connection request failed: Connection timed out (0x4c206035)
>> CMRC 0xbb1b4 disconnected: Connection timed out (0x4c206035)
>> SRP 0xbb134 socket closed: Connection timed out (0x4c206035)
>>
>>
>>
>> From: Itay Gazit [mailto:[email protected]]
>> Sent: Friday, June 25, 2010 11:47 AM
>> To: Stefan Hajnoczi; M Lowe
>> Cc: [email protected]; gpxe; Michael Brown
>> Subject: Re: [Etherboot-discuss] SRP timeout
>>
>>
>> Hi Matthew,
>> Stefan is right, you should reduce the DEBUG messages depth to find the
>> fail cause.
>> I have tried SRP boot only with Hermon driver (ConnectX) and it worked
>> for me.
>> Regards,
>> Itay
>>
>>
>>
>>
>> ------------------------------------------------------------------------=
------
>> This SF.net email is sponsored by Sprint
>> What will you do first with EVO, the first 4G phone?
>> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
>>
>> _______________________________________________
>> Etherboot-discuss mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/etherboot-discuss
>>
>>
>

--001485f8d0f6f5b3be048c201295
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Viswa,</div>
<div>Michael is trying to add the RC support for the Arbel.</div>
<div>=A0</div>
<div>Michael,</div>
<div>I suspect something earlier went wrong before posting the WQE. Can you=
 verify the all MAP_FA process suceed.</div>
<div>=A0</div>
<div>Regards,</div>
<div>=A0</div>
<div>Itay<br></div>
<div class=3D"gmail_quote">On Fri, Jul 23, 2010 at 8:03 AM, Viswanath Krish=
namurthy <span dir=3D"ltr">&lt;<a href=3D"mailto:[email protected]">vis=
[email protected]</a>&gt;</span> wrote:<br>
<blockquote style=3D"BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex=
; PADDING-LEFT: 1ex" class=3D"gmail_quote">I just looked into arbel code. I=
t does not implement RC qp type at all (It implements UD only). This means =
SRP will not work with arbel card since SRP uses RC/RDMA for data transfer.=
<br>
<br>Your best bet at this moment is to us Hermon card (unless you want to a=
dd RC QP capability into arbel driver :-) )<br><br>Itay,<br><br>Can you rec=
onfirm this from the arbel code?<br><br>Thx,<br>Viswa<br><br><br>
<div class=3D"gmail_quote">
<div>
<div></div>
<div class=3D"h5">On Wed, Jul 21, 2010 at 9:19 PM, M Lowe <span dir=3D"ltr"=
>&lt;<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>&g=
t;</span> wrote:<br></div></div>
<blockquote style=3D"BORDER-LEFT: rgb(204,204,204) 1px solid; MARGIN: 0pt 0=
pt 0pt 0.8ex; PADDING-LEFT: 1ex" class=3D"gmail_quote">
<div>
<div></div>
<div class=3D"h5">
<div lang=3D"EN-US" vlink=3D"purple" link=3D"blue">
<div>
<p class=3D"MsoNormal"><span style=3D"COLOR: rgb(31,73,125); FONT-SIZE: 11p=
t">Even ideas on how I can debug this issue further would help. I don=92t m=
ind putting in the leg work at all, but a lot of this code is over my head.=
 </span></p>

<p class=3D"MsoNormal"><span style=3D"COLOR: rgb(31,73,125); FONT-SIZE: 11p=
t">=A0</span></p>
<p class=3D"MsoNormal"><span style=3D"COLOR: rgb(31,73,125); FONT-SIZE: 11p=
t">=A0</span></p>
<div style=3D"BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING=
-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: 1pt solid;=
 BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<p class=3D"MsoNormal"><b><span style=3D"FONT-SIZE: 10pt">From:</span></b><=
span style=3D"FONT-SIZE: 10pt"> Itay Gazit [mailto:<a href=3D"mailto:itayga=
[email protected]" target=3D"_blank">[email protected]</a>] <br><b>Sent:</b> =
Tuesday, July 13, 2010 8:41 AM<br>
<b>To:</b> M Lowe; Michael Brown<br><b>Cc:</b> Stefan Hajnoczi; <a href=3D"=
mailto:[email protected]" target=3D"_blank">etherboot=
[email protected]</a>; gpxe=20
<div>
<div></div>
<div><br><b>Subject:</b> Re: [Etherboot-discuss] SRP timeout</div></div></s=
pan>
<p></p></p></div>
<div>
<div></div>
<div>
<p class=3D"MsoNormal">=A0</p>
<div>
<div>
<p class=3D"MsoNormal">Michael,</p></div>
<div>
<p class=3D"MsoNormal">Do you have an idea? What can be the problem with th=
e arbel driver?</p></div>
<div>
<p class=3D"MsoNormal">=A0</p></div>
<div>
<p style=3D"MARGIN-BOTTOM: 12pt" class=3D"MsoNormal">Itay</p></div>
<div>
<p class=3D"MsoNormal">On Mon, Jul 12, 2010 at 3:18 AM, M Lowe &lt;<a href=
=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>&gt; wrote:</p=
>
<p class=3D"MsoNormal">I have been able to log the debug messages now howev=
er I see no errors<br>that would indicate where the problem is.<br><br>Just=
 to recap quickly, the problem is that san-booting over InfiniBand<br>using=
 SRP doesn&#39;t work and just times out. The timeout occurs while<br>
waiting for a response to the SRP login request. I&#39;m fairly certain the=
<br>problem lies within gPXE because I can access the SRP target just fine<=
br>through a local installation of Windows. In addition, on the SRP target<=
br>
side I have traced through the ib_srpt module and found that a login<br>res=
ponse is generated and sent (or at least posted to the mthca module<br>work=
 queue).<br><br>On the gPXE side I&#39;ve found that I&#39;m not receiving =
the SRP_LOGIN_RSP<br>
packet even at the InfiniBand protocol level (net/infiniband.c). So far<br>=
I have been able to determine the packet is lost at some point in the<br>Ar=
bel driver (drivers/infiniband/arbel.c) before arbel_complete().This<br>
would indicate the problem exists within the Arbel driver and explains<br>w=
hy SRP sanboot worked with the Hermon driver. Despite compiling with<br>DEB=
UG=3Darbel:3 I get no errors indicating there are any problems or<br>droppe=
d packets.<br>
<br>Here is the output from autoboot with<br>DEBUG=3Dsrp,ipoib,arp,infiniba=
nd,ib_cm,ib_cmrc,ib_mcast,ib_mi,ib_packet,ib<br>_pathrec,ib_sma,ib_smc,ib_s=
rp<br><br>Note: I have added some debug messages to help illustrate the flo=
w of<br>
packets. At the beginning of ipoib_complete_recv, ib_complete_recv, and<br>=
ib_mi_complete_recv I have added &quot;RX&quot; debug messages.<br><br>Boot=
ing from root path<br>&quot;ib_srp::::fe800000000000000002c9020022e5e5::000=
2c9020022e5e4::0002c9020<br>
022e5e4:0002c9020022e5e4&quot;<br>SRP 0xbb134 using<br>ib_srp::::fe80000000=
0000000002c9020022e5e5::0002c9020022e5e4::0002c90200<br>22e5e4:0002c9020022=
e5e4<br>SRP attached successfully<br>IBDEV 0xb9a84 creating completion queu=
e<br>
IBDEV 0xb9a84 created 8-entry completion queue 0xbb4c4 (0xbb214) with<br>CQ=
N 0x83<br>IBDEV 0xb9a84 creating queue pair<br>IBDEV 0xb9a84 created queue =
pair 0xbb4f4 (0xbb5c4) with QPN 0x550403<br>IBDEV 0xb9a84 QPN 0x550403 has =
4 send entries at [0xbb5a0,0xbb5b0)<br>
IBDEV 0xb9a84 QPN 0x550403 has 2 receive entries at [0xbb5b0,0xbb5b8)<br>CM=
RC 0xbb1b4 using QPN 550403<br>SRP 0xbb134 TX login request tag 00000000000=
00001<br>CM 0xbbb64 created for IBDEV 0xb9a84 QPN 550403<br>CM 0xbbb64 conn=
ecting to fe800000:00000000:0002c902:0022e5e5<br>
0002c902:0022e5e4<br>MI 0xba564 TX TID 6750584500000003 (03,02,01,0035) sta=
tus 0000<br>infiniband RX<br>MI 0xba564 RX<br>MI 0xba564 RX TID 67505845000=
00003 (03,02,81,0035) status 0000<br>IBDEV 0xb9a84 path to fe800000:0000000=
0:0002c902:0022e5e5 is 0007 sl 0<br>
rate 6<br>MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000<br=
>MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000<br>MI 0xba5=
64 TX TID 6750584500000004 (07,02,03,0010) status 0000<br>MI 0xba564 TX TID=
 6750584500000004 (07,02,03,0010) status 0000<br>
infiniband RX<br>IPoIB 0xb9ccc RX<br>ARP cache add: IP 10.20.76.1 =3D&gt; I=
PoIB<br>80000404:fe800000:00000000:0002c902:0022e5e5<br>ARP reply: IP 10.20=
.76.45 =3D&gt; IPoIB<br>00550402:fe800000:00000000:0002c902:00243035<br>IPo=
IB peer 4 has MAC 80000404:fe800000:00000000:0002c902:0022e5e5<br>
MI 0xba564 TX TID 6750584500000005 (03,02,01,0035) status 0000<br>infiniban=
d RX<br>MI 0xba564 RX<br>MI 0xba564 RX TID 6750584500000005 (03,02,81,0035)=
 status 0000<br>MI 0xba564 RX TID 6750584500000005 handling via transaction=
 handler<br>
IBDEV 0xb9a84 path to fe800000:00000000:0002c902:0022e5e5 is 0007 sl 0<br>r=
ate 6<br>infiniband RX<br>IPoIB 0xb9ccc RX<br>ARP cache update: IP 10.20.76=
.1 =3D&gt; IPoIB<br>80000404:fe800000:00000000:0002c902:0022e5e5<br>ARP rep=
ly: IP 10.20.76.45 =3D&gt; IPoIB<br>
00550402:fe800000:00000000:0002c902:00243035<br>MI 0xba564 TX TID 675058450=
0000004 (07,02,03,0010) status 0000<br>MI 0xba564 abandoning TID 6750584500=
000004<br>CM 0xbbb64 connection request failed: Connection timed out (0x4c2=
06035)<br>
CMRC 0xbb1b4 disconnected: Connection timed out (0x4c206035)<br>SRP 0xbb134=
 socket closed: Connection timed out (0x4c206035)<br><br><br><br>From: Itay=
 Gazit [mailto:<a href=3D"mailto:[email protected]" target=3D"_blank">ita=
[email protected]</a>]<br>
Sent: Friday, June 25, 2010 11:47 AM<br>To: Stefan Hajnoczi; M Lowe<br>Cc: =
<a href=3D"mailto:[email protected]" target=3D"_blank=
">[email protected]</a>; gpxe; Michael Brown<br>Subje=
ct: Re: [Etherboot-discuss] SRP timeout</p>

<div>
<div>
<p class=3D"MsoNormal"><br>Hi Matthew,<br>Stefan is right, you should reduc=
e the DEBUG messages depth to find the<br>fail cause.<br>I have tried SRP b=
oot only with Hermon driver (ConnectX) and it worked<br>for me.<br>Regards,=
<br>
Itay</p></div></div></div>
<p class=3D"MsoNormal">=A0</p></div></div></div></div></div><br></div></div=
>--------------------------------------------------------------------------=
----<br>This SF.net email is sponsored by Sprint<br>What will you do first =
with EVO, the first 4G phone?<br>
Visit <a href=3D"http://sprint.com/first" target=3D"_blank">sprint.com/firs=
t</a> -- <a href=3D"http://p.sf.net/sfu/sprint-com-first" target=3D"_blank"=
>http://p.sf.net/sfu/sprint-com-first</a>=20
<div class=3D"im"><br>_______________________________________________<br>Et=
herboot-discuss mailing list<br><a href=3D"mailto:[email protected]=
ourceforge.net" target=3D"_blank">[email protected]</=
a><br>
<a href=3D"https://lists.sourceforge.net/lists/listinfo/etherboot-discuss" =
target=3D"_blank">https://lists.sourceforge.net/lists/listinfo/etherboot-di=
scuss</a><br><br></div></blockquote></div><br></blockquote></div><br></div>

--001485f8d0f6f5b3be048c201295--

--===============0677286110==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
gPXE mailing list
[email protected]
http://etherboot.org/mailman/listinfo/gpxe

--===============0677286110==--