Re: SRP timeout
"M Lowe" <[email protected]> Wed, 21 Jul 2010 22:19:55 -0600
| Newsgroups | gmane.network.etherboot.user,gmane.network.etherboot.gpxe |
|---|---|
| Message-ID | <5DA43AA4A04C014F8833467372C303FC044351@spireserv.ads.spiretechit.com> |
This is a multi-part message in MIME format. --===============8414660675604209835== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CB2955.24535F56" This is a multi-part message in MIME format. ------_=_NextPart_001_01CB2955.24535F56 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Even ideas on how I can debug this issue further would help. I don't mind putting in the leg work at all, but a lot of this code is over my head.=20 =20 =20 From: Itay Gazit [mailto:[email protected]]=20 Sent: Tuesday, July 13, 2010 8:41 AM To: M Lowe; Michael Brown Cc: Stefan Hajnoczi; [email protected]; gpxe Subject: Re: [Etherboot-discuss] SRP timeout =20 Michael, Do you have an idea? What can be the problem with the arbel driver? =20 Itay On Mon, Jul 12, 2010 at 3:18 AM, M Lowe <[email protected]> wrote: I have been able to log the debug messages now however I see no errors that would indicate where the problem is. Just to recap quickly, the problem is that san-booting over InfiniBand using SRP doesn't work and just times out. The timeout occurs while waiting for a response to the SRP login request. I'm fairly certain the problem lies within gPXE because I can access the SRP target just fine through a local installation of Windows. In addition, on the SRP target side I have traced through the ib_srpt module and found that a login response is generated and sent (or at least posted to the mthca module work queue). On the gPXE side I've found that I'm not receiving the SRP_LOGIN_RSP packet even at the InfiniBand protocol level (net/infiniband.c). So far I have been able to determine the packet is lost at some point in the Arbel driver (drivers/infiniband/arbel.c) before arbel_complete().This would indicate the problem exists within the Arbel driver and explains why SRP sanboot worked with the Hermon driver. Despite compiling with DEBUG=3Darbel:3 I get no errors indicating there are any problems or dropped packets. Here is the output from autoboot with DEBUG=3Dsrp,ipoib,arp,infiniband,ib_cm,ib_cmrc,ib_mcast,ib_mi,ib_packet,i= b _pathrec,ib_sma,ib_smc,ib_srp Note: I have added some debug messages to help illustrate the flow of packets. At the beginning of ipoib_complete_recv, ib_complete_recv, and ib_mi_complete_recv I have added "RX" debug messages. Booting from root path "ib_srp::::fe800000000000000002c9020022e5e5::0002c9020022e5e4::0002c9020 022e5e4:0002c9020022e5e4" SRP 0xbb134 using ib_srp::::fe800000000000000002c9020022e5e5::0002c9020022e5e4::0002c90200 22e5e4:0002c9020022e5e4 SRP attached successfully IBDEV 0xb9a84 creating completion queue IBDEV 0xb9a84 created 8-entry completion queue 0xbb4c4 (0xbb214) with CQN 0x83 IBDEV 0xb9a84 creating queue pair IBDEV 0xb9a84 created queue pair 0xbb4f4 (0xbb5c4) with QPN 0x550403 IBDEV 0xb9a84 QPN 0x550403 has 4 send entries at [0xbb5a0,0xbb5b0) IBDEV 0xb9a84 QPN 0x550403 has 2 receive entries at [0xbb5b0,0xbb5b8) CMRC 0xbb1b4 using QPN 550403 SRP 0xbb134 TX login request tag 0000000000000001 CM 0xbbb64 created for IBDEV 0xb9a84 QPN 550403 CM 0xbbb64 connecting to fe800000:00000000:0002c902:0022e5e5 0002c902:0022e5e4 MI 0xba564 TX TID 6750584500000003 (03,02,01,0035) status 0000 infiniband RX MI 0xba564 RX MI 0xba564 RX TID 6750584500000003 (03,02,81,0035) status 0000 IBDEV 0xb9a84 path to fe800000:00000000:0002c902:0022e5e5 is 0007 sl 0 rate 6 MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000 MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000 MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000 MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000 infiniband RX IPoIB 0xb9ccc RX ARP cache add: IP 10.20.76.1 =3D> IPoIB 80000404:fe800000:00000000:0002c902:0022e5e5 ARP reply: IP 10.20.76.45 =3D> IPoIB 00550402:fe800000:00000000:0002c902:00243035 IPoIB peer 4 has MAC 80000404:fe800000:00000000:0002c902:0022e5e5 MI 0xba564 TX TID 6750584500000005 (03,02,01,0035) status 0000 infiniband RX MI 0xba564 RX MI 0xba564 RX TID 6750584500000005 (03,02,81,0035) status 0000 MI 0xba564 RX TID 6750584500000005 handling via transaction handler IBDEV 0xb9a84 path to fe800000:00000000:0002c902:0022e5e5 is 0007 sl 0 rate 6 infiniband RX IPoIB 0xb9ccc RX ARP cache update: IP 10.20.76.1 =3D> IPoIB 80000404:fe800000:00000000:0002c902:0022e5e5 ARP reply: IP 10.20.76.45 =3D> IPoIB 00550402:fe800000:00000000:0002c902:00243035 MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000 MI 0xba564 abandoning TID 6750584500000004 CM 0xbbb64 connection request failed: Connection timed out (0x4c206035) CMRC 0xbb1b4 disconnected: Connection timed out (0x4c206035) SRP 0xbb134 socket closed: Connection timed out (0x4c206035) From: Itay Gazit [mailto:[email protected]] Sent: Friday, June 25, 2010 11:47 AM To: Stefan Hajnoczi; M Lowe Cc: [email protected]; gpxe; Michael Brown Subject: Re: [Etherboot-discuss] SRP timeout Hi Matthew, Stefan is right, you should reduce the DEBUG messages depth to find the fail cause. I have tried SRP boot only with Hermon driver (ConnectX) and it worked for me. Regards, Itay =20 ------_=_NextPart_001_01CB2955.24535F56 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3DContent-Type content=3D"text/html; = charset=3Dus-ascii"> <meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)"> <style> <!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Tahoma; panose-1:2 11 6 4 3 5 4 4 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman","serif";} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} span.EmailStyle17 {mso-style-type:personal-reply; font-family:"Calibri","sans-serif"; color:#1F497D;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --> </style> <!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3DEN-US link=3Dblue vlink=3Dpurple> <div class=3DWordSection1> <p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif"; color:#1F497D'>Even ideas on how I can debug this issue further would = help. I don’t mind putting in the leg work at all, but a lot of this code = is over my head. <o:p></o:p></span></p> <p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif"; color:#1F497D'><o:p> </o:p></span></p> <p class=3DMsoNormal><span = style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif"; color:#1F497D'><o:p> </o:p></span></p> <div style=3D'border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt = 0in 0in 0in'> <p class=3DMsoNormal><b><span = style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span>= </b><span style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Itay Gazit [mailto:[email protected]] <br> <b>Sent:</b> Tuesday, July 13, 2010 8:41 AM<br> <b>To:</b> M Lowe; Michael Brown<br> <b>Cc:</b> Stefan Hajnoczi; [email protected]; = gpxe<br> <b>Subject:</b> Re: [Etherboot-discuss] SRP = timeout<o:p></o:p></span></p> </div> <p class=3DMsoNormal><o:p> </o:p></p> <div> <div> <p class=3DMsoNormal>Michael,<o:p></o:p></p> </div> <div> <p class=3DMsoNormal>Do you have an idea? What can be the problem with = the arbel driver?<o:p></o:p></p> </div> <div> <p class=3DMsoNormal> <o:p></o:p></p> </div> <div> <p class=3DMsoNormal style=3D'margin-bottom:12.0pt'>Itay<o:p></o:p></p> </div> <div> <p class=3DMsoNormal>On Mon, Jul 12, 2010 at 3:18 AM, M Lowe <<a href=3D"mailto:[email protected]">[email protected]</a>> = wrote:<o:p></o:p></p> <p class=3DMsoNormal>I have been able to log the debug messages now = however I see no errors<br> that would indicate where the problem is.<br> <br> Just to recap quickly, the problem is that san-booting over = InfiniBand<br> using SRP doesn't work and just times out. The timeout occurs while<br> waiting for a response to the SRP login request. I'm fairly certain = the<br> problem lies within gPXE because I can access the SRP target just = fine<br> through a local installation of Windows. In addition, on the SRP = target<br> side I have traced through the ib_srpt module and found that a login<br> response is generated and sent (or at least posted to the mthca = module<br> work queue).<br> <br> On the gPXE side I've found that I'm not receiving the SRP_LOGIN_RSP<br> packet even at the InfiniBand protocol level (net/infiniband.c). So = far<br> I have been able to determine the packet is lost at some point in = the<br> Arbel driver (drivers/infiniband/arbel.c) before = arbel_complete().This<br> would indicate the problem exists within the Arbel driver and = explains<br> why SRP sanboot worked with the Hermon driver. Despite compiling = with<br> DEBUG=3Darbel:3 I get no errors indicating there are any problems or<br> dropped packets.<br> <br> Here is the output from autoboot with<br> DEBUG=3Dsrp,ipoib,arp,infiniband,ib_cm,ib_cmrc,ib_mcast,ib_mi,ib_packet,i= b<br> _pathrec,ib_sma,ib_smc,ib_srp<br> <br> Note: I have added some debug messages to help illustrate the flow = of<br> packets. At the beginning of ipoib_complete_recv, ib_complete_recv, = and<br> ib_mi_complete_recv I have added "RX" debug messages.<br> <br> Booting from root path<br> "ib_srp::::fe800000000000000002c9020022e5e5::0002c9020022e5e4::0002c= 9020<br> 022e5e4:0002c9020022e5e4"<br> SRP 0xbb134 using<br> ib_srp::::fe800000000000000002c9020022e5e5::0002c9020022e5e4::0002c90200<= br> 22e5e4:0002c9020022e5e4<br> SRP attached successfully<br> IBDEV 0xb9a84 creating completion queue<br> IBDEV 0xb9a84 created 8-entry completion queue 0xbb4c4 (0xbb214) = with<br> CQN 0x83<br> IBDEV 0xb9a84 creating queue pair<br> IBDEV 0xb9a84 created queue pair 0xbb4f4 (0xbb5c4) with QPN 0x550403<br> IBDEV 0xb9a84 QPN 0x550403 has 4 send entries at [0xbb5a0,0xbb5b0)<br> IBDEV 0xb9a84 QPN 0x550403 has 2 receive entries at = [0xbb5b0,0xbb5b8)<br> CMRC 0xbb1b4 using QPN 550403<br> SRP 0xbb134 TX login request tag 0000000000000001<br> CM 0xbbb64 created for IBDEV 0xb9a84 QPN 550403<br> CM 0xbbb64 connecting to fe800000:00000000:0002c902:0022e5e5<br> 0002c902:0022e5e4<br> MI 0xba564 TX TID 6750584500000003 (03,02,01,0035) status 0000<br> infiniband RX<br> MI 0xba564 RX<br> MI 0xba564 RX TID 6750584500000003 (03,02,81,0035) status 0000<br> IBDEV 0xb9a84 path to fe800000:00000000:0002c902:0022e5e5 is 0007 sl = 0<br> rate 6<br> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000<br> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000<br> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000<br> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000<br> infiniband RX<br> IPoIB 0xb9ccc RX<br> ARP cache add: IP 10.20.76.1 =3D> IPoIB<br> 80000404:fe800000:00000000:0002c902:0022e5e5<br> ARP reply: IP 10.20.76.45 =3D> IPoIB<br> 00550402:fe800000:00000000:0002c902:00243035<br> IPoIB peer 4 has MAC 80000404:fe800000:00000000:0002c902:0022e5e5<br> MI 0xba564 TX TID 6750584500000005 (03,02,01,0035) status 0000<br> infiniband RX<br> MI 0xba564 RX<br> MI 0xba564 RX TID 6750584500000005 (03,02,81,0035) status 0000<br> MI 0xba564 RX TID 6750584500000005 handling via transaction handler<br> IBDEV 0xb9a84 path to fe800000:00000000:0002c902:0022e5e5 is 0007 sl = 0<br> rate 6<br> infiniband RX<br> IPoIB 0xb9ccc RX<br> ARP cache update: IP 10.20.76.1 =3D> IPoIB<br> 80000404:fe800000:00000000:0002c902:0022e5e5<br> ARP reply: IP 10.20.76.45 =3D> IPoIB<br> 00550402:fe800000:00000000:0002c902:00243035<br> MI 0xba564 TX TID 6750584500000004 (07,02,03,0010) status 0000<br> MI 0xba564 abandoning TID 6750584500000004<br> CM 0xbbb64 connection request failed: Connection timed out = (0x4c206035)<br> CMRC 0xbb1b4 disconnected: Connection timed out (0x4c206035)<br> SRP 0xbb134 socket closed: Connection timed out (0x4c206035)<br> <br> <br> <br> From: Itay Gazit [mailto:<a = href=3D"mailto:[email protected]">[email protected]</a>]<br> Sent: Friday, June 25, 2010 11:47 AM<br> To: Stefan Hajnoczi; M Lowe<br> Cc: <a = href=3D"mailto:[email protected]">etherboot-discuss= @lists.sourceforge.net</a>; gpxe; Michael Brown<br> Subject: Re: [Etherboot-discuss] SRP timeout<o:p></o:p></p> <div> <div> <p class=3DMsoNormal><br> Hi Matthew,<br> Stefan is right, you should reduce the DEBUG messages depth to find = the<br> fail cause.<br> I have tried SRP boot only with Hermon driver (ConnectX) and it = worked<br> for me.<br> Regards,<br> Itay<o:p></o:p></p> </div> </div> </div> <p class=3DMsoNormal><o:p> </o:p></p> </div> </div> </body> </html> ------_=_NextPart_001_01CB2955.24535F56-- --===============8414660675604209835== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first --===============8414660675604209835== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Etherboot-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/etherboot-discuss --===============8414660675604209835==--