Re: Protocol Question
Thomas Heller <[email protected]> Sat, 16 Aug 2014 12:01:13 +0200
| Newsgroups | gmane.comp.db.postgresql.interfaces |
|---|---|
| Message-ID | <CAGTxmOt3TO6WsUtB0+jt20O_FzvOPw47q2bn=tO3oRYq9M09XA@mail.gmail.com> |
--001a11c26902c95e020500bc378b Content-Type: text/plain; charset=UTF-8 I just encountered a weird protocol error while stress testing my client. I'm receiving unexpected 'n'/NoData messages while reading 'D'/DataRow messages. Basically I connect, prepare a query, execute, close, terminate, repeat The command flow is -> P/D/S <- 1/t/T/Z -> B/E/S <- 2/D+/C/Z -> C/S <- 3/Z -> X, close socket What's weird is that I normally expect a 2/D+/C/Z as the result of the execute, which it is most of the time. But sometimes I receive interleaved 'n'/NoData. According to the protocol documentation NoData is only received after a Descibe to describe a query that returns no results. If I just ignore the message and continue reading the query completes normally. Can someone explain why I get a 'n'/NoData here? Not a problem really, just curious. Thanks, /thomas On Thu, Aug 14, 2014 at 10:00 AM, Thomas Heller <[email protected]> wrote: > Yeah, I went with that and let the server send me everything. If more > control is required, a cursor can be used. > > Thanks for the feedback. > > Onto implementing the Types. I assume the only place the binary encoding > is documented is the source? Can't find anything anywhere else. > > Regards, > /thomas > > > > On Thu, Aug 14, 2014 at 6:44 AM, Alistair Bayley <[email protected]> > wrote: > >> Why wouldn't TCP be sufficient to throttle the data rate from the server? >> If the client stops/slows consuming bytes from its socket, TCP will make >> the server slow its sending. >> >> >> On 13 August 2014 12:09, Thomas Heller <[email protected]> wrote: >> >>> I tried sending P/D/S...B/E/S...E/S... but the second Execute fails >>> because the portal no longer exists. >>> >>> So I need a BEGIN/COMMIT for a SELECT in order to read in multiple steps? >>> >>> I can process the data on-the-fly, my motivation for using Execute >>> with a limit was to introduce some kind of control on how fast things >>> travel over the wire. I'm not sure how the backend handles large results >>> sets and clients that can't keep up. Say I have a SELECT that returns 100k >>> rows, after 60k rows the clients into some sort of resource limit. Since >>> the backend just dumps everything on the wire I now have to continue >>> reading and discarding the remaining rows? I doubt I can Close the >>> statement early since the buffers on the backend should be full with more >>> rows to send to the frontend. I don't expect into these kinds of situations >>> very often, just want the client to "do the right thing". If I run into >>> some sort of hard limit I can just terminate the connection since I >>> probably have other things to worry about than reusing a connection. >>> >>> >>> Anyways, I might be overthinking things. The "standard" flow of >>> P/D/S.../B/E/S is probably the way to go then. >>> >>> Regards, >>> /thomas >>> >>> PS: The whole thing is available at >>> https://github.com/thheller/shadow-pgsql but be warned its very alpha. >>> >>> >>> >>> >>> >>> On Wed, Aug 13, 2014 at 12:08 AM, Tom Lane <[email protected]> wrote: >>> >>>> Thomas Heller <[email protected]> writes: >>>> > In an Extended Query Lifecycle, in order to prepare a query I send the >>>> > Commands >>>> >>>> > Parse('P') / Describe('D') / Sync('S') >>>> >>>> > read 1/t/T/Z then to execute >>>> >>>> > Bind('B') / Execute('E') / Flush('H') >>>> >>>> This is not a good idea. You *need* to use Sync to terminate a command >>>> sequence in order to be sure of proper error recovery (because if >>>> there's >>>> an error during the Execute, the backend will discard subsequent >>>> messages >>>> until it sees Sync). >>>> >>>> > If I skip the Flush after Execute I receive no data, if I Execute and >>>> Sync >>>> > I receive the the Limit of rows and a ReadyForQuery('Z'). >>>> >>>> That's probably because you're not wrapping this in a transaction so the >>>> Sync implicitly does a commit, discarding the open portal. If you want >>>> to read from a portal in multiple steps then you should issue a BEGIN >>>> first and a COMMIT (or ROLLBACK) after you're done. However, have you >>>> considered just processing the data on-the-fly instead of using a limit? >>>> >>>> regards, tom lane >>>> >>> >>> >> > --001a11c26902c95e020500bc378b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">I just encountered a weird protocol error while stress tes= ting my client.<div><br></div><div>I'm receiving unexpected 'n'= /NoData messages while reading 'D'/DataRow messages.<br><div><br></= div> <div>Basically I connect, prepare a query, execute, close, terminate, repea= t</div><div><br></div><div>The command flow is</div><div><br></div><div>-&g= t; P/D/S</div><div><- 1/t/T/Z</div><div>-> B/E/S</div><div><- 2/D+= /C/Z</div> <div>-> C/S</div><div><- 3/Z</div><div>-> X, close socket</div><di= v><br></div><div>What's weird is that I normally expect a 2/D+/C/Z as t= he result of the execute, which it is most of the time. But sometimes I rec= eive interleaved 'n'/NoData. According to the protocol documentatio= n NoData is only received after a Descibe to describe a query that returns = no results. If I just ignore the message and continue reading the query com= pletes normally.</div> <div><br></div><div>Can someone explain why I get a 'n'/NoData here= ? Not a problem really, just curious.</div><div><br></div><div>Thanks,</div= ><div>/thomas</div><div><br></div></div></div><div class=3D"gmail_extra"> <br><br><div class=3D"gmail_quote">On Thu, Aug 14, 2014 at 10:00 AM, Thomas= Heller <span dir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D= "_blank">[email protected]</a>></span> wrote:<br><blockquote class=3D"gma= il_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef= t:1ex"> <div dir=3D"ltr">Yeah, I went with that and let the server send me everythi= ng. If more control is required, a cursor can be used.<div><br><div>Thanks = for the feedback.</div></div><div><br></div><div>Onto implementing the Type= s. I assume the only place the binary encoding is documented is the source?= Can't find anything anywhere else.</div> <div><br></div><div>Regards,</div><div>/thomas</div><div><br></div></div><d= iv class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"><br><br><d= iv class=3D"gmail_quote">On Thu, Aug 14, 2014 at 6:44 AM, Alistair Bayley <= span dir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Why wouldn't TCP be suf= ficient to throttle the data rate from the server? If the client stops/slow= s consuming bytes from its socket, TCP will make the server slow its sendin= g.</div> <div><div><div class=3D"gmail_extra"> <br><br><div class=3D"gmail_quote">On 13 August 2014 12:09, Thomas Heller <= span dir=3D"ltr"><<a href=3D"mailto:[email protected]" target=3D"_blank">= [email protected]</a>></span> wrote:<br><blockquote class=3D"gmail_quote"= style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr">I tried sending P/D/S...B/E/S...E/S... but the second Exec= ute fails because the portal no longer exists.<div><br></div><div>So I need= a BEGIN/COMMIT for a SELECT in order to read in multiple steps?<div><br> </div> <div>I can process the data on-the-fly, my motivation for using Execute wit= h a limit was to introduce some kind of control on how fast things travel o= ver the wire. I'm not sure how the backend handles large results sets a= nd clients that can't keep up. Say I have a SELECT that returns 100k ro= ws, after 60k rows the clients into some sort of resource limit. Since the = backend just dumps everything on the wire I now have to continue reading an= d discarding the remaining rows?=C2=A0I doubt I can Close the statement ear= ly since the buffers on the backend should be full with more rows to send t= o the frontend. I don't expect into these kinds of situations very ofte= n, just want the client to "do the right thing". If I run into so= me sort of hard limit I can just terminate the connection since I probably = have other things to worry about than reusing a connection.</div> <div><br></div><div><br></div><div>Anyways, I might be overthinking things.= The "standard" flow of P/D/S.../B/E/S is probably the way to go = then.</div><div><br></div><div>Regards,</div><div>/thomas</div><div><br> </div><div>PS: The whole thing is available at=C2=A0<a href=3D"https://gith= ub.com/thheller/shadow-pgsql" target=3D"_blank">https://github.com/thheller= /shadow-pgsql</a> but be warned its very alpha.</div><div><br></div><div><b= r></div> <div><br></div> </div></div><div><div><div class=3D"gmail_extra"><br><br><div class=3D"gmai= l_quote">On Wed, Aug 13, 2014 at 12:08 AM, Tom Lane <span dir=3D"ltr"><<= a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>= ></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div>Thomas Heller <<a href=3D"mailto:inf= [email protected]" target=3D"_blank">[email protected]</a>> writes:<br> > In an Extended Query Lifecycle, in order to prepare a query I send the= <br> > Commands<br> <br> > Parse('P') / Describe('D') / Sync('S')<br> <br> > read 1/t/T/Z then to execute<br> <br> > Bind('B') / Execute('E') / Flush('H')<br> <br> </div>This is not a good idea. =C2=A0You *need* to use Sync to terminate a = command<br> sequence in order to be sure of proper error recovery (because if there'= ;s<br> an error during the Execute, the backend will discard subsequent messages<b= r> until it sees Sync).<br> <div><br> > If I skip the Flush after Execute I receive no data, if I Execute and = Sync<br> > I receive the the Limit of rows and a ReadyForQuery('Z').<br> <br> </div>That's probably because you're not wrapping this in a transac= tion so the<br> Sync implicitly does a commit, discarding the open portal. =C2=A0If you wan= t<br> to read from a portal in multiple steps then you should issue a BEGIN<br> first and a COMMIT (or ROLLBACK) after you're done. =C2=A0However, have= you<br> considered just processing the data on-the-fly instead of using a limit?<br= > <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 regards, tom lane<br> </blockquote></div><br></div> </div></div></blockquote></div><br></div> </div></div></blockquote></div><br></div> </div></div></blockquote></div><br></div> --001a11c26902c95e020500bc378b--