Re: Dealing with large datasets

--===============1631062778310788274==
Content-Type: multipart/alternative; boundary=089e0118429eb23c9b052b1b5225

--089e0118429eb23c9b052b1b5225
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks!  That is helpful and encouraging.  It would be great to get an
official word on this regarding all of the mentioned constructs.  But it
also leads to my second question having to do with nested queries.  It goes
as follows:

 In C, Java, C#, Python, etc. there is not an issue because they all have
cursors.  I don't see cursors on CLSQL.

You write a utility function that encapsulates some needed functionality.
It gets called when needed.  What it actually does is, rightfully,
abstracted.  Actually, what it does is perform a select against a database,
do some calculations on the records, and return the result.

Now, let's say you have another function.  I'll call this one an
application function.  This function does a select on the database, spins
through the records and calls the utility function for each of its
records.  So, you have a nested select.  Obviously, the utility function
spin should not interfere with the application function spin.  Normally,
this is not a problem when cursors are used, but I see no cursors in
CLSQL.  Is this an issue with CLSQL?

Thanks!

Blake McBride



On Sat, Feb 6, 2016 at 4:07 AM, <jhyiugjhvbjh234-JGs/[email protected]> wrote:

> >The CLSQL documentation documents DO-QUERY, LOOP, and MAP-QUERY but
> doesn't
> >make clear what is going on inside.  Are _all_ the records obtained firs=
t
> >as one big list and than mapped over, or is it more intelligent obtainin=
g
> >manageable sized blocks at a time, or even dealing one record at a time
> and
> >let the driver buffer it?
> >
> >This is a really big issue.  I appreciate clarification.
>
> I am not a CLSQL developer, but I use CLSQL in production.
>
> I have skimmed the do-query code some time ago, and I did compare
> performance of do-query and clsql:select.
>
> do-query never loads the entire dataset in the Lisp memory, it only
> keeps the current piece there.
>
> do-query seems to request rows one-by-one from the driver, but it is
> possible that the driver code does some buffering =E2=80=94 I have never
> checked.
>
>
>
>

--089e0118429eb23c9b052b1b5225
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thanks!=C2=A0 That is helpful and encouraging.=C2=A0 It wo=
uld be great to get an official word on this regarding all of the mentioned=
 constructs.=C2=A0 But it also leads to my second question having to do wit=
h nested queries.=C2=A0 It goes as follows:<div><br></div><div><div>=C2=A0I=
n C, Java, C#, Python, etc. there is not an issue because they all have cur=
sors.=C2=A0 I don&#39;t see cursors on CLSQL.=C2=A0</div><div><br></div><di=
v>You write a utility function that encapsulates some needed functionality.=
=C2=A0 It gets called when needed.=C2=A0 What it actually does is, rightful=
ly, abstracted.=C2=A0 Actually, what it does is perform a select against a =
database, do some calculations on the records, and return the result.</div>=
<div><br></div><div>Now, let&#39;s say you have another function.=C2=A0 I&#=
39;ll call this one an application function.=C2=A0 This function does a sel=
ect on the database, spins through the records and calls the utility functi=
on for each of its records.=C2=A0 So, you have a nested select.=C2=A0 Obvio=
usly, the utility function spin should not interfere with the application f=
unction spin.=C2=A0 Normally, this is not a problem when cursors are used, =
but I see no cursors in CLSQL.=C2=A0 Is this an issue with CLSQL?</div></di=
v><div><br></div><div>Thanks!</div><div><br></div><div>Blake McBride</div><=
div><br></div><div><br></div><div class=3D"gmail_extra"><br><div class=3D"g=
mail_quote">On Sat, Feb 6, 2016 at 4:07 AM,  <span dir=3D"ltr">&lt;<a href=
=3D"mailto:jhyiugjhvbjh234-JGs/[email protected]" target=3D"_blank">jhyiugjhvbjh234@mail.=
ru</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"marg=
in:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=3D""=
>&gt;The CLSQL documentation documents DO-QUERY, LOOP, and MAP-QUERY but do=
esn&#39;t<br>
&gt;make clear what is going on inside.=C2=A0 Are _all_ the records obtaine=
d first<br>
&gt;as one big list and than mapped over, or is it more intelligent obtaini=
ng<br>
&gt;manageable sized blocks at a time, or even dealing one record at a time=
 and<br>
&gt;let the driver buffer it?<br>
&gt;<br>
&gt;This is a really big issue.=C2=A0 I appreciate clarification.<br>
<br>
</span>I am not a CLSQL developer, but I use CLSQL in production.<br>
<br>
I have skimmed the do-query code some time ago, and I did compare<br>
performance of do-query and clsql:select.<br>
<br>
do-query never loads the entire dataset in the Lisp memory, it only<br>
keeps the current piece there.<br>
<br>
do-query seems to request rows one-by-one from the driver, but it is<br>
possible that the driver code does some buffering =E2=80=94 I have never<br=
>
checked.<br>
<br>
<br>
<br>
</blockquote></div><br></div></div>

--089e0118429eb23c9b052b1b5225--

--===============1631062778310788274==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KQ0xTUUwgbWFp
bGluZyBsaXN0CkNMU1FMQGxpc3RzLmtwZS5pbwpodHRwOi8vbGlzdHMua3BlLmlvL2NnaS1iaW4v
bWFpbG1hbi9saXN0aW5mby9jbHNxbAo=

--===============1631062778310788274==--