CLSQL:MAP-QUERY more memory hungry than CLSQL:QUERY

Jason H <[email protected]> Thu, 8 Jul 2010 22:30:52 -0700
Newsgroups gmane.lisp.clsql.general
Message-ID <[email protected]>
Hi,

I'm using the ODBC back-end of clsql-5.0.2 to load data from an MDB file.
The data consists of 1839364 rows and each row has 7 integers.  At first,
I used CLSQL:QUERY to get a list of 1839364 7-element tuples and processed
this long list into something else.  This worked but generated lots of garbage
because of the transient long list returned by CLSQL:QUERY.  To reduce the
amount of garbage, I switched to CLSQL:MAP-QUERY but was quite surprised
to find out CLSQL:MAP-QUERY used even more memory to do the same thing.

To make what I just described more concrete for this email, I created
the following 2 test functions:

(defun test-1 ()
  (clsql:map-query nil
      	           #'identity
		   (clsql:query "select c1, c2, c3, c4, c5, c6, c7 from Cells")))

(defun test-2 ()
  (clsql:query "select c1, c2, c3, c4, c5, c6, c7 from Cells") )

The 2 test functions were run in the following way:
  (progn
    (test-1)
    (null (test-2)))
while being profiled.

This is the profile report:

  seconds  |    consed   | calls |  sec/call  |  name
---------------------------------------------------------
    52.395 | 147,196,856 |     1 |  52.395000 | TEST-2
    44.894 | 161,911,208 |     1 |  44.894000 | TEST-1
---------------------------------------------------------
    97.289 | 309,108,064 |     2 |            | Total

estimated total profiling overhead: .000 seconds
overhead estimation parameters:
  4.e-8s/call, 6.028e-6s total profiling, 1.48e-6s internal profiling

This result is contrary to what I expected based on what I saw in
the source code of CLSQL-SYS::MAP-QUERY-FOR-EFFECT: the same ROW was
passed to CLSQL-SYS::DATABASE-STORE-NEXT-ROW repeatedly.  I, a CLSQL idiot,
must be missing something big time.  Could somebody enlighten me?

Best wishes,

Jason

ps.  i'm using win32 version of sbcl 1.0.37.12.