Re: Using UNIX:UNIX-MMAP to read large text files under CMUCL?

	[CC'ed to cmucl-imp -- see issue on CVS tree]

* "Denis Papathanasiou" <e4044eb40704050934m6f8536ceu29e406a1c33e8216@XXX> :

| I'm familiar with how to use mmap() in C for this (reading large text
| files), but I'm not sure how to go about it under CMUCL.
|
| I found this example in comp.lang.lisp, but since it just returns the
| pointer (the output of the mmap call as an integer), how do I convert
| that pointer into some kind of lisp object/structure that I can use to
| read the results, either with (read-sequence) or (read-line)?
<snip>

I don';t think you can call lisp functions directly on the storage
returned by unix:mmap. Maybe you can use OPEN with :MAPPED T, which
should give you a simple-stream to a mmaped backed file on which you
can use lisp functions.

| Also, when using CMUCL's mmap call, am I forced to open the file with
| (unix:unix-open) or can I use (with-open-file) as well?
|
| When I tried the latter, the call to (unix:unix-mmap) returned a null value:


| CL-USER> (with-open-file (file-stream "/path/to/very_large_file.txt"
| CL-USER> :direction :input)
           (let ((mmap-ptr (unix:unix-mmap nil (file-length
| file-stream) unix:PROT_READ unix:MAP_PRIVATE (system:fd-stream-fd
| file-stream) 0))) mmap-ptr))

There was a problem reported and apparently fixed in

http://permalink.gmane.org/gmane.lisp.cmucl.devel/9910

My local CVS tree shows this entry for unix-glibc2.lisp
----------------------------
revision 1.37
date: 2006/09/06 01:07:09;  author: rtoy;  state: Exp;  lines: +14 -4
o Add MAP_FAILED
o UNIX-MMAP cannot use the syscall macro because addresses can be
  "negative".   (Steal implementation from unix.lisp.)
----------------------------

However the CVS-Cmucl tree at common-lisp.net does not seem to have
this fix. Perhaps it was lost during the restore after it was
corrupted?

I'm not sure if this problem is related as I believed UNIX-MMAP should
fail unconditionally without the fix.
--
Madhu