Using UNIX:UNIX-MMAP to read large text files under CMUCL?

"Denis Papathanasiou" <[email protected]>
Newsgroups gmane.lisp.cmucl.general
Message-ID <[email protected]>
I'm familiar with how to use mmap() in C for this (reading large text
files), but I'm not sure how to go about it under CMUCL.

I found this example in comp.lang.lisp, but since it just returns the
pointer (the output of the mmap call as an integer), how do I convert
that pointer into some kind of lisp object/structure that I can use to
read the results, either with (read-sequence) or (read-line)?

    (defun mmap (file offset length &optional write-p)
      (let* ((ofs32 (if (zerop (logand offset #x80000000))
                      offset
                      (- offset #x100000000))) ; Fake up a "(signed-byte 32)"
             (mode (logior #+linux unix::O_SYNC     ; Not on BSD (*sigh*)
                           (if write-p unix:O_RDWR unix:O_RDONLY)))
             (fd (unix:unix-open file mode 0)))
        (if (not fd)
          (error "Can't open ~s: ~s" file (unix:get-unix-error-msg))
          (unwind-protect
              (let ((mode (logior unix:PROT_READ
                                  (if write-p unix:PROT_WRITE 0))))
                (system:sap-int
                  (unix:unix-mmap nil length mode unix:MAP_SHARED fd ofs32)))
            (unix:unix-close fd)))))

Also, when using CMUCL's mmap call, am I forced to open the file with
(unix:unix-open) or can I use (with-open-file) as well?

When I tried the latter, the call to (unix:unix-mmap) returned a null value:

CL-USER> (with-open-file (file-stream "/path/to/very_large_file.txt" :direction
:input)
           (let ((mmap-ptr (unix:unix-mmap nil (file-length
file-stream) unix:PROT_READ unix:MAP_PRIVATE (system:fd-stream-fd
file-stream) 0)))
             mmap-ptr))
NIL
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.