Re: Re: Streaming MIME parser

suresh! wrote :

> >mmap() will not store the entire content of the file
> >in memory. That will fetch the content of the file
> >on-demand and remove the less used parts of the file
> >from memory when needed.
> 
> I read the documentation for mmap. It does not clearly
> say how mmap handles lot of memory. According to my
> understanding, if a huge file needs to be mmaped then
> a virtual address range of length requested is
> obtained. How much of the file actually remains in
> memory and how much is swapped out? So, does the
> unused portions of the file remains in the swap space
> until it is demanded? Please tell if my understanding
> is correct?

Unused portions of the file will remain on disk in the file.
Especially, on most Unix or Unix-like systems, with flags PROT_READ or 
when MAP_SHARED is used, when physical memory is needed for some other 
needs, it will stay on disk in the file.
This is to the system to decide how to handle it, this is system 
specific. This depends on the system, on the physical memory you have, 
the available physical memory when running this part of the application. 
You can also read some papers on virtual memory.
for example :
http://www.usenix.org/events/usenix99/full_papers/cranor/cranor_html/

What you have to understand is that :
The unused portions of the file will remain on disk until it is 
demanded, then when physical memory is needed, this used part of memory 
will be released and made available for other use (other use includes 
reading other parts of the file).

> >I think that mmap() is really what you need there, I
> >suggest you to read further documentation about mmap
> 
> Not running out of memory for huge messages is my
> primary concern at this point of time. A stream based
> SAX stuff would be in the wish list :)

mmap() will still fit your needs for that.

-- 
DINH V. Hoa,

"Pétage de plomb, excès de bière, de coke... probablement..."
     -- Emmanuel Delahaye

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQBCaUsN7w+pfAWHa3MRAnLlAJ9oa7p9PMhRdRrlF+iTwDJgF8xngwCeLwn6
IVPtcS/CQM0xkorsGEG1sHg=
=7g+j
-----END PGP SIGNATURE-----