Re: multispectral file access

On Mon, 1 Sep 2014, [email protected] wrote:
>
> Oh that's interesting. Yes, vips did something similar: too much VM
> was being chewed up when entire image files were mapped, so it now
> just maps "windows" of a few hundred scanlines and scrolls them around
> as operations read pixels.

This helps with VM address space but does not solve the caching issue 
unless you have a properly working madvise() and use it appropriately 
with MADV_DONTNEED or MADV_FREE so there is free memory available to 
handle subsequent major page faults due to file access.  Some time 
ago, I tested madvise() and found that it worked perfectly under 
Solaris 9, pretty good under FreeBSD (current at that time) if the 
memory span was large enough, and poorly under Linux (current at that 
time).  Obviously, this test was done quite some time ago. 
Everything has likely changed since then.  In particular, Linux had a 
bottleneck with expiring pages at that time (should be all different 
now).

> Measuring RSS: I spent ages on this, it's a very hard thing to do
> well. I tried five or six different things, including parsing strace
> output. The numbers above are found by running "ps" at 100 Hz and
> watching the RES line.

On Linux, a better way is likely to repeatedly parse the output of one 
of these /proc files:

/proc/PID/status
/proc/PID/stat
/proc/PID/statm

Where PID is the process ID of the process being executed.  I believe 
that 'ps' is often using the output of /proc/PID/stat.

The truely serious will try to grock the content of /proc/PID/smaps.

Bob
-- 
Bob Friesenhahn
[email protected], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/