RE: Package caching

> Ben> I ran the Cywgin `xemacs'; I notice it's quite fast once 
> it starts 
> Ben> up but dog-slow doing so.
> 
> Ben> I imagine much of the time is spent doing multiple stats down 
> Ben> through the package tree.
> 
> Unless something's changed that I don't know about, it does 
> only one pass.  But you're right that they take up some time.

I see.

> Ben> Then, next time, use this cache instead of actually reading the 
> Ben> directories. However, go through each resulting directory in 
> Ben> load-path and stat it to see what the modification time 
> is; if so, 
> Ben> we need to discard the file listing for that directory 
> and reread 
> Ben> it (thereby potentially adding new directories to the cache that 
> Ben> need to be recursed into, or removing directory trees from the 
> Ben> cache).
> 
> It seems to me this would incur the same number of stats as 
> we have. We don't actually stat the individual files---just 
> the directories. Or am I misunderstanding something you said?

Well, in that case my strategy would require only a single stat, not a stat
of each directory.  I was assuming you had to stat each file of each
directory to see if it was a subdirectory.

Btw I just did an strace under Cygwin and found some interesting things.
Unfortunately the results appear skewed; could some people under Unix run
strace (trace, truss, etc.) and send me the results?

Some things I noticed:

-- each .xpm file in the toolbar is getting loaded 3 times -- once when the
fallback, once when the X value, once when the mswin value.  Under Cygwin,
for some reason, these loads are taking absolutely forever; maybe it's
because we're using XpmReadDataFromFile.  Simple caching could avoid the 3x
load, but I wonder if we shouldn't just dump the XPM info directly into the
exe?
-- every time you load an autoload, you get the following sequence:

1. stat foo.elc.  This occurs in locate_file_in_directory_mapper,
`locate-file' from `load'.
2. access foo.elc.  From locate_file_open_or_access_file, same..
3. stat foo.elc.  From `insert-file-contents' from `find-magic-cookie...'
from `load'.
4. open foo.elc.  From same.
5. lseek(#, 1, SEEK_SET); lseek(#, 0, SEEK_CUR); fstat; readv(808 bytes);
readv(0 bytes); lseek(#, 1, SEEK_SET); readv(808 bytes); readv(0 bytes);
readv(0 bytes); close.  From same.
6. stat foo.elc.  This occurs in locate_file_in_directory_mapper,
`locate-file' from `load-internal'.
7. open foo.elc.  From locate_file_open_or_access_file, same.
8. fstat.  From load-internal, checking for out-of-date elc.
9. stat foo.el.  From same.
10. lseek(#, 0, SEEK_CUR); fstat; lseek(#, 0, SEEK_SET); readv(809 bytes);
readv(0 bytes); readv(0 bytes); close

A lot of statting here for one file.

#5 appears to be

lseek(#, 1, SEEK_SET) from insert-file-contents-internal
lseek(#, 0, SEEK_CUR) from make_filedesc_stream_1
fstat from filedesc_seekable_p, undecided_init_coding_stream
read(3000, returns 808 bytes) from i-f-c-i -> Lstream_read -> coding_reader
-> Lstream_read -> filedesc_reader
read(0, returns 0 bytes) from same
lseek(#, 1, SEEK_SET) ??? Can't find this
read = readv(808 bytes) ??? Can't find this
read = readv(0 bytes) ??? Can't find this
read(0, returns 0 bytes) from same
close

#10 appears to be

lseek(#, 0, SEEK_CUR) from make_filedesc_stream_1
fstat ??? Can't find this
lseek(#, 0, SEEK_SET)  ??? Can't find this
read(8192, returns 809 bytes)
read(8192, returns 0 bytes)
read(8192, returns 0 bytes)
close

"Can't find this" is presumably because something changed (for the better)
between 21.4 and 21.5; but it might be a Windows-Cygwin difference, since
the strace is 21.4 Cygwin and the debugging is 21.5 Windows.

BTW the 1 in lseek is due to the "son of a bitch" bug I just corrected.