Re: Switching from CVS to GIT

> Date: Mon, 15 Oct 2007 20:45:02 -0400 (EDT)
> From: Daniel Barkalow <[email protected]>
> cc: Alex Riesen <[email protected]>, [email protected], [email protected], 
>     [email protected], [email protected], [email protected]
> 
> I believe the hassle is that readdir doesn't necessarily report a README in 
> a directory which is supposed to have a README, when it has a readme 
> instead.

Sorry I'm asking potentially stupid questions out of ignorance: why
would you want readdir to return `README' when you have `readme'?

> I think we want O(n) comparison of sorted lists, which doesn't 
> work if equivalent names don't sort the same.

You comparison function should be case-insensitive on Windows, or am I
missing something?

> > > - no acceptable level of performance in filesystem and VFS (readdir,
> > >   stat, open and read/write are annoyingly slow)
> > 
> > With what libraries?  Native `stat' and `readdir' are quite fast.
> > Perhaps you mean the ported glibc (libgw32c), where `readdir' is
> > indeed painfully slow, but then you don't need to use it.
> 
> We want getting stat info, using readdir to figure out what files exist, 
> for 106083 files in 1603 directories with a hot cache to take under 1s; 
> otherwise "git status" takes a noticeable amount of time with a medium-big 
> project, and we want people to be able to get info on what's changed 
> effectively instantly. My impression is that Windows' native stat and 
> readdir are plenty fast for what normal Windows programs want, but we 
> actually expect reasonable performance on an unreasonably-big 
> metadata-heavy input.

If that's the issue, then it's not a good idea to call `stat' and
`readdir' on Windows at all.  `stat' is a single system call on Posix
systems, while on Windows it usually needs to go out of its way
calling half a dozen system services to gather the `struct stat' info.
You need to call something like FindFirstFile, which can do the job of
`stat' and `readdir' together (and of `fnmatch', if you need to filter
only some files) in one go.  I don't know whether this will scan 100K
files under one second (maybe I will try it one of these days), but it
will definitely be faster than `readdir'+`stat' by maybe as much as an
order of magnitude.

> > > - no real "mmap" (which kills perfomance and complicates code)
> > 
> > You only need mmap because you are accustomed to use it on GNU/Linux.
> 
> I believe the need here is quick setup and fast access to sparse portions 
> of several 100M files. It's hard to beat a page fault for read speed.

If you need memory-mapped files, they are available on Windows.  I
thought the original comment about `mmap' was because it was used to
allocate memory, not read files into memory.

> We also expect to be able to make a sequence of file system operations 
> such that programs starting at any time see the same database as the files 
> containing the database get restructured.

Sorry, I don't understand this; please tell more about the operations,
``the same database'' issue (what database?) and what do you mean by
``the files containing the database get restructured''.

> A unixy pipeline was convenient

Windows supports pipelines with almost 100% the same functionality as
Posix.  Again, perhaps I'm missing something.