Re: crash managing a large FSFS repository

Eric Gillespie wrote:

>That's exactly what it is.  It was much, much worse until r11701 and r11706.  However, fs_fs.c:fetch_all_changes still builds a giant hash in memory.  I wasn't sure what to do about this, and so left it alone.  I seem to recall asking for suggestions but not getting a response, but it's possible i overlooked it as i became busy elsewhere right afterwards.
>  
>
I've been looking at scaling issues with fs_fs, but mostly looking at 
repository size related issues.  This issue is isolated to individual 
transactions, so it's simpler to fix and test.

At the moment the code uses memory roughly proportional to the total 
lengths of all paths in the transactions.

One approach to reducing the amount of memory needed would be to use a 
data structure that models directories, rather than complete paths.   
Each directory node should have its own lookup table; the keys can be 
just the name of the immediate child relative to this node.  
Intermediate nodes for path components that haven't been seen themselves 
should be marked as such;  if the path is later explicitly encountered, 
the mark can be cleared (or vice versa).

This approach requires space roughly proportional to the number of  
directories and files in the transaction, rather than total path 
length.  For big, flat namespaces, this isn't much of a win, but it also 
isn't much worse; as the name space gets deeper, and closer to real 
source repositories, the win gets bigger. This approach also makes it 
faster to determine parent/child relationships.

Simon