Re: cvs2git profile data

On 14 December 2010 05:59, Michael Haggerty <[email protected]> wrote:
> On 12/13/2010 05:42 PM, Alex Bennee wrote:
>> Our repository currently takes around 6 hours to do a full import (not
>> including the actual fast-import into git). It is very much a CPU
>> bound problem with the server showing plenty of spare RAM and not
>> overly stretched in terms of IO.
>>
> <snip>
>
> Thanks for the data.
>
> I suppose that there are two classes of CVS repository--those with few
> tags (e.g., tags only set manually) and those with a huge number of tags
> (e.g., tags set by nightly scripts), and that these two classes will
> have very different performance profiles.  I suspect that yours is one
> of the latter :-)

We don't have any automatic tags but there are a fair number of
branches and releases. The repository itself is fairly huge anyway
totalling around 94093 files on a checkout (helped by at least 3
copies of the Linux kernel).

> The expensive thing about tags is tabulating the possible parents for
> each tag.  This is expensive for many-tag repositories because the
> number of possible parent combinations scales something like
>
>    number_of_files * number_of_revisions_per_file *
> average(number_of_tags_per_revision^2)
>
> It is the average of the square that is fatal.

So if I aggressively exclude some branches and tags will that have a
positive impact?

> In case anybody is interested, here are the top 50 functions by "time"
> (time executing, not including time spent in subroutines), with a few notes:

> <snip>
>>   7610543 1295.604    0.000 1295.604    0.000 {posix.read}
>>  18460594 1285.036    0.000 3576.792    0.000 cvs_file_items.py:937(_adjust_tag_parent)
>>    172135 1245.699    0.007 1245.699    0.007 {posix.fork}
>
> The fork calls come from
> abstract_rcs_revision_manager.py:52(get_content), which suggests that
> you are not using the internal code for extracting content from the CVS
> repository.  The internal code should be much faster; I suggest you give
> it a try.

I used the default options and didn't specify --use-cvs or --use-rcs
so I'm not sure why it did that.

-- 
Alex, homepage: http://www.bennee.com/~alex/
http://www.half-llama.co.uk

------------------------------------------------------
http://cvs2svn.tigris.org/ds/viewMessage.do?dsForumId=1667&dsMessageId=2690709

To unsubscribe from this discussion, e-mail: [[email protected]].