cvs2git: One possible bug, and two other issues

Hello all.

I'm in the process of converting a both huge and old CVS repository to
git.  I've found several problems within our repository - but I've
also found what I think are some issues in cvs2git.

I have done my testing against the latest release 2.3.0 - but when I
checked out the latest revision from SVN (5311) it turned out that 2
of my 5 issues had already been taken care of ... :-)

When that is said, it did take me some time to figure out why keywords
were suddenly not expanded any more in rev 5311.  I finally figured
out that I had to change the options file to add
KeywordHandlingPropertySetter('untouched') instead of
KeywordHandlingPropertySetter('collapsed') to
ctx.file_property_setters.  I can see that there is a FIXME to improve
the documentation though ... :-)

Of these three remaining issues one looks like a regular bug, one is a
change which I believe would be of general interest - and the last may
only be interesting to a smaller audience.  I've got patches to all
three.

I've been working against the latest release - 2.3.0 - and my comments
and patches below reflect that.  I can see that some of the code has
changed in the meantime, so I'll have to redo some of that work to
match the SVN head if my proposed changes are accepted.

I was a bit unsure if it was better to keep this as one big mail, or
split it into one for each issue.  In the end I decided to keep it
together - but feel free to split it if that seems better.

For all the examples below I have been using the included options file
cvs2git-test.options with release 2.3.0 and cvs2git-svn.options with
svn 5311.  They are supposed to produce identical behaviour.

The examples are included both as shell-scripts with the CVS commands
to create the repository, and a tar ball of the repository in
question.  Warning!  The shell-scripts will unconditionally remove
three files/dirs in /tmp - repo, sandbox and initial.


Issue 1 - Branches getting the wrong parent(s)
----------------------------------------------

Sample repository: ex1.tar
Creation script  : create_ex1.sh
Patch            : patch1.txt

When I looked at the conversion of our CVS big repository, I noticed
quite a lot of commits manufactured to create branches.  Most of those
were "trivial" and quite unnecessary.  Release 2.3.0 appears to create
an extra commit for every branch - but SVN head has solved this
similarly to how it was already done for tags.

But that still leaves some very strange merge commits created for
branches that are absolutely *not* merges.  In the example repository
branch b2 is created off MAIN, a change in one of the two files is
commited on b2 and branches b1 and b3 are created from that point.

When converting with plain 2.3.0 I get the expected trivial commits to
create b1 and b2 - but a merge commit to create b3.  Note that if I
change the name of b2 to a2, b3 becomes a simple commit.  So changing
the name changes the behaviour!

Retesting with SVN 5311 I get a similar picture.  I don't get
manufactured commits for b1 and b2 any more - but I do get one for
b3.  It is not marked as a merge, but the commit message says that it
sprouted from master with a cherrypick from b1.  Changing branch name
b2 to a2 again gives me the expected behaviour where all three
branches are now on the same commit.

In 2.3.0 I tracked this down to the code in cvs_file_items.py that
grafts branches onto their preferred parents.  Branch b3 turns out to
have b1 as preferred parent.  This works out ok for file1, but for
file2 b1 gets grafted onto b2 *before* b3 is handled.  This causes b1
to leave the set of possible parents, so b3 now has a preferred parent
which isn't possible any more.  Thus it chooses another, and we get
the merge commit.

One way to solve this could be to make a more extensive search for
possible parents.  For my patch I chose a simpler way however - store
the original set of branch_ids in the source commit in a new attribute
branch_ids_orig - and iterate over that to find a parent.  It works
well for the tests I've done so far.



Issue 2 - 'Why'-information in manufactured commits
---------------------------------------------------

Sample repository: ex2.tar
Creation script  : create_ex2.sh
Patch            : patch2.txt

When I test-converted our big CVS repository I saw a huge amount of
commits 'Manufactured by cvs2svn'.  For some files the manufactured
commits outnumbered the real ones 10 to 1, making the history almost
impossible to use.  I wanted to know why they showed up, so I had a
chance to get rid of them.

To that purpose I modified git_output_option.py.  I renamed
_is_simple_copy to _why_not_simple_copy, and modified it to return
None for "Ok, this is a simple copy" and a string if not.  The string
describes why a simple copy wouldn't work.  The callers of
_is_simple_copy are modified to pass the returned string into the
commit message of the resulting manufactured commit.

The text itself could be more readable, but the current state should
be enough as proof-of-concept.  If this idea is accepted the
readability can always be improved.

I think this would be a good general extension of cvs2git.



Issue 3 - Allow selected tags to "expand" to cover more files
-------------------------------------------------------------

Sample repository: ex2.tar
Creation script  : create_ex2.sh
Patch            : patch3.txt

In our CVS repository we normally tag bugfixes with "before" and
"after" tags, to make it easier to apply them to another branch later
on.  Mostly we let those tags cover the entire repository.  But
sometimes - especially when the commit changes only a few files - we
just tag the affected files.

In retrospect we could question if that was a good decision - but it
seemed a good idea at the time ... :-) ... "cvs diff -rbefore -rafter"
works well in both cases - and by just tagging the affected files we
reduced the already huge number of tags on other files.

In any case we now have a repository where a significant number of
tags don't cover all the files in the repository.  When converting
this, cvs2git has to manufacture a commit removing all other files
before it can place the tag.  And that is of course necessary to
accurately represent the repository.

But this was not the intention of those tags - they were meant to
attach to specific commits.  And they do tag the files actually
changed in the commit - just not all the files in the repository.

So I added some more code to allow certain tags to pass the
"is_simple_copy" test.  If they have only one LOD, if the tagged files
is a proper subset of the files in the commit - and if the tag passes
a custom check - then it will be created as a simple copy - in effect
extending the tag to cover all files in the commit.

If the custom check always returns False ("NO") we get the current
situation.  If it always returns True ("YES") tags will only get
manufactured commits if they refer to several LODs.  And in my case I
can have the custom check look at the tag name, and allow those I know
to be before/after tags.

In the sample repository the tags t2 and ok2 refer to the same
revisions.  Similarly for t3 and ok3.  When running with expansion
allowed for tags named 'ok*', ok3 will attach to an existing commit
while t3 will not - because of the names.  Both t2 and ok2 will get
their own commits, as they have two LODs each.

I think this could be a useful extension of cvs2git - but I can also
see that it might be too specialised for general use.

Note that patch3.txt is intended to apply on top of patch2.txt.


Best regards,
  Anders Pilegaard ([email protected], [email protected])

------------------------------------------------------
http://cvs2svn.tigris.org/ds/viewMessage.do?dsForumId=1667&dsMessageId=2695644

To unsubscribe from this discussion, e-mail: [[email protected]].