Re: LFS on github (RFC)

Nice breakdown and some really good tips. I'm not sure if all the developers watch this list. Might be useful to post it on lfs-dev instead.

JH

On Sep 24, 2011, at 10:30 AM, Kevin Lyda wrote:

> This mail is rather long.  It's so long that the tl;dr part is the
> following four paragraphs.  My apologies, but it takes that much to
> explain this.
> 
> I've used LFS from time to time for work at two different companies.
> First as a quick way to a hermetic build environment and then as an
> educational tool.  I would really like to contribute back.  However I
> got distracted when I discovered subversion was still being used.  I'm
> currently getting better with git and thought it might be an
> interesting exercise to see how one ports a subversion repository to
> git.
> 
> So I did: https://github.com/lyda/Linux-From-Scratch
> 
> Now, let me say up front that this is a proof of concept and that it
> is up to yourselves if you want to use it (or rather a clone of it - I
> don't want the /lyda/ bit in that url...).  I can remove this from
> github even faster than I put it up there (for the record it took
> 38.899 wall-clock seconds to push the full repo to github).  I can
> continue to use my private repo for development and push out
> commits/patches with git-svn.
> 
> However it might be of interest to learn how I did this in case you'd
> like to do it yourselves and what further steps would be ideal to
> complete the migration to git.  The rest of this email will be very
> long and broken into the following sections: Stats, HOWTO, and Further
> Work.
> 
> == Stats ==
> Size of git repo with svn/trunk (master) checked out and all
> branches/tags available: 30M (29M when checked out from github -
> git-svn longs aren't there)
> Size of svn repo with all branches/tags available: 283M
> Size of svn repo with just svn/trunk checked out: 13M
> 
> I didn't time it, but it took several hours to git svn clone the LFS
> svn repository.  It takes less than a minute to git clone the LFS git
> repository.
> 
> == HOWTO ==
> 
> To grab what's on github just run:
> 
> git://github.com/lyda/Linux-From-Scratch.git
> 
> Then add this to your .git/config:
> 
> [svn-remote "svn"]
>        url = svn://svn.linuxfromscratch.org/LFS
>        fetch = trunk:refs/remotes/svn/trunk
>        branches = branches/*:refs/remotes/svn/*
>        tags = tags/*:refs/remotes/svn/tags/*
> [svn]
>        authorsfile = /some/path/authors-transform.txt
> 
> Contact me off-list for the authors-transform.txt file.  People get
> cranky about email addresses posted on mailing lists.
> 
> Once you have that you can do a git svn fetch.  The first time it runs
> it will take a long while to build all the bits git-svn needs (it will
> be almost entirely local work).  On my machine it took 21 minutes and
> was all local except for a quick check with the lfs subversion server
> to see if there had been new commits.  As time goes on there will be
> and there will be a bit more external work.
> 
> To build the git repo directly from subversion I did the following.
> As I said, the git svn clone step takes a long, long time so it's
> probably best to avoid it.  However there were some choices I made
> that people might not agree with and might want to do differently.  I
> discuss that with comments broken out with ------- within the scripts
> below.
> 
> # The following commands create the mapping between svn and git user names.
> svn co svn://svn.linuxfromscratch.org/LFS lfs.svn
> cd lfs.svn
> svn log -q \
>  | awk -F '|' \
>        '/^r/ {sub("^ ", "", $2);sub(" $", "", $2);print $2" = "$2" <"$2">"}' \
>  | sort -u > ../authors-transform.txt
> # edit ../authors-transform.txt
> #   Used http://www.linuxfromscratch.org/lfs/view/6.8/appendices/acknowledgements.html
> #   and other acknowledgement files to find addresses.
> #   Also used commands like:
> #     awk '"jon" == $3 { print $1 }' ../svn.log|xargs -n1 svn log -r
> #   to find log messages and then use that in a search.
> 
> --------
> There's really no other way to do this.  I'm happy to mail this file
> to any developer that would want it.
> --------
> 
> # The following will clone the svn repo and keep the metadata used
> # by git-svn.
> git svn clone \
>      -A authors-transform.txt \
>      --stdlayout \
>      --prefix=svn/ \
>      svn://svn.linuxfromscratch.org/LFS/ lfs.git
> 
> --------
> Now there are many things here that one could do differently.
> 
> First, I could have used --no-metadata.  To explain what that is, it's
> helpful to look at a commit in the LFS git repo.  Here's the current
> HEAD:
> 
> commit 0e976ecfebf55dfc8f471394c46add3c73cce9c6
> Author: Bruce Dubbs <[email protected]>
> Date:   Fri Sep 23 17:06:43 2011 +0000
> 
>    Allow variables in rc.site to override defaults
> 
>    git-svn-id: svn://svn.linuxfromscratch.org/LFS/trunk@9602
> 4aa44e1e-78dd-0310-
> 
> Note the git-svn-id thing tacked onto the end.  That was obviously not
> in Bruce's original commit.  That's used by git-svn to rebuild it's
> mapping of svn and git - it's why you can clone from github and git
> svn fetch will work.
> 
> However if you were to decide to ditch subversion entirely, this
> wouldn't really be required and --no-metadata will remove it.
> 
> The --prefix=svn/ isn't required either.  It prefixes all tags with
> svn.  You might not want that.  Of course it's quite possible to to a
> mass change of tags.  See below for one example of doing that.
> 
> Lastly this command might quit before completion due to a network
> issue.  If it does, merely do the following:
> 
> git svn fetch
> 
> and it will continue on.  When it completes run:
> 
> git reset --hard
> 
> and that will set up your working directory.
> --------
> 
> # Copy the svnignore settings:
> # TODO: Include cd command.
> git svn show-ignore > .gitignore
> git add .gitignore
> git commit -m 'Convert svn:ignore properties to .gitignore.'
> 
> --------
> I did not do this, but it's simple to do.  Currently the .gitignore
> file would contain:
> 
> # /BOOK/
> /BOOK/*.bz2
> /BOOK/*swp
> 
> # /BOOK/bootscripts/
> --------
> 
> # Convert svn tags from lightweight git branches to git tags.
> # TODO: Is the ref correct here?  Shouldn't they be refs/svn/...?
> git for-each-ref --format='%(refname)' refs/heads/tags |
> cut -d / -f 4 |
> while read ref
> do
>  git tag "$ref" "refs/heads/tags/$ref";
>  git branch -D "tags/$ref";
> done
> 
> --------
> I didn't do this either but it seems like a good thing to do.  I'm not
> sure the script is right though.  I think it needs to use
> refs/remotes/svn/tags/
> --------
> 
> # Push to github:
> git remote add origin [email protected]:lyda/Linux-From-Scratch.git
> # Move all the remotes/svn/* tags into svn/* in order to push them to github:
> git for-each-ref --format='%(refname)' refs/remotes/svn \
>  | sed s-refs/remotes/-- \
>  | while read b; do
>      git branch -t $b remotes/$b
>    done
> # Tested with this command to make sure push would work:
> git push -n -u --mirror origin
> # It did so did real push:
> time git push -u --mirror origin
> 
> == Future Work ==
> 
> I would really suggest that the developers to a re-import as I
> describe above considering the options as I describe them.  I also
> would suggest a clean cut-over.  Close down subversion and switch to
> git on github (or gitorious - http://gitorious.org/).  The git hosts
> have nice issue trackers and wikis (which is itself a git repository)
> and has many of the tools you'd need.
> 
> Doing the git-svn thing to fetch and dcommit changes is fiddly and
> prone to error and will not be a nice experience for people.  That
> said, what I've pushed to github should be able to do that for people.
> 
> On the github side I think there are a number of useful tools you
> might consider.  First you can make a README.md file that will display
> on the opening page.  The discussion on lfs-chat about git workflow
> would make a fantastic README.md file to ease concerns about using a
> new VCS.  It is intimidating and I'm only just learning git.  Clear
> instructions for common workflows would lower the barrier to
> contributing.
> 
> Speaking of barriers to entry, apparently github has subversion
> support.  Not sure if it is wise, but information about it is here:
> https://github.com/blog/644-subversion-write-support  I've never used
> it and have no idea how well it works (or even if it still works!).
> 
> On the topic of development workflow/culture I don't think git
> requires a dictator for life person (mentioned on the lfs-dev list).
> With github you can have multiple committers to a tree.  Anyone can
> clone it and you can then do pull requests which makes it easier to
> send in contributions.  If someone's being a jerk, the non-jerks can
> clone the tree and announce they're the non-jerk lfs tree!  This seems
> easier than in the current system.
> 
> BTW, I have no connection to github, they're just who I've used so
> far.  I'm guessing gitorious has similar features.
> 
> Lasty, the created tags and branches have some flaws.  Some tags and
> branches were created multiple times and the conversion process
> reflects that in an odd way.  For example:
> 
>  remotes/svn/6.1
>  remotes/svn/6.1@4846
>  remotes/svn/6.1@4851
> 
> The "real" 6.1 I suspect is remotes/svn/6.1@4851.  After importing
> it's a simple matter of a few git branch commands to clean that up.
> However I left the tree on github in its post-import state so people
> could see what they get after importing.
> 
> Anyway, thanks if you read this far and hope the info here was useful.
> LFS has been an excellent tool for me at work.  You guys have done a
> fantastic job.  If this is not desired I'm happy to remove it.  If
> it's useful that's also great.  And if it's offensive/unwanted, please
> accept my apologies in advance.
> 
> Kevin
> 
> -- 
> Kevin Lyda
> Dublin, Ireland
> US Citizen overseas? We can vote.
> Register now: http://www.votefromabroad.org/
> -- 
> http://linuxfromscratch.org/mailman/listinfo/lfs-chat
> FAQ: http://www.linuxfromscratch.org/faq/
> Unsubscribe: See the above information page

-- 
http://linuxfromscratch.org/mailman/listinfo/lfs-chat
FAQ: http://www.linuxfromscratch.org/faq/
Unsubscribe: See the above information page