Re: comm-central future as a branch

I wasn't working on searchfox during the early days when the
permalinks were made based on the git revision, so I don't have the
full context on that. But searchfox generally has git-centric
architecture (e.g. the tool to generate the blame repo works off a git
repo). In general more codebases have git repos than hg repos, so it
makes sense from that perspective.

On Thu, Oct 11, 2018 at 9:58 AM Magnus Melin <[email protected]> wrote:
>
> We definitely want to keep Searchfox working, I use it all the time.
>
> I think at this stage it's hard to say anything about the details. If
> Searchfox permalinks aren't based on the hg revisions, I imagine
> supporting old permalinks can get somewhat difficult. OT but why isn't
> it based on the mercurial version?
>
>   -Magnus
>
> On 09-10-2018 19:32, Kartikaya Gupta wrote:
> > Have you given any thought as to what would happen to the git mirror
> > of the comm-central repo that currently lives at
> > https://github.com/mozilla/releases-comm-central ? I ask because
> > searchfox uses this git repo to do the indexing of comm-central and
> > that might break with these changes. If you care about still having
> > searchfox indexing available for comm-central after this change it
> > would be good to continue having some sort of git mirror. If it's a
> > brand new git mirror with different hashes we'll have to rebuild the
> > blame repo and any comm-central permalinks (in bugs, etc.) will likely
> > break. Or we can try harder to keep permalinks working by mapping the
> > old hashes to the new ones but that's more work.
> >
> > Cheers,
> > kats
> > On Tue, Oct 9, 2018 at 11:48 AM Ben Bucksch <[email protected]> wrote:
> >> I like this idea. I'm in favor.
> >>
> >> If I understood correctly, you're proposing to take a mozilla-central
> >> (or a mozilla-unified) repo, put comm-central checkout into the comm/
> >> subdirectory, and push that to the branch "comm/default". I think that's
> >> the right way to do things, yes.
> >>
> >> We do not even need buy-in from Firefox people, because hg is a
> >> distributed system, so we can simply have our own repository that
> >> contains a copy of mozilla-central plus comm/ (from former comm-central).
> >>
> >> I'm also in favor of re-adding CVS history. It's important for us, more
> >> important than for Firefox, due to the longevity of our project.
> >>
> >> As for "which combination of mozilla and thunderbird is building and
> >> working?", I would suggest to make a branch comm/bleeding where we merge
> >> mozilla-central immediately and completely without tests. So, if there
> >> are API changes in Mozilla and Thunderbird hasn't been updated yet, it
> >> will be broken. (Code-wise, this is what we currently get, by checking
> >> out mozilla-central and comm-central independently.) So, you have the
> >> option of easily continuing to check out the latest and bleeding edge.
> >> Then, once we tested the combination and we confirmed that it's working,
> >> we merge comm/bleeding into comm/default, and it becomes the new
> >> official Thunderbird development branch. This way, fixing an API
> >> breakage becomes a hg commit that fixes the API and hg merge operation.
> >>
> >> The main advantage of this setup is that it's clear which version of
> >> Mozilla works with which version of the Thunderbid code, down to the
> >> commit level. And we capture this information using the version control
> >> system itself, but with some build scripts.
> >>
> >> That should dramatically simplify development and build scripts.
> >>
> >> Ben
> >>
> >>
> >> Am 10/9/2018 um 10:06 AM schrieb Magnus Melin:
> >>
> >>> I'd like feedback on the below plan for having Thunderbird in a single
> >>> repository.
> >>> Let's have follow-ups go to dev-planning
> >>>
> >>> TL;DR version: planning to have current comm-central live as a branch
> >>> instead of a separate repository.
> >>>
> >>> ---
> >>>
> >>> Background
> >>> ----------------
> >>>
> >>> When the Mozilla move from CVS to hg took place in 2008, the code
> >>> related to Thunderbird was not included in mozilla-central and instead
> >>> placed in the comm-central repository. Under the hood Thunderbird uses
> >>> the mozilla-central code, so to build it currently has to check out
> >>> both repositories. This is not ideal, for many many reasons.
> >>>
> >>> I wrote [an email to
> >>> maildev](http://lists.thunderbird.net/pipermail/maildev_lists.thunderbird.net/2017-November/000875.html)
> >>> last year listing some of the drawbacks of this setup - but
> >>> essentially what it comes down to is that version control was not
> >>> meant to be used like this and it makes workflows error prone, in
> >>> addition to having to do various hacks in the build automation.
> >>> Enabling autoland, phabricator and other tools used by Mozilla core
> >>> could also be done much more easily for comm-central with a single
> >>> repository.
> >>>
> >>> The mail spurred some related discussion on git vs hg and importing of
> >>> history, but basically there was agreement among the comm-central
> >>> developers to go ahead with this change.
> >>>
> >>> Thunderbird has now migrated its build automation to Taskcluster, and
> >>> also moved to building with mozilla-central as topdir, with a comm/
> >>> sub-folder containing comm-central code. (We used to instead have a
> >>> mozilla subdir checkout of mozilla-central as child the comm-central
> >>> checkout.)
> >>>
> >>> In 2014 there was [a proposal to merge comm-central code into
> >>> mozilla-central](https://groups.google.com/forum/#!msg/mozilla.dev.planning/P8oHKTCWSiQ/76ECituhK-IJ)
> >>> but this was rejected since the powers that be did not want 1) the
> >>> confusion about what code belongs to what project, 2) pull/tree size,
> >>> and 3) it was seen as irrelevant to Firefox. The proposal below has
> >>> none of these obstacles, and depending on the physical repository
> >>> location, may not change anything at all for a Firefox developer.
> >>>
> >>> I think it's now time to revisit the single repository idea and pull
> >>> that through in one way or the other. So, on to the proposal. For
> >>> reference, on disk comm/ is on currently 122 MB (+ comm/.hg 192 MB),
> >>> so size-wise it is comparatively small, with a mozilla-central
> >>> checkout weighing in at 2.3 GB. Including the relevant CVS history,
> >>> the comm/.hg would be 317 MB in total.
> >>>
> >>> How
> >>> ------
> >>>
> >>> To capture the essence of what Thunderbird is, we would make
> >>> comm-central a named branch of the mozilla-central. Let's call this
> >>> branch "comm/default".
> >>>
> >>> Initially, we'd pull in a hg converted version of the current
> >>> comm-central default branch, putting all the files into a comm
> >>> sub-folder in the base directory of the comm/default branch. This is
> >>> the way that the Thunderbird build system already wants the code laid
> >>> out. With the comm sub-folder it will be clear to everyone involved
> >>> which code is on mozilla-central code and which is not. It is possible
> >>> to import the CVS history for the related directories too while we're
> >>> at it, but it adds some size. Importing related CVS history only adds
> >>> around 125 MB extra so I'm inclined to include it. With the new
> >>> comm/default branch pulled in, we merge the curent mozilla-central
> >>> default branch to comm/default. The resulting code is now the same as
> >>> status quo (of the both repos pulled in), just that it's all version
> >>> controlled in the same repository.
> >>>
> >>> Once this is all set up, Thunderbird development work would move on on
> >>> the comm/default branch, and there would be periodical merges with the
> >>> (mozilla-central) default branch. These merges could be automated if
> >>> wanted.
> >>>
> >>> Quite often it's the case that Thunderbird code needs to be adjusted
> >>> to build, due to changes in mozilla-central code. For cases of known
> >>> incoming bustage Thunderbird now would have the possibility of waiting
> >>> to do the merge until a fix is available. I would suggest never to
> >>> back out bustage causing mozilla-central changesets from the
> >>> comm/default branch, but to detect other incoming bustage by doing
> >>> builds from a comm/band-aid branch (branced from comm/default). The
> >>> details on this can be discussed later though, not to derail
> >>> discussion about the main issue here. There are many alternatives, and
> >>> it's also completely possible to do what we currently do.
> >>>
> >>> Where
> >>> ---------
> >>>
> >>> In the earlier discussions i's become clear there is a bunch of
> >>> confusion around what is a branch and what is a repository, so please
> >>> note the difference. Aa repository is the physical location where the
> >>> history of a project is stored. Every repository does not have to
> >>> carry every branch. I.e., the mozilla-central repository could carry
> >>> the comm/default branch, or the comm/default branch could exist only
> >>> in a comm-central2 repository somewhere else. In all cases it's
> >>> trivial for developers who have mozilla-central checked out, to add
> >>> the comm/default branch to their local checkout.
> >>>
> >>> So for the location there are at least three options:
> >>>
> >>> A: branch in the mozilla-central repository
> >>> B: branch in the mozilla-unified repository
> >>> C: branch in a repository elsewhere
> >>>
> >>> Not to disrupt operations too much I think it would be preferable to
> >>> use option B - create the branch in the mozilla-unified repository. It
> >>> does depend a bit on what plans Mozilla has for these repositories.
> >>>
> >>> Creating the branch in mozilla-unified of course needs buy-in from the
> >>> Mozilla hg people, so please let me hear your opinions.
> >>>
> >>> What about history?
> >>> --------------------------
> >>>
> >>> For the proposed approach (going through `hg convert`) commits are
> >>> preserved, but AFAIK it's not possible to preserve actual commit
> >>> hashes. `hg convert` will graft the original commits, and this adds
> >>> the original hash as an extra field to the new commit (use `hg log
> >>> --debug` to see it). I don't think this is such a big problem. The
> >>> original commits would be linked in the pushlog, the same way there is
> >>> "converted from" for mozilla-central-cvs, like
> >>> https://hg.mozilla.org/experimental/mozilla-central-cvs/rev/ab117d946959.
> >>> I don't see them e.g. in
> >>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/rev/8f8b0933b611
> >>> so perhaps there is some server side feature that needs turning on?
> >>>
> >>> Can I try it?
> >>> ---------------
> >>>
> >>> You can check
> >>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central/ which
> >>> carries the comm/default branch.
> >>>
> >>> In your mozilla tree, pull in the branch like this:
> >>>
> >>>       hg pull -u -b comm/default
> >>> https://hg.mozilla.org/users/mkmelin_iki.fi/thunderbird-central
> >>>
> >>> To completely remove it again, use `hg strip "branch(comm/default)`
> >>>
> >>> How did you do it?
> >>> ------------------------
> >>>
> >>>      CVS_GIT_REPO=~/tmp/mozilla-cvs-history/
> >>>      CC_HG_REPO=~/Code/tb/mozilla/comm/
> >>>
> >>>      git clone https://github.com/ehsan/mozilla-cvs-history.git
> >>> $CVS_GIT_REPO
> >>>
> >>>      echo default comm/default | tee branchmap.txt
> >>>      echo rename . comm | tee filemap.txt cc0.txt
> >>>
> >>>      hg --cwd=$CC_HG_REPO up 0
> >>>      (cd $CC_HG_REPO && ls -d */ | cut -f1 -d'/' | sed 's/^/include /')
> >>>>> cc0.txt
> >>>      hg convert --filemap=cc0.txt --branchmap=branchmap.txt
> >>> $CVS_GIT_REPO single-repo/
> >>>
> >>>      CVS_TIP=`hg id --cwd=single-repo -i -r tip --debug`
> >>>      CC_0=`hg id --cwd=$CC_HG_REPO -i -r 0 --debug`
> >>>      echo $CC_0 $CVS_TIP | tee splicemap.txt
> >>>      # e4f4569d451a5e0d12a6aa33ebd916f979dd8faa
> >>> 93dd94bad983ce2eff5f228719c349b78a8a7f7d
> >>>
> >>>      hg convert --config convert.hg.saverev=True
> >>> --splicemap=splicemap.txt --filemap=filemap.txt
> >>> --branchmap=branchmap.txt $CC_HG_REPO single-repo/
> >>>
> >>> For fun, see all the history down to 1999 is there, like
> >>>
> >>>      hg log comm/mailnews/mime/src/mimei.cpp
> >>>
> >>> Then go on and add the default branch from mozilla-central and do the
> >>> merge
> >>>
> >>>      cd single-repo
> >>>      hg pull -f -b default $CC_HG_REPO/..
> >>>
> >>>      hg up comm/default
> >>>      hg merge default && hg commit -m "Merge default to comm/default
> >>> branch"
> >>>
> >>>      hg push -f -b comm/default --new-branch thunderbird-central-push
> >>>
> >>> ---
> >>>
> >>> Thoughts, comments, feedback appreciated!
> >>>
> >>>   -Magnus
> >>>
> >>> _______________________________________________
> >>> tb-planning mailing list
> >>> [email protected]
> >>> https://mail.mozilla.org/listinfo/tb-planning
> >> _______________________________________________
> >> dev-planning mailing list
> >> [email protected]
> >> https://lists.mozilla.org/listinfo/dev-planning
> > _______________________________________________
> > dev-planning mailing list
> > [email protected]
> > https://lists.mozilla.org/listinfo/dev-planning
> _______________________________________________
> dev-planning mailing list
> [email protected]
> https://lists.mozilla.org/listinfo/dev-planning