Re: REST-API: Merging/Rebasing with Mercurial?

Hi John!

* John P. Rouillard <[email protected]> [20190122 23:49]:
> I have a few questions about your process so I can understand what you
> are doing.
> 
> In message <[email protected]>,
> Thomas Arendsen Hein writes:
> >* Ralf Schlatterbeck <[email protected]> [20190118 16:24]:
> >> Can someone outline how to achieve this with mercurial?
> >
> >I assume you only want the changes from the REST branch (with the
> >CRLF fixed), but not the changes from the bugs.python.org branch?
> >
> >I just did this using the mq (Mercurial Queues) extension:
> >
> ># pull bpo+rest changes into my existing clone of the roundup repo:
> >cd roundup.hg
> >hg pull https://bitbucket.org/kinggreedy1991/roundup-bpo
> 
> Does the roundup-bpo repo have to have been a clone of the roundup hg
> repo in order for this to work? So that way trunk of roundup-bpo will
> match the roundhup hg repo up to a certain revision.

No, but that makes it much easier. See below.

> ># mark pulled changes as yet unpublished to allow importing them into
> ># a stack of mq patches for further modification:
> >hg phase --force --draft -r 'outgoing()'
> 
> What does this bit of magic do? I assume it sets the stage for the
> following qimport in some way?

From "hg help phases":

|   Phases are a system for tracking which changesets have been or should be
|   shared. This helps prevent common mistakes when modifying history (for
|   instance, with the mq or rebase extensions).
|
|   Each changeset in a repository is in one of the following phases:
|
|    - public : changeset is visible on a public server
|    - draft : changeset is not yet published
|    - secret : changeset should not be pushed, pulled, or cloned

From "hg help revisions":

|   "outgoing([path])"
|     Changesets not found in the specified destination repository, or the
|     default push location.

It marks all changes I have pulled from roundup-bpo that are not
included in the main roundup repo as "draft", so I can edit them
with mq. The --force is needed as a safe guard, because these
changes have been available in the public and users of the
roundup-bpo repo now manually need to strip their REST branch to
prevent duplication of changesets if they pull from roundup.

There are more modern approaches using the evolve extension, that
allow safer rebasing and history editing, which will add markers
like "this changeset has been replaced by that changeset", see
https://www.mercurial-scm.org/doc/evolution/, but until this is
enabled by default in Mercurial's core, I still prefer mq.

> >hg qimport -r 'branch("REST")'
> 
> This I assume takes the rest branch from roundup-bpo and creates a
> patch from it that mq can apply?

Slightly more: It converts all changesets from the REST branch to
multiple patches, that are now stacked on top the regular repo.

> ># pop the stack of REST patches and remove (strip) the bpo changes:
> >hg qpop -a
> 
> The pop applies the patches created in the qimport to my current
> working version right?

No, the opposite: It removes (un-applies) the stack of patches from
the repo, so they are just a bunch of patch files in .hg/patches/

> >hg strip -r 'outgoing()'
> 
> How does this limit the strip to the revisions that are identical to
> the versions on the bpo branch?

See above, 'outgoing()' specifies all changesets that are in the
current repo (cloned from roundup or pulled from roundup-bpo), that
are not in the repo that you cloned from (the default repo).

Since all non-REST changesets in the roundup-bpo repo are on the
branch "bugs.python.org", the following command would have done the
same:

  hg strip -r 'branch("bugs.python.org")'

> ># fix CRLF and two references to run_cgi_outer, which is
> ># a bpo-only wrapper around run_cgi:
> >sed -i 's/\r$//' .hg/patches/*
> >sed -i 's/run_cgi_outer$/run_cgi/' .hg/patches/*
> 
> I assume these are left around because they are ignore by the strip
> because they aren't identical to the bpo branch?

No:
- The changesets (which are now plain patch files) in the REST
  branch contained some CRLF line endings, because earlier commits
  in the roundup-bpo repo (on the bugs.python.org branch) changed
  some files from LF to CRLF.
- The bugs.python.org branch changed something like "foo = run_cgi"
  to "foo = run_cgi_outer" (and defined run_cgi_outer somewhere
  else), and the REST branch changed this line, so the patch had to
  be changed from
    -    foo = run_cgi_outer
    +    foo = bar = run_cgi_outer
  to
    -    foo = run_cgi
    +    foo = bar = run_cgi
  so it applies to the non-bpo code.

The popped patches are just plain patch files, so you can edit them
as you like.

What I really did was to use "hg qpush -a" (see below), which aborts
at every patch that does not apply, so I know what to edit.
You don't have to edit the patches directly, but due to the nature
of the changes this was the easiest for me.

> ># the patch queue should still be on the REST branch:
> >hg branch REST
> 
> Does the rename the current working files to a new REST branch created
> in the roundup repo? So after I commit, I could push it to roundup at
> sourceforge and create a new REST branch?

Yes, this says "if you commit now (or qpush patches), these
changesets will be on the REST branch".

To push the new branch to the sourceforge repo, you will need
"hg push --new-branch", as a regular "hg push" should tell you.
This is just another safe guard to avoid pushing branches to a
public repo that you did not want to publish yet.

> ># apply all patches and mark them as finished, so they become
> ># regular Mercurial changesets again:
> >hg qpush -a
> 
> What is this pushing?

All (-a is short for --all) patches from .hg/patches/ in the order
specified in the plain text file .hg/patches/series will be applied
to the repo. The terminology is taken from quilt, which manages a
stack/queue of patches.

qpop (queue-push) will pop from the top of the patch queue and qpush
(queue-pop) will push patches on top of the repo.

> >hg qfinish -a
> 
> This commits the patches as a changeset right?

Basically yes, but technically no :)

qpushed patches are technically already changesets floating on top
of the regular changesets. So this just discards the floating
status (and removes the patch file), so the changesets will be
handled as regular changesets by Mercurial.

This is the inverse operation of qimport, which just marks existing
changesets as "floating" (and creates the patch file).

> Thanks for any explanations you can provide to help me understand how
> this is done.

The mq (Mercurial Queues) extension is very powerful, but unless you
already know Quilt it can be a bit confusing. And unlike Mercurial
core, the mq extension allows to shoot yourself in the foot :)

Here are two guides, which explain mq usage and concepts a bit more:
http://stevelosh.com/blog/2010/08/a-git-users-guide-to-mercurial-queues/
https://www.mercurial-scm.org/wiki/MqTutorial


And now to the "See below" part of your first question:

> Does the roundup-bpo repo have to have been a clone of the roundup hg
> repo in order for this to work? So that way trunk of roundup-bpo will
> match the roundhup hg repo up to a certain revision.

The workflow could be:
- clone the roundup-bpo repo and the regular roundup repo
- qimport and qpop the REST changes
- copy roundup-bpo/.hg/patches to roundup/.hg/patches/
- edit the patches
- hg qpush -a  and  hg qfinish -a  (as above)

Alternatively you don't have to use the mq extension, but just use
  hg export -o REST-%n.patch -r 'branch("REST")'
to create the patches, then edit them, then apply them to the
regular roundup repo using "hg import". But with this approach you
have to make sure to do it in the correct order and take care of
patches that fail to apply yourself.

A somewhat dirty approach, which works perfectly fine, would be to
pull the unrelated repo into the regular repo, qimport the REST
changes, strip EVERYTHING else from the unrelated repo, then
continue as above. But since you usually do not want to do that, hg
pull needs --force to do that.

This works, because every Mercurial repo has a common ancestor:
The empty repo, see "hg log -r null" (to see this empty changeset)
or "hg log -r 0 --debug" (to see that the first changeset has
"-1:0000000000000000000000000000000000000000" as its parent.

Regards,
Thomas

-- 
Thomas Arendsen Hein <[email protected]> - OpenPGP key: 0x5BB3F5195816791A
https://blogs.intevation.de/thomas/         - https://intevation.de/~thomas/
Intevation GmbH, Neuer Graben 17, 49074 Osnabrueck - AG Osnabrueck, HR B 18998
Geschaeftsfuehrer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner