Using Mozilla trees more smartly

A month ago I got a new laptop, requiring me to migrate my Mozilla trees, patches, and related work from old laptop to new. My previous setup was the simplest, stupidest thing that could work: individual clones of different trees, no sharing among those trees, sometimes multiple clones of the same tree for substantial, independent patchwork I didn’t want to explicitly order. Others have tried smarter tricks in the past, and I decided to upgrade my setup.

A new setup

The new setup is essentially this:

• I have one local clone of mozilla-inbound in ~/moz/.clean-base which I never develop against or build against, and never modify except by updating it.
• Whenever I want a mozilla-inbound tree, I clone ~/moz/.clean-base. I change the default-push entry in the new clone to point to the original mozilla-inbound. (I don’t change the default entry; pulling is entirely local.)
• If I want to push a patch, I pull and update ~/moz/.clean-base. Then I pull and update the local clone that has the patch I want to push. Then I finish my patch and push it. Because default-push points to the remote mozilla-inbound, hg push as usual does exactly what I want.

Advantages

This setup has many advantages:

• Getting a new mozilla-inbound tree is fast. I never clone the remote mozilla-inbound tree, because I have it locally. It’s not modified by a patch queue where I’d have to temporarily checkpoint work, pop to clone, then reapply after.
• Updating a working mozilla-inbound tree is fast. Pulling and updating are completely local with no network delay.
• I only need to update from the remote mozilla-inbound once for new changes to be available for all local trees. Instead of separately updating my SpiderMonkey shell tree, updating my browser tree, and updating any other trees I’m using, at substantial cost in time, one pull in ~/moz/.clean-base benefits all trees.
• My working trees substantially share storage with ~/moz/.clean-base.

Pitfalls, and workarounds

Of course any setup has down sides. I’ve noticed these so far:

• Updating a working trees is a two-step process: first updating ~/moz/.clean-base, then updating the actual tree.
• I’ll almost always lose a push race to mozilla-inbound. If my local working tree is perfectly up-to-date with my ~/moz/.clean-base, that’s generally not up-to-date with the remote tree, particularly as rebasing my patches is now a two-step process. That produces a larger window of time for others to push things after I’ve updated my clean tree but before I’ve rebased my working tree.
• I have to remember to edit the default-push in new trees, lest I accidentally mutate ~/moz/.clean-base.

Some of these problems are indeed annoying, but I’ve found substantial workarounds for them such that I no longer consider them limitations.

Automate updating ~/moz/.clean-base

Updating is only a two-step process if I update ~/moz/.clean-base manually, but it’s easy to automate this with a cronjob. With frequent updates ~/moz/.clean-base is all but identical to the canonical mozilla-inbound. And by making updates automatic, I also lose push races much less frequently (particularly if I rebase and push right after a regular update).

I’ve added this line to my crontab using crontab -e to update ~/moz/.clean-base every twenty minutes from 07:00-01:00 every day but Sunday (this being when I might want an up-to-date tree):

*/20 00-01,07-23 * * 1-6 /home/jwalden/moz/inflight/pull-updated-inbound >/dev/null 2>&1

I perform the update in a script, piping all output to /dev/null so that cron won’t mail me the output after every update. It seems better to have a simpler crontab entry, so I put the actual commands in /home/jwalden/moz/inflight/pull-updated-inbound:

#!/bin/bash

cd ~/moz/.clean-base/
hg pull -u

With these changes in place, updating a working tree costs only the time required to rebase it: network delay doesn’t exist. And the intermediate tree doesn’t intrude on my normal workflow.

Add a hook to ~/moz/.clean-base to prevent inadvertent pushes

My setup depends on ~/moz/.clean-base being clean. Local changes or commits will break automatic updates and might corrupt my working trees. I want ~/moz/.clean-base to only change through pulls.

I can enforce this using a Mercurial prechangegroup hook. This hook, run when a repository is about to accept a group of changes, can gate changes before they’re added to a tree. I use such a hook to prevent any changes except by a push by adding these lines to ~/moz/.clean-base/.hg/hgrc:

# Prevent pushing into local mozilla-inbound clone: only push after changing a clone's default-push.
[hooks]
prechangegroup.prevent_pushes = python:prevent_pushes.prevent_pushes.hook

This invokes the hook function in prevent_pushes.py:

#!/usr/bin/python

def hook(ui, repo, **kwargs):
  source = kwargs['source']
  if source != 'pull':
    print "Changes pushed into non-writable repository!  Only pulls permitted."
    return 1
  print "Updating pristine mozilla-inbound copy..."
  return 0

On my Fedora-based system, I place this file in /usr/lib/python2.7/site-packages/prevent_pushes/ beside an empty __init__.py. Mercurial will find it and invoke the hook whenever ~/moz/.clean-base receives changesets.

Only pushing from a new clone without a default-push would attempt to modify ~/moz/.clean-base, so the need to prevent changes to ~/moz/.clean-base might seem small. Yet so far this hook has prevented such changes more than once when I’ve forgotten to set a default-push, and I expect it will again.

Conclusion

There are doubtless many good ways to organize Mozilla work. I find this system works well for me, and I hope this description of it provides ideas for others to incorporate into their own setups.

Earlier today I made one of the most, er, special commits to Mozilla code that I’ve ever made — entirely due to the commit message I used:

Bug 466905 - Fix JSOP_NEWARRAY to be not-buggy and use it when possible. NOT REVIEWED YET

If you read bug 466905 you’ll see in comment 20 that the patch was reviewed by Brendan and was committed with the requested changes. So why is “NOT REVIEWED YET” in there? It’s an artifact of how I manage patches-in-progress; I assign the description when I create the patch, and since I can’t know whose review will eventually grace the patch, I just add a note that I’ll see when I review the change just before commit, and I make sure to fix the description immediately before pushing it into the main repository. What happened here is that I forgot to review the commit message for proper reviewed-ness.

<firebot> Check-in: http://hg.mozilla.org/tracemonkey/rev/6475993319c4 - Jeff Walden - Bug 466905 - Fix JSOP_NEWARRAY to be not-buggy and use it when possible. NOT REVIEWED YET
<Waldo> aargh
<shaver> sounds like someone needs a pre-push local hook!
<Waldo> quite possibly!

There’s a clear flaw in this process: the bad-commit-message check (and remembering to do it!) is done manually and can easily be forgotten. So, at shaver‘s suggestion, I dove into the world of Mercurial hooks. After a little reading from chapter 10 and section 11.3, I present you with ensure-not-unreviewed:

To use, simply drop that somewhere on your system, chmod +x it, and copy the following lines into ~/.hgrc (.hg/hgrc if you want this configurable on a per-repository basis):

[hooks]
preoutgoing.ensure_not_unreviewed = /path/where/you/downloaded/ensure-not-unreviewed

That should take care of the problem of buggy descriptions permanently. Now if only I could write a preoutgoing hook to prevent pushing buggy patches…