Beating my head against the threaded runtime some more. I can reproduce
one of the hangs consistently by running 1000 git annex add commands
in a loop. It hangs around 1% of the time, reading from `git cat-file`.

Interestingly, `git cat-file` is not yet running at this point -- 
git-annex has forked a child process, but the child has not yet exec'd it.
Stracing the child git-annex, I see it stuck in a futex. Adding tracing,
I see the child never manages to run any code at all.

This really looks like the problem is once again in MissingH, which uses
`forkProcess`. Which happens to come with a big warning about being very
unsafe, in very subtle ways. Looking at the C code that the newer `process`
library uses when sparning a pipe to a process, it messes around with lots of
things; blocking signals, stopping a timer, etc. Hundreds of lines of C
code to safely start a child process, all doing things that MissingH omits.

That's the second time I've seemingly isolated a hang in the GHC threaded
runtime to MissingH.

And so I've started converting git-annex to use the new `process` library,
for running all its external commands. John Goerzen had mentioned `process`
to me once before when I found a nasty bug in MissingH, as the cool new
thing that would probably eliminate the `System.Cmd.Utils` part of MissingH,
but I'd not otherwise heard much about it. (It also seems to have the
benefit of supporting Windows.)

This is a big change and it's early days, but each time I see a hang, I'm
converting the code to use `process`, and so far the hangs have just gone
away when I do that.

---

Hours later... I've converted *all* of git-annex to use `process`.

In the er, process, the `--debug` switch stopped printing all the commands
it runs. I may try to restore that later.

I've not tested everything, but the test suite passes, even when
using the threaded runtime. **MILESTONE**

Looking forward to getting out of these weeds and back to useful work..

---

Hours later yet.... The `assistant` branch in git now uses the threaded
runtime. It works beautifully, using proper threads to run file transfers
in.

That should fix the problem I was seeing on OSX yesterday. Too tired to
test it now.

--

Amazingly, all the assistant's own dozen or so threads and thread
synch variables etc all work great under the threaded runtime. I had
assumed I'd see yet more concurrency problems there when switching to it,
but it all looks good. (Or whatever problems there are are subtle ones?)

I'm very relieved. The threaded logjam is broken! I had been getting
increasingly worried that not having the threaded runtime available would
make it very difficult to make the assistant perform really well, and cause
problems with the webapp, perhaps preventing me from using Yesod.

Now it looks like smooth sailing ahead. Still some hard problems, but
it feels like with inotify and kqueue and the threaded runtime all
dealt with, the really hard infrastructure-level problems are behind me.