Tag source tree with successful bootstraps

[guest]

Trying to keep up with ghc HEAD can be very frustrating. If you grab the source tree at any point in time there only seems to be about a 10% chance that the compiler can actually successfully bootstrap itself and all the libraries. I suggest that the nightly build should tag the tree when it has successfully bootstrapped everything (for each platform). This way there would be recent and working fall-back points.

Alternatively, make sure that bootstrapping works before pushing the change into the main repo (but given the speed of recompilation this could be painful).

-- Lennart

[claus]

i've been bitten by that, too. but given the time needed for build tests, the buildbots would have to tag the repos before starting their build, and given the variation over platforms, each buildbot would need to do so.. overall, it might be easier for each buildbot to record what versions of the repos it is using, and the commands needed to recreate its build environment, precisely.

a step in the right direction would be to augment the 'Daily report for HEAD' emails. they already summarise how each buildbot fares, by phase. they should also list, for each buildbot, the date/# of the last successfull build (perhaps even with a direct link to the last successful buildbot report, as those are meant to include information on how to reproduce the build environment)

[simonpj]

Maybe just publishng a date-and-time for the last known clean build on each archiecture would do? To save a profusion of tags I mean.

Darcs can check out a repo at a given time, right?

Simon

[igloo]

Tags are a pretty scary prospect. All the builders (or at least, all those we want to do tagging) would need passwordless write access to the repos, and the number of tags would be high in periods when everything is working (at least 3 per day per repo, presumably). I don't think there would be any technical problems with doing it, though.

The summary of my attempt to do something similar with darcs contexts is here:  http://www.haskell.org/pipermail/cvs-ghc/2007-May/035884.html

Dates are a slightly less pleasant way (than contexts) of doing it, but if they don't suffer from the problems I had with contexts then going that route is probably better in the short term.

Making the nightly builders make the contexts or dates of the latest (n) successful build(s) available is easy enough to do, we just need someone to find a way to do something useful with the output (realistically, something that doesn't require doing a get --complete).

[simonmar]

Perhaps the builders could darcs send the tags, and we could publish those tags somewhere, but not actually push them to the repo?

[sorear]

Um, what do you have against darcs get --complete? I've been bitten by enough --partial problems in the past that I just don't do it anymore, and a --complete --extra GHC checkout is pretty small - 60MB and <10 minutes.

[igloo]

To simonmar: Hmm, we could do that, but I'm not sure how we then get the repos we want given we have the tag patches? I don't know if the new lazy patch downloading stuff could be tricked into doing it for us.

To sorear: I'm not sure where you are getting those numbers from; my ghc/_darcs/patches alone is >90M, and according to my e-mail darcs had been going for >2h getting just the ghc and testsuite repos. I don't remember if the get --context had been going slower than a normal get.

(it's also a bit inelegant having to do a complete get when you only actually need to pull a handful of patches)

[simonmar]

Steps to get to a stable repo:

• darcs get; darcs-all get
• download the tags
• darcs apply tags (to sub-repos too)
• darcs unpull --from-tag=... (to sub-repos too)

we could package up these commands as a script, perhaps. This should work with a partial repo, as long as the snapshot was before the tag, right?

[AndyGill]

TO build on SimonM's 05/30/07 13:13:33 comments:

We could also use a staging repo

• At a specific time, every patch is pulled from darcs/ghc to darcs/staging_<uniq>/ghc (same with the related repos for the libraries)
• The buildbots build from the staging versions.
• If a version builds and tests ok, we change the name of the repo

from staging_<uniq> to stable_<uniq>

• We publish a list of the stable repos.

Changes still go back into HEAD.

We definitely do need to overall things a bit; no stable builds and pulling in every patch every time for developers are a huge overhead.

Andy Gill

[igloo]

This is now done, but didn't work for unknown reasons:  http://darcs.haskell.org/buildbot/head/x86%20Windows%20head/builds/140/step-pushchanges/0

Simon Marlow will take a look at what's going on.

[simonmar]

Ran into problems due to partial repositories. We've decided to put this to one side for now:

 http://www.haskell.org/pipermail/cvs-ghc/2007-July/036446.html

[simonmar]

validate has superceded these ideas, the tree now bootstraps 99% of the time.