Changes between Version 15 and Version 16 of Design/BuildSystem

Timestamp:

11/21/08 02:50:40 (5 years ago)

Author:

simonpj

Comment:

--

Design/BuildSystem

@@ -111 +111 @@
    package metadata or the code that processes it, only the build rules. 
 
-== Detailed plan == 
-
- * Rename cabal-bin.hs to ghc-cabal.hs, and move it into utils/ghc-cabal/
-
- * The version of Cabal used by ghc-cabal.hs does not need to be an
-   up-to-the-minute bleeding-edge version. It should be stable and vary
-   slowly. We suck a new version of Cabal into the GHC build system
-   manually, rather than mirroring the Cabal HEAD. 
-
- * Rather than installing things in-place all over the build tree, we
-   will have a single inplace directory at the root of the tree. The
-   structure inside this directory will match that of the normal install,
-   which will simplify various things. There are two slight wrinkles:
-   * The tree will not be complete; for example, the libraries will be
-     registered in-place in their dist directories
-   * Rather than inplace/bin/ghc, we will have inplace/bin/ghc-stage[123]
-   Tools like genprimopcode, genapply etc. will probably also go into
-   inplace/bin, in order to make the makefiles more consistent.
-
- * The build order looks something like:
+== Avoiding recursive makefiles ==
+
+Currently, with recursive make, this means we jump around between
+Makefiles a lot, which isn't good for parallelism in the build.  For example, the current build order looks like this:
      * With bootstrapping compiler:
        * Build libraries/{filepath,Cabal}
@@ -148 +132 @@
        * libraries/*
        * compiler
-   Currently, with recursive make, this means we jump around between
-   Makefiles a lot, which isn't good for parallelism in the build.
-   Instead, we want to move all the logic and dependencies into the root
-   Makefile (or files that get included into it) so that make sees all of
-   it together.
-
-   One concern is that make may take a long time thinking if it can see
-   the rules for the whole system, even when only asked to build a single
-   file. We will have to see how well it performs in practice.
-
- * But we still want "make" to work in subdirectories, so for example the
-   Makefile (actually GNUmakefile, to avoid colliding with Makefile in
-   libraries like Cabal) in libraries/base might look like
+
+Instead, following [http://miller.emu.id.au/pmiller/books/rmch/?ref=DDiyet.Com Recursive make considered harmful], we want to move all the logic and dependencies into the root
+Makefile (or files that get included into it) so that make sees all of
+it together.  Advantages:
+
+   * We get to specify dependencies between different parts of the tree much
+     more easily and precisely.  BIG WIN.  Have you noticed how often you
+     need to `make distclean` in a GHC tree to make sure everything is up to
+     date? (well I rarely do this, because I have the dependencies in my
+     head and I can rebuild manually, but I imagine this isn't the case for
+     most people!)
+
+   * We get more parallelism, more easily.  I'd argue this is a big win too,
+     right now a validate only uses about 1.3 out of 2 cores.
+
+We don't lose modularity: different parts of the build system are still in
+different files, it's just that make sees them all at once.  Right now
+every make invocation already reads thousands of lines of boilerplate
+Makefile code, and we'd be invoking make only once rather than many times.
+
+So we will probably have to worry about efficiency.  For example, it takes
+tens of seconds on Windows for make to discover that compiler/ is up to
+date.  We don't want that happening every time you rebuild some small part
+of the tree, so we plan to cut a few dependencies on purpose.  But I'm
+hoping it'll be necessary to do this in a few well-defined places only, and
+only when building in subdirectories.  For example, If you say 'make' in
+libraries/base, then we won't try to rebuild the stage1 compiler, we'll
+just fail if it does't exist.
+
+
+We still want "make" to work in subdirectories, so for example the
+Makefile (actually GNUmakefile, to avoid colliding with Makefile in
+libraries like Cabal) in libraries/base might look like
 
 {{{
@@ -174 +178 @@
 }}}
 
-   (ghc.mk is discussed later). In actual fact, GNUmakefile will want to
-   be more complicated, to handle "make way=v", "make way=p", "make doc",
-   etc. Where possible, the make code will be "include"d in, rather than
-   generated, so as to make it easier to deal with.
-
-   We need the .NOTPARALLEL or if you say "make foo bar" then the two
-   recursive make calls might both make "quux" (a dependency of foo and
-   bar) at the same time. The main Makefile will be able to do work in
-   parallel when building each of foo and bar, though. The common case,
-   where you only specify 0 or 1 targets, doesn't lose any parallelism.
+(ghc.mk is discussed later). In actual fact, GNUmakefile will want to
+be more complicated, to handle "make way=v", "make way=p", "make doc",
+etc. Where possible, the make code will be "include"d in, rather than
+generated, so as to make it easier to deal with.
+
+We need the .NOTPARALLEL or if you say "make foo bar" then the two
+recursive make calls might both make "quux" (a dependency of foo and
+bar) at the same time. The main Makefile will be able to do work in
+parallel when building each of foo and bar, though. The common case,
+where you only specify 0 or 1 targets, doesn't lose any parallelism.
+
+
+== Detailed plan == 
+
+ * Rename cabal-bin.hs to ghc-cabal.hs, and move it into utils/ghc-cabal/
+
+ * The version of Cabal used by ghc-cabal.hs does not need to be an
+   up-to-the-minute bleeding-edge version. It should be stable and vary
+   slowly. We suck a new version of Cabal into the GHC build system
+   manually, rather than mirroring the Cabal HEAD. 
+
+ * Rather than installing things in-place all over the build tree, we
+   will have a single inplace directory at the root of the tree. The
+   structure inside this directory will match that of the normal install,
+   which will simplify various things. There are two slight wrinkles:
+   * The tree will not be complete; for example, the libraries will be
+     registered in-place in their dist directories
+   * Rather than inplace/bin/ghc, we will have inplace/bin/ghc-stage[123]
+   Tools like genprimopcode, genapply etc. will probably also go into
+   inplace/bin, in order to make the makefiles more consistent.
+
 
  * In e.g. utils/ghc-pkg, the default target will be