please eliminate error message from Make on every build

[nr]

If I attempt to build a .o file (which is my normal workflow), I always get this error message:

ghc/ghc.mk:90: ghc/stage1/build/.depend: No such file or directory

If this outcome is expected (is it an early-phase thing?), it would be very, very nice to have stderr redirected.

[nr]

It turns out that by going directly to Makefile-stage1 I can avoid this error message and also a powerful lot of futzing around. I probably lose some dependencies but it is worth it for the speed.

I've knocked the severity down as low as possible; if someone wants to mark it wontfix, I won't bleat.

[simonmar]

Makefile-stage1 doesn't exist any more - what GHC version are you building here?

The missing .depend file errors are annoying, but unfortunately not easy to silence. I think I remember coming across a trick somewhere for shutting them up, but can't remember where it was now.

[nr]

I just pulled from the darcs head a few days ago, so I presume I'm building 6.11.

I don't know where Makefile-stage1 comes from---maybe it is a stale version from a while back---but it is very, very useful. Comparing

make -f Makefile dist/stage1/ZipGF.o

and

make -f Makefile-stage1 dist/stage1/ZipGF.o

the second is considerably faster.

I basically am asking for a fast path to attempt to compile a .o file from possibly stale dependencies. I am very, very eager to trade accuracy for speed here. If something strange happens I can always resort to the full Makefile.

[simonpj]

Try

make 1

 http://hackage.haskell.org/trac/ghc/wiki/Building/Using#RebuildingtheGHCbinaryaftermakingchanges

Does that do it? If the above documentation isn't clear enough, or isn't easy enough to find, could you modify it so that it is? It's a wiki. Thanks.

Simon

[nr]

I'm afraid

make 1

suffers from the same problem:

: nr@labrador 3648 [branch=norman] ; make 1
make -C .. stage=1 all_ghc_stage1 compiler_stage1_NO_BUILD_DEPS=YES
make[1]: Entering directory `/usr/local/nr/git/ghc/experimental'
===--- updating makefiles phase 0
make -r --no-print-directory -f ghc.mk phase=0 just-makefiles
===--- updating makefiles phase 1
make -r --no-print-directory -f ghc.mk phase=1 just-makefiles
===--- updating makefiles phase 2
make -r --no-print-directory -f ghc.mk phase=2 just-makefiles
===--- updating makefiles phase 3
make -r --no-print-directory -f ghc.mk phase=3 just-makefiles
ghc/ghc.mk:90: ghc/stage1/build/.depend: No such file or directory
/usr/bin/ghc  -H64m -O0 -fasm  -package-conf libraries/bootstrapping.conf  -package-name ghc-6.11.20090513 -hide-all-packages -i -icompiler/nativeGen -icompiler/basicTypes -icompiler/cmm -icompiler/codeGen -icompiler/coreSyn -icompiler/cprAnalysis -icompiler/deSugar -icompiler/ghci -icompiler/hsSyn -icompiler/iface -icompiler/main -icompiler/parser -icompiler/prelude -icompiler/profiling -icompiler/rename -icompiler/simplCore -icompiler/simplStg -icompiler/specialise -icompiler/stgSyn -icompiler/stranal -icompiler/typecheck -icompiler/types -icompiler/utils -icompiler/vectorise -icompiler/stage1/build -icompiler/stage1/build/autogen -Icompiler/stage1/build -Icompiler/stage1/build/autogen -Icompiler/stage1 -Icompiler/../libraries/base/cbits -Icompiler/../libraries/base/include -Icompiler/. -Icompiler/parser -Icompiler/utils   -optP-include -optPcompiler/stage1/build/autogen/cabal_macros.h -package Cabal-1.7.0 -package array-0.2.0.0 -package base-4.0.0.0 -package bytestring-0.9.1.4 -package containers-0.2.0.0 -package directory-1.0.0.2 -package filepath-1.1.0.1 -package haskell98-1.0.1.0 -package hpc-0.5.0.2 -package old-time-1.0.0.1 -package process-1.0.1.0 -package unix-2.3.1.0  -#include cutils.h -DSTAGE=1 -O -fasm -Wall -fno-warn-name-shadowing -fno-warn-orphans -XCPP -XMagicHash -XUnboxedTuples -XPatternGuards -XForeignFunctionInterface -XEmptyDataDecls -XTypeSynonymInstances -XMultiParamTypeClasses -XFlexibleInstances -XRank2Types -XScopedTypeVariables -XDeriveDataTypeable -XRelaxedPolyRec    -odir compiler/stage1/build -hidir compiler/stage1/build -stubdir compiler/stage1/build -hisuf hi -osuf  o -hcsuf hc -c compiler/codeGen/StgCmmProf.hs -o compiler/stage1/build/StgCmmProf.o

compiler/codeGen/StgCmmProf.hs:350:41:
    `CmmAGraph' is not applied to enough type arguments
    Expected kind `*', but `CmmAGraph' has kind `* -> * -> *'
    In the type `FCode CmmAGraph'
    In the type `CollectedCCs -> FCode CmmAGraph'
    In the type signature for `initCostCentres':
      initCostCentres :: CollectedCCs -> FCode CmmAGraph

compiler/codeGen/StgCmmProf.hs:417:30:
    `CmmAGraph' is not applied to enough type arguments
    Expected kind `?', but `CmmAGraph' has kind `* -> * -> *'
    In the type `CostCentre -> CmmAGraph'
    In the type signature for `mkRegisterCC':
      mkRegisterCC :: CostCentre -> CmmAGraph

compiler/codeGen/StgCmmProf.hs:435:36:
    `CmmAGraph' is not applied to enough type arguments
    Expected kind `?', but `CmmAGraph' has kind `* -> * -> *'
    In the type `CostCentreStack -> CmmAGraph'
    In the type signature for `mkRegisterCCS':
      mkRegisterCCS :: CostCentreStack -> CmmAGraph

compiler/codeGen/StgCmmProf.hs:479:27:
    `CmmAGraph' is not applied to enough type arguments
    Expected kind `?', but `CmmAGraph' has kind `* -> * -> *'
    In the type `CmmExpr -> CmmAGraph'
    In the type signature for `bumpSccCount':
      bumpSccCount :: CmmExpr -> CmmAGraph
make[2]: *** [compiler/stage1/build/StgCmmProf.o] Error 1
make[1]: *** [all_ghc_stage1] Error 2
make[1]: Leaving directory `/usr/local/nr/git/ghc/experimental'
make: *** [1] Error 2

I may have a go at the doco once I understand a little better what's going on.

[simonmar]

Norman - welcome to the world of non-recursive make :-) Even when you say make in compiler/, the build system is bringing every dependency of GHC up to date in order to start building GHC itself. Of course this might well be overkill, but the design principle was "safety first": we want to ensure that the tree never ends up in an inconsistent state, at least unless the user explicitly says they want to play fast and loose.

What we should do now, having established the safe but slow path, is to identify exactly the places where we want to omit dependencies in order to make development a little less painful. So thanks for starting the discussion.

make 1 is an example of one such target: it prevents the dependencies of GHC being rebuilt. I think what you're additionally worried about is the "updating makefiles..." stages, which are also about consistency. If you want, you can cheat and omit these too; e.g. at the top level

$ make -f ghc.mk all_ghc_stage1 compiler_stage1_NO_BUILD_DEPS=YES

this should be the fastest way to rebuild stage1. Perhaps this is something we should enshrine as a rule, or perhaps this should be what happens when you say make 1? I have some concerns about what could happen if you have touched something else in the tree - weird and wacky failures could ensue because we're not bringing the included makefiles up to date in the right order, but hopefully this wouldn't permanently kill your tree, and a top-level make should be able to recover. Let me know your experiences anyway.

[nr]

Two comments:

• Regarding the speed of the build, I think it is really worth exploring Glenn Fowler's nmake. nmake is engineered by real engineers to be portable and to work at scale. It is entirely possible that it might make the safe path also fast. It could even make it possible (although time-consuming and tedious) to eliminate the separate 'configure' step. What do you think of this as a possible student project?

• Having established the safe but slow path, is there a way to make it less noisy? In particular, is there a way to change ghc/ghc.mk so that it does not bleat if the dependency file is missing? I have upgraded severity to 'minor' (from 'trivial') because this error message interrupts my workflow in emacs (C-x ` now has to stop at an extra error message... sometimes).

[simonmar]

Replying to nr:

Two comments: * Regarding the speed of the build, I think it is really worth exploring Glenn Fowler's nmake. nmake is engineered by real engineers to be portable and to work at scale. It is entirely possible that it might make the safe path also fast. It could even make it possible (although time-consuming and tedious) to eliminate the separate 'configure' step. What do you think of this as a possible student project?

The speed of make is not the limiting factor. IIRC, at the end of a complete GHC build the single make process has used only a couple of seconds of CPU time.

nmake may well be better than GNU make (though it's not clear that it does everything we need). Anyway I don't have the energy to rewrite the build system again right now - maybe in a few years!

The real problem we would like to solve is the way GNU make handles rebuilding included makefiles, which leads to the need for the phase ordering. This adds 4 seconds or so to each rebuild, not to mention complexity and fragility in the build system. Now, two of those phases are only needed because GHCs before 6.11 didn't generate makefile dependencies between packages in the way the new build system needs, so in the future when bootstrapping with 6.11+ we can drop the two slowest phases, which should help a lot.

For now I've made make 1 and make 2 omit phases 1-3. It's much quicker for me now.

* Having established the safe but slow path, is there a way to make it less noisy? In particular, is there a way to change ghc/ghc.mk so that it does not bleat if the dependency file is missing? I have upgraded severity to 'minor' (from 'trivial') because this error message interrupts my workflow in emacs (C-x ` now has to stop at an extra error message... sometimes).

I agree the message is annoying, but I can't see an easy way to quiet it short of filtering the output of make (which is possible, but exceedingly ugly). Surely most of the time the dependency file will be there anyway?

[simonmar]

Tue May 19 01:33:50 PDT 2009  Simon Marlow <marlowsd@gmail.com>
  * allow phases to be omitted by setting OMIT_PHASE_[123]=YES

Tue May 19 01:34:19 PDT 2009  Simon Marlow <marlowsd@gmail.com>
  * make [123] omits phases 1,2, and 3

[nr]

Replying to simonmar:

I certainly didn't mean to suggest that you even consider another attack on the build system! I really was thinking about a student here :-)

I'm interested in nmake exactly because relative to gmake or standard make, nmake definitely requires fewer phase distinctions, and it may eliminate phase distinctions altogether. No phase distinctions -> no phase ordering -> speed. But it sounds like the hacks you've put in will help a lot.

Surely most of the time the dependency file will be there anyway?

Actually no. The problem is that I'm propagating a change in the representation of low-level intermediate code, and it will be a very long time before all the modules in the compiler are once again mutually consistent. And apparently as long as any module does not compile, GHC cannot generate the dependency file. I hope I am wrong about this and am just doing something stupid. Please say I am?

If I'm not doing something stupid, can you suggest where I might go to filter the output of make in my own private world?

[simonmar]

Replying to nr:

Replying to simonmar: Surely most of the time the dependency file will be there anyway? Actually no. The problem is that I'm propagating a change in the representation of low-level intermediate code, and it will be a very long time before all the modules in the compiler are once again mutually consistent. And apparently as long as any module does not compile, GHC cannot generate the dependency file. I hope I am wrong about this and am just doing something stupid. Please say I am?

If I touch a source file in the compiler and then say make 1 (in compiler/), I don't see the error message. Even if I just say make in compiler, which rebuilds the dependencies if anything is out of date, I still don't see the error message.

If you start from a position where you don't have the dependency file, then you do need to get to a state where all the imports exist in order to build the dependencies (the dependency-generator parses each source file up the end of the imports, so errors further down won't affect it).

If I'm not doing something stupid, can you suggest where I might go to filter the output of make in my own private world?

In the top-level Makefile, something like this:

	@echo "===--- finished updating makefiles"
	$(MAKE) -r --no-print-directory -f ghc.mk $@  | sed 'd/.depend: No such file or directory/'

[igloo]

I don't think there's a cure for this that isn't worse than the disease, so I'm closing this as wont-fix.