Make distclean (still) doesn't

[simonpj]

There are various ways in which make distclean does not remove all the files it should. Please keep adding to this bug description as you find more.

The following things are not currently distcleaned and should be, and hence are ending up in the source distribution:

libraries/haddock.css
libraries/haddock-util.js

[claus]

part of the problem is that distclean shouldn't clean everything, so there should be a reallyclean or darcsclean target doing the full job. but distclean is the strictest currently working clean target (maintainer-clean bails out when descending into the subrepos), so it will be good to record here all the things it doesn't clean, then to make either distclean or darcsclean cover them.

a good way to get an idea of leftover files (usually before a build) would be

make distclean
./darcs-all whatsnew -l --boring

but that will report all the subrepo files when processing the main repo. an alternative is to use a smaller boringfile that excludes just the subrepos and files we know we don't want to clean (mk/build.mk, ghc-<version>):

make distclean
darcs setpref boringfile dont-touch-these.boring
./darcs-all whatsnew -l 2>&1 | tee ../whatsnew-l-all.log
darcs setpref boringfile .darcs-boring

a draft of such a secondary boringfile is attached (note that libraries/time has its own .darcs-boring, is that intended? also, all darcs repos have _darcs/pref/boring files, which will hide left-over .o/.hi files and the like - how can we disable all of them temporarily? for the subrepos, we could use

darcs whatsnew -l --boring

but not for the main repo (see above), and darcs-all uses the same flags for all repos at the moment. it might be best to add a 'list-non-darcs-files' target to darcs-all that takes care of these details. it might also be useful to run this target at the start of 'sh boot', keeping a record of the state of the repo at the beginning of the build.

see also  http://www.haskell.org/pipermail/cvs-ghc/2007-September/038355.html

[claus]

secondary boringfile for finding things that might need cleaning

[simonmar]

On a Unix platform you can make a build tree with lndir, build GHC in it, make distclean, and find the leftover files with

find . ! -type d -a ! -type l

This is rather easier than fiddling around with boringfiles.

[claus]

thanks, but i'm on windows, and

$ darcs changes utils/lndir/lndir.c
Changes to utils/lndir/lndir.c:

Mon Jul 30 23:31:52 GMT Daylight Time 2007  Ian Lynagh <igloo@earth.li>
  * Make lndir work on Windows (by copying)
..

and i rather like the idea of asking darcs which files it doesn't know about. of course, 'make darcsclean' should only clean files that configure or make can regenerate, not files which users added.

which reminds me: current distclean also deletes more files than it should, using wildcards when specifying *_CLEAN_FILES. that way, it once deleted a pre-patch test.log i kept for comparison with post-patch build. so distclean should not only have better coverage, it should also be more accurate.

[claus]

what clean target should be responsible for these files/directories?

./compiler/stage1/
./compiler/stage2/
./compiler/gcc-lib/
./compiler/cmm/CmmLex.hs
./compiler/cmm/CmmParse.hs
./compiler/gcc-lib/
./compiler/gcc-lib/ld.exe
./compiler/main/ParsePkgConf.hs
./compiler/parser/HaddockLex.hs
./compiler/parser/HaddockParse.hs
./compiler/parser/Lexer.hs
./compiler/parser/Parser.hs
./compiler/parser/Parser.y
./compiler/parser/ParserCore.hs
./configure
./driver/ghci/ghci.exe
./extra-gcc-opts -- covered by recent patch?
./gmp/gmp.h
./mk/config.h.in
./stamp.inplace-gcc-lib
./utils/genapply/genapply.exe
./utils/genprimopcode/Lexer.hs
./utils/genprimopcode/Parser.hs
./utils/genprimopcode/genprimopcode.exe
./utils/ghc-pkg/ghc-pkg-inplace.exe
./utils/hasktags/hasktags.exe
./utils/hp2ps/hp2ps.exe
./utils/hpc/HpcParser.hs
./utils/hsc2hs/hsc2hs-inplace.exe
./utils/prof/cgprof/cgprof.exe
./utils/prof/ghcprof-inplace -- covered by recent patch?
./utils/pwd/pwd.exe
./utils/touchy/touchy.exe
./utils/unlit/unlit.exe

i think most of these should disappear during distclean, but which clean target would i use to get rid of the others, or should make regenerate them if it finds that parser/lexer generators have been updated?

[simonmar]

These should be left by distclean:

./compiler/cmm/CmmLex.hs
./compiler/cmm/CmmParse.hs
./compiler/main/ParsePkgConf.hs
./compiler/parser/HaddockLex.hs
./compiler/parser/HaddockParse.hs
./compiler/parser/Lexer.hs
./compiler/parser/Parser.hs
./compiler/parser/Parser.y
./compiler/parser/ParserCore.hs
./configure
./mk/config.h.in
./utils/genprimopcode/Lexer.hs
./utils/genprimopcode/Parser.hs
./utils/hpc/HpcParser.hs

These should be removed by distclean:

./compiler/stage1/
./compiler/stage2/
./compiler/gcc-lib/
./compiler/gcc-lib/ld.exe
./driver/ghci/ghci.exe
./gmp/gmp.h
./stamp.inplace-gcc-lib
./utils/genapply/genapply.exe
./utils/genprimopcode/genprimopcode.exe
./utils/ghc-pkg/ghc-pkg-inplace.exe
./utils/hasktags/hasktags.exe
./utils/hp2ps/hp2ps.exe
./utils/hsc2hs/hsc2hs-inplace.exe
./utils/prof/cgprof/cgprof.exe
./utils/pwd/pwd.exe
./utils/touchy/touchy.exe
./utils/unlit/unlit.exe

[igloo]

Let's do this better, by comparing file lists bofore and after "make; make distclean" in the nightly builds or something.

[igloo]

The problem with make distclean in the nightly builds is that we sometimes want to poke around in them to see what happened. Also, it would be nice if validate checked it too.

Perhaps a better idea would be to have $(RM) and $(RMREC) make variables that, depending no whether we really want to clean, either run rm, or print out what would be removed. Then a script could work out if the tree would be correctly cleaned.

[morabbin]

Bump; is this still an issue?

[simonmar]

Replying to morabbin:

Bump; is this still an issue?

You tell me :-)

[morabbin]

Yes: make distclean gets rid of all of the files you specify for removal above, but also removes these, which you'd like to keep:

./compiler/cmm/CmmLex.hs
./compiler/cmm/CmmParse.hs
./compiler/main/ParsePkgConf.hs
./compiler/parser/HaddockLex.hs
./compiler/parser/HaddockParse.hs
./compiler/parser/Lexer.hs
./compiler/parser/Parser.hs
./compiler/parser/Parser.y
./compiler/parser/ParserCore.hs
./utils/genprimopcode/Lexer.hs
./utils/genprimopcode/Parser.hs
./utils/hpc/HpcParser.hs

all of which are produced by alex or happy, methinks.

[simonmar]

Those files have moved, e.g. compiler/parser/Parser.hs is now compiler/stage1/parser/Parser.hs.

I agree they should not be removed by make distclean though. Or perhaps we are changing the policy, and you need Happy & Alex even when building from a source dist? I guess that would be ok, it's a bit of a pain to arrange to keep these generated files around for a source dist. Ian?