Cabal Configurations Progress

On this page I (Thomas Schilling) will document some of the design decisions and my progress on implementing Cabal configurations. If you have any questions ping me on #haskell or send a mail to me or, better, the cabal-devel mailing list. My nickname is "nominolo", my email is <my_nickname> at gmail.com.

I am working mostly based on the  latest proposal.

.cabal Syntax

Old cabal files still work, but they do not support configurations features. Thus, if you use the new syntax and features, be nice to others and include the following line, so that they get a more useful error messago than "Syntax error". :)

Cabal-version: >= 1.1.7

Here is an example of a .cabal file that uses configurations.

Name: Test1
Version: 0.0.1
Cabal-Version: >= 1.1.7
License: BSD3
-- License-File: LICENSE
Author: Thomas Schilling <notme@nospam.com>
Maintainer: Thomas Schilling <notme@nospam.com>
--Homepage: http://www.example.com/
Synopsis: Test package to test configurations
Description:
        See synopsis.
        .
        Really.
Category: Useless

Flag Debug {
  Description: Enable debug support
  Default:     False
}

Flag NoBuild {
  Description: Inhibit building this package.
  -- defaults to True
}

Library {
  Build-Depends:   base
  Exposed-Modules: Testing.Test1
  Extensions:      CPP

  if flag(debuG) {
    CC-Options: "-DDEBUG"
    GHC-Options: -DDEBUG
  } 

  if flag(NoBuild) {
    Build-Depends: nonexistentpackage
  }
}

Executable test1e {
  Main-is: T1.hs
  Other-modules: Testing.Test1

  if flag(deBug) {
    CC-Options: "-DDEBUG"
    GHC-Options: -DDEBUG
  } 
}

When configuring it with the usual command line, you now get an additional line, showing which flags were chosen:

$ ./Setup.lhs configure
configure: Reading installed packages...
Configuring Test1-0.0.1...
configure: Flags chosen: nobuild=False, debug=False
Setup.lhs: Warning: No license-file field.
configure: Dependency base-any: using base-2.1.1
  [...]

Note that, even though the default for the "nobuild" flag was True, the required dependencies weren't present, so it was forced to False. If you want to force flags to certain values you can do so by giving the --flags or -f flag to configure. Listing the (case-insensitive) name forces it to True, putting a "-" in front of the name forces it to False. For example:

$ ./Setup.lhs configure -fdebug
configure: Reading installed packages...
Configuring Test1-0.0.1...
configure: Flags chosen: nobuild=False, debug=True
Setup.lhs: Warning: No license-file field.
[...]

or

$ ./Setup.lhs configure --flags="-debug nobuild" 
configure: Reading installed packages...
Configuring Test1-0.0.1...
Setup.lhs: At least the following dependencies are missing:
    nonexistentpackage -any
$

Note that if you want to change the configuration of a package, you have to ./Setup.lhs clean first, to make sure everything gets recompiled.

TODO

• Move the resolved package description into the local build info.
  • Requires changing the workings of the sdist and clean commands.
    • clean should really be just a simple rm -rf ./dist. To get there preprocessed files must never be written outside of ./dist, which requires some changes:
    • capability to resolve a package by just ignoring all conditions.

Progress Log

2007-07-30:

Started refactoring/cleanup branch that will hopefully make Cabal more testable and easier to maintain.

2007-07-29: Configurations are in head

2007-06-24:

Sent all local patches to mailing list for review,

2007-06-14:

Refactorings, also some pretty printing. Interface seems okay now; parseDescription now returns a PreparedPackageDescription? which, given some environmental parameters, can be resolved to an old PackageDescription? (or not, if the dependencies could not be resolved). We probably want a hook for this.

Compatibility mode working.

2007-06-12:

Prototypical implementation up and running. Next step is integrating it with the standart build process. Not sure what the best interface should be.

2007-05-30:

Started modifying the higher levels of the parsing code to incorporate support for conditions. Lot's of cleaning up required. This is when you learn the merits of XML.

2007-05-29:

Not much hacking today; had exam. Parsing and simplifying conditions works.

2007-05-28:

Low-level parsing works. We now have three syntactic categories:

• Normal properties: fieldname: value
• If-blocks: 'if' condition '{' ... '}' ['else' '{' ... '}']
• Sections: sectionname sectionlabel '{' ... '}' where sectionname can currently be only library or executable. Whether or not a label is required is checked at a higher level.

This scheme is backwards compatible. I plan to add a warning when blocks are used but no Cabal-version: > 1.1.7 or equivalent is present.

Discussions

How to make Cabal more maintainable?

(2007-07-29, CEST)

23:34 <  nominolo> dcoutts: i don't know.  a CabalM monad?
23:34 <     Igloo> It needs to either be something conditionals can talk about or to be something that can be build-depend'ed on, though
23:34 <   dcoutts> nominolo: and what would that monad provide ? (I was thinking of logging at least)
23:35 <  nominolo> dcoutts: and the verbosity level, and the output path, and ...
23:35 <   dcoutts> nominolo: verbosity is easy since we use it everywhere, but some of those other vars are only relevant in some places
23:35 <   dcoutts> with different bits of the code using different environments it's tricky
23:35 <  nominolo> dcoutts: i guess i should really try to create a proposal
23:36 <   dcoutts> nominolo: I'm not sure it needs to be that formal, we should just toss around ideas
23:36 <  nominolo> dcoutts: yes, the issue is that a monad would hide, what's used where
23:36 <   dcoutts> for things that are realistic
23:36 <   dcoutts> eg ignoring ndm's suggestion of throwing the whole code out
23:36 <  nominolo> dcoutts: OTOH, this would make the interface more robust to changes
23:37 <   dcoutts> nominolo: I was thinking of a cabal monad too, to abstract out things like logging and indeed the whole UI
23:37 <  nominolo> dcoutts: well. i'd like to at least mask io, so we can have test cases, by using a fake IO
23:37 <   dcoutts> nominolo: yes!
23:37 <   dcoutts> nominolo: I would get rid of IO in the cabal monad
23:37 <   dcoutts> it'd only have requests to an outside interpreter
23:38 <  nominolo> dcoutts: even better, yep
23:38 <   dcoutts> so it'd be pure internally
23:38 <     Igloo> dcoutts: You need to have some logic looking at IO results, to decide what preprocessors to run, for example
23:38 <     Igloo> But it could certainly be more separate than it is now, yes
23:38 <   dcoutts> it's an interpreter
23:38 <  nominolo> yep, some DSL
23:38 <   dcoutts> it's a conversation between what's running in the cabal monad and an external interpreter
23:39 <   dcoutts> that executes the commands on behalf
23:39 <   dcoutts> and decides security
23:39 <   dcoutts> it could decide to allow anything, or eg it could white-list certain programs
23:40 <   dcoutts> and obviously it could do logging
23:40 <   dcoutts> or it could allow step-by-step debugging
23:40 <   dcoutts> or a gui
23:40 <   dcoutts> by implementing different interpreters of the cabal monad
23:41 <  nominolo> dcoutts: whoa, let's start small ;)
23:41 <  nominolo> dcoutts: i'll start a branch
23:41 <   dcoutts> nominolo: yeah, that's my grand plan, but we could probably do smaller cleanups
23:41 <   dcoutts> nominolo: like Igloo recently cleaned up all the verbose stuff, so it's no longer insane

Dependency Tracking (Parallel Builds, less rebuilding)

23:42 <   dcoutts> the other major thing that would really improve cabal is doing a proper dependency graph, & dep tracking
23:42 <  nominolo> dcoutts: do all compilers support this?
23:42 <   dcoutts> it should not depend on the compiler
23:42 <  nominolo> i know ghc does
23:43 <   dcoutts> cabal can read the .hs files and find the deps
23:43 <   dcoutts> nominolo: since we have to do it for non .hs files too
23:43 <   dcoutts> eg .chs files
23:43 <  nominolo> whoa, that's quite heavy though
23:43 <   dcoutts> not really, we steal the .hs reading code from ghc, or yhc or whereever
23:43 <   dcoutts> ot hmake
23:43 <       ndm> i have written an initial make library
23:43 <       ndm> which i think should meet your needs perfectly
23:43 <   dcoutts> and we use something like ndm's make lib for the dep graph
23:43 <  nominolo> ndm: cool
23:44 <   dcoutts> ndm: and partial rebuilds ?
23:44 <       ndm> dcoutts, the works
23:44 <   dcoutts> excelent
23:44 <       ndm> writing a dependency parser should (in my opinion) be done from scratch, from the haskell report
23:44 <       ndm> i can't imagine its more than a few hours, and does make sure you are 100% outside of other implementations
23:45 <     therp> dcoutts: main/HeaderInfo.hs makes use of a direct call to parseHeader to get the module dependencies. maybe just rip out the header directive from parser/Parser.y?
23:45 <     therp> dcoutts: that way it prevents to parse the whole .hs file (that might contain parse errors) and is still able to get the dep. info
23:46 <   dcoutts> right
23:46 <       ndm> i think it can be done in pure Haskell, without parsec/happy etc, without too much hassle
23:46 <   dcoutts> I think ghc uses happy's partial parser feature
23:46 <   dcoutts> to stop parsing after reading the imports
23:46 <   dcoutts> ndm: probably, though imho it wouldn't matter too much if we used happy
23:47 <   dcoutts> anyway it's not a major problem
23:47 <       ndm> dcoutts, perhaps, i was more thinking for simplicities sake, rather than for minimizing deps
23:47 <   dcoutts> sure
23:47 <       ndm> yeah, its entirely well abstracted, making it easier to change
23:47 <   dcoutts> it has to be pluggable anyway
23:47 <       ndm> anyway, i'm off, i'll pimp my make lib to you at anglohaskell
23:47 <       ndm> bye
23:47 <   dcoutts> since we have to support extracting deps from various file types
23:47 <   dcoutts> .hs .chs etc
23:47 <   dcoutts> ndm: ok cool :-)

Hooks Interface

23:52 <  nominolo> dcoutts: i think the hooks interface should return interpreted instructions, too.  this is the only way i can think of to make it sufficiently future proof
23:52 <   dcoutts> nominolo: yes
23:52 <   dcoutts> not IO
23:53 <  nominolo> and more like hook :: Hookargs -> HookResults
23:53 <   dcoutts> though initially we'd probably need to make it an instance of liftIO
23:53 <   dcoutts> erm I mean MonadIO
23:53 <  nominolo> yep, sure
23:53 <   dcoutts> but the interpreter could always refuse to execute arbitrary IO
23:53 <  nominolo> if we get this running we can safely call it Cabal-2.0

2007-07-31

nominolo + syntaxninja on #haskell-soc

19:29 <  nominolo> we missed our meeting again 
19:30 <  nominolo> but we now have cabal configs in head
19:30 <  nominolo> and dcoutts and Igloo are using them for lots of ghc libraries
19:31 <  nominolo> i'm starting to write the doc now and will try to fix cabal-install
19:32 <  nominolo> meanwhile me and duncan, also persue an idea how to make the cabal codebase more maintainable
19:33 <  nominolo> the basic idea is to have a CabalM monad, that requests some IO actions
19:33 <  nominolo> this can then be run using several ways
19:33 <  nominolo> e.g., single-step debugging or test suites
19:35 <  nominolo> the goal is also to get rid of Setup.hs for standard cases.
19:35 <  nominolo> both changes will allow us to eventually have nicer frontends for cabal
19:36 <  nominolo> oh, and i think the next release of Cabal should be called 1.2 rather than 1.1.7, since this is what we test for when writing Cabal files with newer syntax 
19:37 <  nominolo> also, i think configurations are a rather big change
19:37 < SyntaxNin> whew! sounds great.
19:37 < SyntaxNin> I like the idea of making it 1.2 or even higher.
19:38 < SyntaxNin> 2.0?
19:38 < SyntaxNin> configurations are a big change, as is the Setup change.
19:38 <  nominolo> well, i'd like to do 2.0 after the cleanups
19:38 < SyntaxNin> 2.0 would help prepare people for the magnitude of it :)
19:39 <  nominolo> yes, but we'd like to try to do as many incompatible changes at once in that case
19:39 <  nominolo> ie, other hook interface
19:39 < SyntaxNin> I'm dubious about CabalM, but can trust you guys for it.  I fear it would make using cabal more awkward.
19:39 < SyntaxNin> in my experience w/ the speacial ReadIO and WriteIO monads in Halfs, I've found that they are surprisingly difficult for people to get their heads around.
19:40 <  nominolo> right now we have all IO stuff duplicated
19:40 < SyntaxNin> duplicated?
19:40 <  nominolo> or, rather, what core cabal needs
19:40 <  nominolo> http://code.haskell.org/~nominolo/src/cabal-cleanup/Distribution/Types.hs
19:40 < lambdabot> http://tinyurl.com/242xwu
19:40 <  nominolo> that's my start
19:41 <  nominolo> not sure yet, what we do about the hooks
19:41 <  nominolo> ideally, we don't need them most of the time
19:42 <  nominolo> they can stay in IO, for now
19:42 < SyntaxNin> hm. how pragmatic is this? pragmatism is one of cabal's big strengths.  if the code is cleaner, easier to maintain, that's nice, but not at the cost of easy to use.
19:43 <  nominolo> ok, we'll keep that as a guideline
19:43 <  nominolo> or, directive 
19:43              SyntaxNinja isn't in the position to issue directives.
19:44 < SyntaxNin> consider it advice :)
19:44 <  nominolo> ok :)
19:44 < SyntaxNin> how does IO make the code hard to maintain?
19:45 <  nominolo> it's very hard to test
19:45 < SyntaxNin> was it a serious struggle to add configurations (which is a major change)?
19:45 <  nominolo> we actually have no good test suite
19:45 <  nominolo> well, i don't feel as confident as i should
19:46 < SyntaxNin> the test suite tested most of the major features using IO.
19:46 <  nominolo> it works, mostly, but i'm not so confident we won't have some problems with it
19:46 < SyntaxNin> with configurations specifically?
19:47 < SyntaxNin> how does CabalM help with testing? can you help me understand?
19:48 < SyntaxNin> I don't mean to be injecting doubt, just trying to help explore options.
19:49 <  nominolo> instead of doing the actions, we could record them and put them in a test case
19:49 <  nominolo> or, you could log them somewhere
19:50 < SyntaxNin> and then you'd have some kind of oracle to tell you if the test case is correct?
19:51 <  nominolo> well you get something like [Copy "foo" "bar", RmDir "muh", ..] and that's easy to test
19:51 <  nominolo> *to compare
19:51 < SyntaxNin> so the test case would contain the expected list of actions verses the expected result of the actions?
19:52 <  nominolo> well, it doesn't work if the result is produced by some external program of course
19:53 <  nominolo> the test case would run the CabalM, get some list of actions and compare them, yes
19:54 <  nominolo> we can even embed "unknown" IO actions.  the run-function then decides if it runs them or not
19:55 < SyntaxNin> hm. so if you're comparing against the list of actions, how do you verify that those actions produce the correct result? 
19:57 <  nominolo> hm, if it's simple stuff we just trust the os
19:57 <  nominolo> for external programs we can't
19:57 <  nominolo> though we can build up some fake environments
19:58 <  nominolo> and see if cabal calls with the right parameters
19:58 <  nominolo> we cannot test if external tools actually do TRT
19:59 < SyntaxNin> well, I mean that if you have a list of  actions like [Ghc "foo", Cp "bar", Rm "baz" "booz" "bang"] how do you verify that when you run these actions, the resulting build tree is what you expected (that is, dist/build/foo.o dist/build/docs/bar, etc. end up in the right place)
20:00 < SyntaxNin> the test suite I wrote does lots of builds and checks the build tree to make sure that things are in the right place.  all the .o files, etc.
20:01 <  nominolo> hm, ok, then i have a closer look at that
20:05 < SyntaxNin> cool.
20:11 <  nominolo> ok, it's weird.  it builds moduleTests but doesn't run it
20:11 <  nominolo> at least so it seam
20:11 <  nominolo> *seems
20:12 <  nominolo> and the best thing is: it cleans it right after it built it
20:13 <  nominolo> ah, the right target is check
20:14 <  nominolo> yow, 8 errors
20:16 <  nominolo> actually, it's more
20:19 <  nominolo> http://hpaste.org/1996
20:29 <  nominolo> "dcoutts> ok, it's not IO that's the primary problem"
20:29 <  nominolo> "dcoutts> it's the lack of separation between deciding what to do, and doing it"
20:29 <  nominolo> "dcoutts> because increasingly, deciding what to do is influenced by many factors"
20:39 <  nominolo> SyntaxNinja: i'll put up the logs from #ghc for you in about 1h.  i'll be afk, now
20:41 < SyntaxNin> ok thanks.

nominolo, dcoutts, Igloo on #ghc

No tabs in .cabal files

19:40 <   dcoutts> nominolo: bos doesn't like your tab patch, bos is another tab hater like Igloo :-)
19:40 <       bos> i've nothing against tabs, just against their non-portability
19:40 <   dcoutts> bos: I know ;-)
19:40 <       bos> they're fine upstanding characters, if a bit invisible
19:41 <   dcoutts> I'm not sure it's clear here that it'd break things
19:41 <   dcoutts> and I'm not sure that cabal has rejected tabs in the past
19:41 <       bos> perhaps not, but they always phill me with the phear
19:41 <   dcoutts> aye :-)
19:42 <  nominolo> ok, so disallow them?
19:43 <   dcoutts> nominolo: lemme check something...
19:44 <  nominolo> we do have quite a couple of .cabal files that use them ..
19:44 <   dcoutts> nominolo: and Cabal-1.1.6 allowed them, I just checked
19:44 <   dcoutts> nominolo: so somewhere in recent parser changes they got missed
19:45 <   dcoutts> perhaps it's more important now, I'm not sure
19:45 <   dcoutts> though we've always had layout somewhat
19:45 <   dcoutts> exposed-module:
19:45 <   dcoutts>    Foo -- must be indented
19:47 <  nominolo> before, tabs and spaces didn't matter
19:47 <  nominolo> they were all space
19:47 <  nominolo> and *some* space was all that matters
19:47 <    sorear> I'm not a tab hater.  I'm a guy who hates whoever added :set ts to vi.
19:47 <   dcoutts> hmm
19:47 <   dcoutts> nominolo: I see, and now the amount of space matters
19:47 <  nominolo> dcoutts: exactly
19:48 <  nominolo> well not much
19:48 <  nominolo> it just has to be more than the field name
19:48 <   dcoutts> bos: what would you recommend? we've allowed tabs in the past...
19:48 <  nominolo> but if you mix tabs and spaces, then things can get confusing
19:48 <   dcoutts> bos: for reference the important case is:
19:49 <   dcoutts> library {
19:49 <   dcoutts>   field-foo:
19:49 <   dcoutts>     value
19:49 <   dcoutts> value must be indented more than the field
19:49 <   dcoutts> nominolo: does the field have to be indented? I'm guessing not, but if it is the value has to be indented more
19:51 <       bos> dcoutts: if value must be indented more than field, then tabs will surely break things
19:51 <   dcoutts> bos: so do you think on balance we should start rejecting tabs now?
19:51 <       bos> especially if the indentation is relative, rather than absolute
19:51 <  nominolo> it's relative
19:51 <       bos> dcoutts: well, has that syntax change gone into circulation yet?
19:52 <  nominolo> i could disallow indenting field names
19:52 <  nominolo> but that'll look ugly
19:52 <       bos> the indentation is nice to have.
19:52 <  nominolo> if you have if ... { ... {
19:52 <   dcoutts> nominolo: do you think we could reject tabs in the new library { .. stanzas but keep the original parser for the old style field: value  bits? or is that too complex? I got the feeling the parsers were separate anyway, but I may be wrong.
19:52 <       bos> that's what i'd do.
19:53 <   dcoutts> bos: sorry, what were you referring to?
19:54 <       bos> what you just suggested :-)
19:55 <  nominolo> dcoutts: we could replace tabs by spaces if it's old style
19:55 <   dcoutts> bos: ok :-)
19:55 <  nominolo> at least tabs at the beginning of the line
19:55 <  nominolo> the parsers are not separate
19:55 <   dcoutts> nominolo: aye, if you think that doesn't make the code insane
19:56 <   dcoutts> ah ok
19:56 <   dcoutts> I was just hoping not to break existing .cabal files too much
19:56 <  nominolo> the new old one is just a subset of the new one
19:56 <   dcoutts> for new syntax we can be stricter
19:56 <  nominolo> -old
19:59 <  nominolo> ok, i'll try that

Cabal Test suite failing

20:18 <  nominolo> dcoutts: can you run "make check" on the latest cabal?
20:18              dcoutts tries
20:19 <  nominolo> i get: http://hpaste.org/1996
20:21 <   dcoutts> nominolo: I don't get that far even :-)
20:21 <  nominolo> ah right
20:21 <   dcoutts> nominolo: fails compiling tests/ModuleTest.hs here
20:21 <  nominolo> sorry change import HUnit to import Test.HUnit in ModuleTest.hs
20:21 <  nominolo> sorry. forgot about that
20:22              dcoutts makes again
20:23 <   dcoutts> Cases: 12  Tried: 4  Errors: 0  Failures: 0ghc-pkg: cannot find package HUnitTest-1.0
20:24 <   dcoutts> Cases: 28  Tried: 28  Errors: 5  Failures: 0
20:24 <   dcoutts> several errors
20:24 <  nominolo> better than mine
20:25 <  nominolo> which lets me assume these are not all cabal errors (per se)
20:25 <   dcoutts> in the past I've generally ignored the unit tests 'til just before a release
20:25 <   dcoutts> probably not best practise on my part there :-)
20:25 <  nominolo> tsk tsk tsk

Regarding Cabal refactorings

20:26 <  nominolo> isaac is skeptical about our cabal refactorings
20:27 <  nominolo> he likes the simplicity of IO
20:27 <   dcoutts> hah
20:27 <  nominolo> better: practicality
20:28              dcoutts likes IO soup, it's so opaque and homogeneous
20:28 <   dcoutts> ok, it's not IO that's the primary problem
20:28 <  nominolo> yes, but we need a way to test external tools
20:28 <   dcoutts> it's the lack of separation between deciding what to do, and doing it
20:29 <   dcoutts> because increasingly, deciding what to do is influenced by many factors
20:29 <   dcoutts> and at the moment the decision procedures are all mixed together with each other, and also with the actual doing of stuff
20:30 <   dcoutts> things like, if we're using haddock >= 0.8 then it's ok to use cpp with line pragmas
20:30 <  nominolo> but a CabalM monad would help immediately
20:31 <  nominolo> *would _not_
20:31 <  nominolo> afaics
20:31 <   dcoutts> aye, I've been starting today with some other cleanups
20:31 <   dcoutts> like making greater use of the Program abstraction
20:32 <  nominolo> i don't libe the configure function at all
20:32 <   dcoutts> in theory that should also allow things like my haddock example to be done in a more modular way
20:32 <   dcoutts> nominolo: what is wrong with it do you think? (I'm sure there are many things)
20:32 <  nominolo> it's too long
20:33 <  nominolo> it should be more like.  detectCompiler; getInstalledPackages; checkDependencies; ...
20:34 <  nominolo> and if we have different behaviour for different compilers, we should use a type class
20:34 <   dcoutts> or simply a record of functions
20:35 <   dcoutts> like we do for Program and PreProcessor
20:35 <   dcoutts> nominolo: feel free to have a go at some refactorings
20:35 <  nominolo> yep.  but aren't type classes just syntactic sugar for records of functions?
20:35 <  nominolo> (in a way)
20:36 <  nominolo> dcoutts: please push your changes to the cabal-cleanup repo
20:37 <  nominolo> (or head, if you prefer)
20:37 <   dcoutts> nominolo: oh ok. I just pushed one to head
20:37 <   dcoutts> extracting the haddock code into it's own module
20:37 <  nominolo> yes, good move :)
20:38 <   dcoutts> nominolo: I'm off now for a few hours, I might push the Program abstraction cleanups later this evening
20:38 <  nominolo> ok
20:38 <   dcoutts> nominolo: but yes, good idea to have a -cleanups staging area repo