setup sdist places implementation-dependent preprocessor output in the source bundle

[ross@…]

e.g. happy -agc output in the haskell-src package.

Suggested new scheme:  http://www.haskell.org/pipermail/cabal-devel/2007-May/000529.html.

[ross@…]

see also #92: issues with 'clean'.

[igloo]

It also includes files generated by "Setup configure", e.g. HsGLUT.h in the GLUT package.

[duncan]

Replying to igloo:

It also includes files generated by "Setup configure", e.g. HsGLUT.h in the GLUT package.

Which is because Cabal has no idea that HsGLUT.h is the result of HsGLUT.h.in.

[ross@…]

Replying to duncan:

Replying to igloo:

It also includes files generated by "Setup configure", e.g. HsGLUT.h in the GLUT package.

Which is because Cabal has no idea that HsGLUT.h is the result of HsGLUT.h.in.

Maybe #152 wasn't a bug.

[duncan]

Or maybe we should assume that foo is generated (by an unknown pre-processor) by foo.in (if foo.in exists) and so it is foo.in that should go in the sdist and not foo.

Though that also violates our rule about putting generated files in dist/ and not back in the source tree.

[igloo]

Incidentally, include/HsGLUT.h is in extra-tmp-files; should Cabal look at that when deciding what to put in the sdist?

[dankna]

Attached, please find three patches for Cabal and one for Cabal-Install, in "darcs send" format. Also find a change to Happy, in the form of an entire Setup.lhs file to replace the corresponding file in Happy-1.18.4 (I guess I must not have been able to find the original repo for Happy - I can prepare a diff if you find that easier). These implement the stuff we talked about with regard to preprocessor output. Hopefully you'll like the way I did it. I'm afraid it's only easy for me to test with ghc, but it /should/ support all compilers, since I did make appropriate changes to all of their backends. If this is a problem I can try to get the others set up, but I hoped you might already be set for that and able to do it more quickly than I could.

There are three main pieces to the patch, as the commit messages will explain. Briefly, the first piece simply redirects the output, the second piece makes sure the output is included in the sdist tarball, and the third piece adds a user hook to implement the subdirectory-naming policy we discussed.

With the redirection of the output, I found that by doing it at the proper layer it wasn't necessary to add any new code to compare timestamps; the existing codepath appeared to handle it. This could use more testing; I used Happy as my test case.

With the inclusion in sdist, I added a new parameter to prepareTree, which necessitated the change to Cabal-Install. As I explained in a comment somewhere, it includes ALL immediate subdirectories of preprocessed/, but within those, ONLY the files that have names which could have been generated by a known preprocessor from one of the included source files.

With the subdirectory-naming, I added a new user hook that implements the concept of "specializing" the preprocessed output on a set of variables. Rather than defining a new vocabulary of variables which would then become maintenance overhead, I reused PathTemplateVariable? for this purpose. It constructs a subdirectory name as a canonical representation of the set of variables specialized on, with their values. It uses URL-escaping for characters that we can't use for various reasons.

I also made a trivial change to Happy's Setup.lhs, to test the specialization feature. I've included that one too; once the Cabal changes are released, either you or I can submit it to the Happy maintainer for inclusion there. It doesn't actually implement using Happy's --ghc option; it merely specializes the build properly, outputting into "compiler=ghc-6.blahblahblah" if the compiler is GHC or into "generic" otherwise. It does this by installing a preprocesserSpecialization hook which looks at the LocalBuildInfo? it's given, checks the compiler flavor therein, and returns [CompilerVar?] or [] as appropriate. Hopefully this is a sufficiently general capability for the use-cases that exist in the wild.

As always, see the source for more details. I tried to follow your coding style as I understood it; in particular when I needed to add imports I enumerated each symbol used. I also tried to comment everything of any importance. I'll be interested in your feedback on how well I managed.

[dankna]

Patch for Cabal in "darcs send" format.

[dankna]

Patch for Cabal-Install in "darcs send" format.

[dankna]

For testing purposes, modified Setup.lhs for Happy 1.18.4.

[nomeata]

Is there still something happening on this front? It seems that, for example, uuagc could greatly benefit if Setup sdist would include all the generated .hs files in the tarball required to bootstrap the program; the current scheme (a separate uuagc-bootstrap package) feels like unnecessary overkill to me.