HsFFI.h is not in the default include path for hsc2hs

[cjs]

Using the ghc 6.1.0 package for libedit2 downloaded from the GHC downloads page, on Ubuntu 8.04.1, attempting to compile an hsc file fails with:

In file included from /u/cjs/co/client/tsuru/trader/build/src/trader/src/HX/Empty_hsc_make.c:1:
/usr/local/lib/ghc-6.10.1/hsc2hs-0.67/template-hsc.h:4:19: error: HsFFI.h: No such file or directory
compiling /u/cjs/co/client/tsuru/trader/build/src/trader/src/HX/Empty_hsc_make.c failed
command was: /usr/bin/gcc -c /u/cjs/co/client/tsuru/trader/build/src/trader/src/HX/Empty_hsc_make.c -o /u/cjs/co/client/tsuru/trader/build/src/trader/src/HX/Empty_hsc_make.o

I note that the command does not include a -I option. Either running the command by hand with an appropriate -I option, or setting the environment variable CPATH=/usr/local/lib/ghc-6.10.1/include fixes the issue.

[duncan]

Honestly I'm not sure how this is supposed to work. hsc2hs is a compiler-independent tool however it needs the HsFFI.h file from the compiler you want to use it with. Previously ghc shipped a hsc2hs fork that used ghc as the C compiler. Now it uses gcc but does not include the ghc FFI dir by default. It's not clear what we should make it do since it's nice for it to work by default but it is also rather useful to be able to share a single installation of hsc2hs between multiple versions of ghc or indeed hugs or nhc98.

We should perhaps look at how this was ever supposed to work, before the ghc fork of hsc2hs came along.

Note that the reason not everyone is trupping over this is because most people do not call hsc2hs directly but have it called via Cabal which passes all the extra appropriate flags.

[igloo]

Fixed in HEAD and 6.10.2 by passing a suitable -I flag at the end of the commandline:

Tue Feb 17 12:24:07 PST 2009  Ian Lynagh <igloo@earth.li>                       
  * Add a -I flag with the path to HsFFI.h                                      
  This can still be overridden by adding a -I flag to the commandline,          
  but it means that hsc2hs will work if no flags are given.                     

But we may want to fix it differently in the future; see  http://www.haskell.org/pipermail/glasgow-haskell-users/2009-February/016606.html for more details.

[duncan]

I propose that the next hsc2hs release should have a default template file that does not include HsFFI.h. It's not needed by the template itself or any code hsc2hs generates. It's only provided for the convenience of .hsc modules, but they can #include "HsFFI.h" themselves if they need it. I expect (but have not checked) that very very few .hsc modules actually do need HsFFI.h and most of the ones that do are probably in the core libs which are the easiest to fix. A survey of hackage is needed to confirm this.

[ezyang]

There are two “spurious” test failures in the MinGW test suite, namely hsc2hs001 and hsc2hs002. They pass if you run the tests from the root directory, but fail if you run them from the testsuite directory. The invocation of hsc2hs is as follows:

In file included from hsc2hs002_hsc_make.c:1:
C:/cygwin/home/ezyang/ghc/inplace/lib/template-hsc.h:4:19: HsFFI.h: No such file or directory
compiling hsc2hs002_hsc_make.c failed (exit code 1)
command was: C:/cygwin/home/ezyang/ghc/inplace/lib/../mingw/bin/gcc.exe -c hsc2hs002_hsc_make.c -o hsc2hs002_hsc_make.o -IC:/cygwin/home/ezyang/ghc/inplace/lib/include/

inplace/lib/include/ does not exist; maybe our build scripts should copy the includes directory there, and then we should stop failing these two tests.

[ezyang]

(Note: I was going to fix this myself, but I couldn't figure out where in GHC's build system such a change would go)

[igloo]

I'm going to declare this fixed by:

commit 4f8f41929c42f5680e745f4a3ceebb4ff26e5cc8
Author: Ian Lynagh <igloo at earth.li>
Date:   Sun Sep 11 18:44:35 2011 +0100

    Don't include <HsFFI.h> in template-hsc.h; fixes #2897
    
    This works around the problem of hsc2hs not knowing the path to get it
    from. We don't think many, if any, users actually need it. Those that do
    will have to explicitly include it themselves, and their build system
    will be responsible for passing a suitable -I flag to hsc2hs (Cabal
    already does so).

[simonmar]

This fix has caused problems on my machine: System.Posix.modificationTime is using the wrong offset into the stat structure.

I tracked it down, and the reason is that the C file generated by hsc2hs now include template-hsc.h before anything else, and this header includes some other standard headers (stdlib.h etc.). This means that after including template-hsc.h it is too late to #define _FILE_OFFSET_BITS to be 64. It worked before because HsFFI.h was included first, and that included ghcautoconf.h which set _FILE_OFFSET_BITS to 64.

So I'm not sure what the right fix is here. The way it worked before was fragile, so I'm not surprised it broke.

hsc2hs never puts anything before #include "template-hsc.h", but we could make it put --include headers in front. However, we would then have to arrange to pass the right --include options to hsc2hs, which means modifying both Cabal and GHC's build system to do that (and which headers? the ones from the includes: line in the .cabal file?).

Any ideas? I'm backing out this patch in my tree for now.

[simonmar]

Symptoms of this problem: #5503.

[simonmar]

More symptoms: #5504

[igloo]

Or perhaps hsc2hs could not insert #include "template-hsc.h" if it's already in the file? Similar to how implicit Prelude imports work in Haskell.

[duncan]

I suggest we go back to having the standard template #include HsFFI.h first.

The point is:

• hsc2hs is a generic tool that can be used for different Haskell implementations
• The output of hsc2hs is however not independent of the Haskell implementation. This is because each Haskell implementation has its own C environment, and hsc2hs must use that same environment or it will produce incorrect results.
• The HsFFI.h is a reasonable way for each Haskell implementation to set up its C environment.

Therefore, when using hsc2hs, the user (or build tool) needs to tell hsc2hs which Haskell implementation they are using, by pointing hsc2hs to that implementation's HsFFI.h file.

So running hsc2hs foo.hsc on it's own should fail. That is the right thing to do. There is not enough information from that invocation to produce correct results. It should not guess, that will also produce incorrect results.

However it should fail in a comprehensible way and provide the user with an easy way to do the basic thing:

1. search the given -I dirs for HsFFI.h.
2. if it's not found, complain that the user needs to tell it which Haskell implementation to use (because the results depend on the C environment of that implementation). Tell the user they can either specify the -I dir for their impl, or they can use the --ghc flag.
3. if the --ghc flag is given, hsc2hs calls ghc-pkg field rts include-dirs, using the ghc-pkg on the $PATH, to find the include dir for HsFFI.h. No support for other ghc instances/versions, just the one on the $PATH. All non-trivial uses should work by specifying the appropriate -I dir for the Haskell impl instance.

This combination means that people who really want to invoke hsc2hs manually can do so by giving the --ghc flag. Everyone else can continue to have other build tools call hsc2hs, and pass in the appropriate -I flags for the Haskell impl that they are using. This makes it a relatively lightweight solution, hsc2hs doesn't need to know much about lots of different Haskell implementations, just enough to satisfy casual users.

[simonmar]

There's another solution:

• Have hsc2hs emit two C files, one that looks like the current one, and another one that defines simple wrappers for the C library functions that we call (fprintf etc.)

• template-hsc.h declares prototypes for the wrappers, and defines its own macros in terms of them. It does not need to #include any system headers.

• hsc2hs will have to compile both C files and link them together into the final program.

So then because we're not including any system headers in template-hsc.h, the problem should go away.

Even if we adopt Duncan's proposal, I think we should do this too, because including system headers in the same C file that generates the .hs output has the potential to create other subtle problems too. By avoiding doing this we give the user full control over what headers are included and in what order.

[igloo]

Simon's proposal seems like a sensible thing to do regardless, so let's do it.

[igloo]

Done:

commit 5f3ac5267914cc020c11b148cdbc4356bd6eb02a
Author: Ian Lynagh <igloo@earth.li>
Date:   Tue Oct 25 15:45:36 2011 +0100

    Various hsc2hs improvements

    * Fix GHC trac #2897: Generate a separate "utils" C file containing
      wrappers for functions like printf. This means template-hsc.h
      doesn't need to includes any headers, so we don't pollute the C
      environment.
      (actually, we still need to include stddef.h).

    * Use runProcess rather than system. It used to try to conditionally
      use runProcess if it was available, but never actually did. Now it
      uses it unconditionally.

    * Some tidying up