-fPIC without -dynamic silently ignored

[asuffield]

$ ghc -fPIC -c -o TestF.o TestF.hs
$ ghc -shared -dynamic TestF.so TestF.o
/usr/bin/ld: TestF.o: relocation R_X86_64_PC32 against undefined symbol `stg_CAF_BLACKHOLE_info' 
can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: ld returned 1 exit status

$ ghc -fPIC -dynamic -c -o TestF.o TestF.hs
$ ghc -shared -dynamic TestF.so TestF.o
$

If you attempt to use -fPIC without -dynamic, then you get non-PIC code in the .o file. This behaviour is surprising and quite useless.

[duncan]

It looks like this is behaving as specified. Yes, the message says to use -fPIC, and that'd probably be right if we were talking about C code and gcc.

The symbol type R_X86_64_PC32 looks like a PC-relative reference to a symbol it expects to find within the same shared object. That's just what you would expect from -fPIC without -dynamic.

Of course what you're doing here is making PIC code that expects to find other symbols within the same shared object, but then linking it in such a way that the other symbols are in an external shared object. That indeed will not work.

It should in principle be possible to build the rts and standard libs with -fPIC and without -dynamic. These would be made into .a libs. Then it should be possible to make one huge .so linking in all the Haskell packages and rts. Internally to this big .so, all the symbol references would be PC-relative and not go via the GOT.

[asuffield]

While true in the general case, we can detect that this particular ghc was built in the normal fashion, and hence can detect the error while parsing the options.

Furthermore, this is an "unbreak me" option - it's worth rethinking the defaults here. There don't seem to be any users who want to build gigantic shared libraries containing the rts, while there are probably some users who want to build normal shared libraries.

[duncan]

Replying to asuffield:

While true in the general case, we can detect that this particular ghc was built in the normal fashion, and hence can detect the error while parsing the options.

Well, we could detect it if we hard-coded that this ghc instance did not have the core libs built a particular way. We have been considering building libs with -fPIC by default on x86-64, even for the static linking case. If we did, this use case would work fine.

Furthermore, this is an "unbreak me" option - it's worth rethinking the defaults here. There don't seem to be any users who want to build gigantic shared libraries containing the rts, while there are probably some users who want to build normal shared libraries.

But there are. That's exactly what people do on Windows (indeed it's the only supported mode).

As I said, better would be to record that packages or individual .o/.hi files were built expecting external symbols inside or outside the same shared object.

[simonmar]

Currently the options make sense to someone who understands the implementation, but they make little sense to the user. I suggest we rethink the options, with a focus on the various modes of use that we want to support. e.g. just adding -dynamic-lib for building a module to be placed in a shared library would probably be enough: the idea would be that you use -dynamic-lib when building the library, and -dynamic when building the exe.

-dynamic-lib would imply -dynamic, -fPIC where necessary, and -shared when linking.

[duncan]

Replying to simonmar:

Currently the options make sense to someone who understands the implementation, but they make little sense to the user.

True and true. When I first started on the shared lib hacking I was convinced that we did not need all of -dynamic, -fPIC and -shared. But it turns out all the combinations make sense for something.

Even -dynamic need not imply -fPIC or vice versa.

I suggest we rethink the options, with a focus on the various modes of use that we want to support. e.g. just adding -dynamic-lib for building a module to be placed in a shared library would probably be enough: the idea would be that you use -dynamic-lib when building the library, and -dynamic when building the exe. -dynamic-lib would imply -dynamic, -fPIC where necessary, and -shared when linking.

Yep, that works. The docs will have to present it well so that it reduces confusion rather than adding to it by adding yet another similar-named option.

Note that this only really helps people in the case that they're making shared libs for integration with C code. For constructing Haskell packages there are a bunch more conventions they must follow to make it work (in particular getting the right lib name).

Also, I still think there's a case to be made for adding some extra consistency info so that we get better diagnostics when linking incompatible stuff. This stuff is certainly confusing, even when you know how it works. The key thing to track about a built package/object is whether inter-package references are expected to be in the same or a different shared object.

[simonmar]

Two new tickets created to replace this one:

#3712

Implement -dynamic-lib option

#3713

Track -dynamic/-fPIC to avoid obscure linker errors