Missing -lz option in testsuite

[daniel.is.fischer]

Running the testsuite with a newly built 6.12.1, I got

OVERALL SUMMARY for test run started at Mo 14. Dez 18:06:00 CET 2009
    2352 total tests, which gave rise to                            
   13034 test cases, of which                                       
       0 caused framework failures                                  
    2760 were skipped                                               

    9471 expected passes
     328 expected failures
       0 unexpected passes
     475 unexpected failures

All but 8 (unless I miscounted) of the unexpected failures were in way threaded1, the overwhelming majority due to

/usr/src/packages/BUILD/binutils-2.19/build-dir/bfd/../../bfd/compress.c:96:0:
     undefined reference to `inflateInit_'                                    

/usr/src/packages/BUILD/binutils-2.19/build-dir/bfd/../../bfd/compress.c:103:0:
     undefined reference to `inflate'                                          

/usr/src/packages/BUILD/binutils-2.19/build-dir/bfd/../../bfd/compress.c:106:0:
     undefined reference to `inflateReset'                                     

/usr/src/packages/BUILD/binutils-2.19/build-dir/bfd/../../bfd/compress.c:108:0:
     undefined reference to `inflateEnd'                                       
collect2: ld returned 1 exit status                                            

*** unexpected failure for fileStatus(threaded1)

That looks like a missing -lz option in the testsuite. Adding "EXTRA_HC_OPTS += -optl-lz" to test.mk and running "make stage=2 WAY=threaded1" produced

OVERALL SUMMARY for test run started at Di 15. Dez 12:18:20 CET 2009
    2352 total tests, which gave rise to
   13034 test cases, of which
       0 caused framework failures
   12545 were skipped

     467 expected passes
      19 expected failures
       0 unexpected passes
       3 unexpected failures

Unexpected failures:
   2317(threaded1)
   galois_raytrace(threaded1)
   qq005(threaded1)

which seems to confirm the suspicion.

[daniel.is.fischer]

Also: galois_raytrace fails with

Compile failed (status 256) errors were:
<command line>: cannot satisfy -package parsec
    (use -v for more information)

*** unexpected failure for galois_raytrace(normal)

What's best to do about tests needing packages no longer included?

[daniel.is.fischer]

Further unexpected failures due to packages no longer included:

=====> 2317(normal)
cd ./concurrent/2317 && '/home/dafis/GHC121/ghc-6.12.1/inplace/bin/ghc-stage2' -fforce-recomp 
-dcore-lint -dcmm-lint -no-user-package-conf -optl-lz -dno-debug-output --make -o 2317 2317    
>2317.comp.stderr 2>&1
Compile failed (status 256) errors were:

2317.hs:4:7:
    Could not find module `Control.Parallel':
      Use -v to see a list of the files searched for.

*** unexpected failure for 2317(normal)

=====> pkg02(normal)
cd ./cabal/pkg02 && '/home/dafis/GHC121/ghc-6.12.1/inplace/bin/ghc-stage2' -fforce-recomp 
-dcore-lint -dcmm-lint -no-user-package-conf -optl-lz -dno-debug-output --make -o pkg02 A  -v0  
>pkg02.comp.stderr 2>&1
Compile failed (status 256) errors were:

A.hs:3:7:
    Could not find module `Network.Socket':
      Use -v to see a list of the files searched for.

*** unexpected failure for pkg02(normal)

=====> qq005(normal)
cd ./quasiquotation/qq005 && '/home/dafis/GHC121/ghc-6.12.1/inplace/bin/ghc-stage2' 
-fforce-recomp -dcore-lint -dcmm-lint -no-user-package-conf -optl-lz -dno-debug-output 
--make -o qq005 Main    >qq005.comp.stderr 2>&1
Compile failed (status 256) errors were:

Expr.hs:9:7:
    Could not find module `Text.ParserCombinators.Parsec.Char':
      Use -v to see a list of the files searched for.

*** unexpected failure for qq005(normal)

=====> qq006(normal)
cd ./quasiquotation/qq006 && '/home/dafis/GHC121/ghc-6.12.1/inplace/bin/ghc-stage2' 
-fforce-recomp -dcore-lint -dcmm-lint -no-user-package-conf -optl-lz -dno-debug-output 
--make -o qq006 Main  -v0  >qq006.comp.stderr 2>&1
Actual stderr output differs from expected:
--- ./quasiquotation/qq006/qq006.stderr.normalised	2009-12-15 13:47:10.000000000 +0100
+++ ./quasiquotation/qq006/qq006.comp.stderr.normalised	2009-12-15 13:47:10.000000000 +0100
@@ -1,4 +1,4 @@

-Main.hs:8:20:
-    Conflicting definitions for `x'
-    In a case alternative
+Expr.hs:9:7:
+    Could not find module `Text.ParserCombinators.Parsec.Char':
+      Use -v to see a list of the files searched for.
*** unexpected failure for qq006(normal)

Expected (?) failures due to packages not present:

okeefe_neural: lang
galois_raytrace: parsec
cg025: regex-compat
SampleVar001, Chan001, MVar001, QSemN001, QSem001, ghci014, maessen_hashtab: QuickCheck
regex001, regex002, regex003: regex-posix
dynamic002, packedstring001: packedstring
2185: parallel

Probably they should have failed for other reasons.

More interesting unexpected failures:

=====> T1969(normal)
cd ./perf/compiler && '/home/dafis/GHC121/ghc-6.12.1/inplace/bin/ghc-stage2' -fforce-recomp 
-dcore-lint -dcmm-lint -no-user-package-conf -optl-lz -dno-debug-output -c T1969.hs   
+RTS -V0 -tT1969.comp.stats --machine-readable -RTS  >T1969.comp.stderr 2>&1
bytes allocated 209182332 is less than minimum allowed 210000000
If this is because you have improved GHC, please
update the test so that GHC doesn't regress again
*** unexpected failure for T1969(normal)

=====> annrun01(dyn)
cd ./annotations/should_run && '/home/dafis/GHC121/ghc-6.12.1/inplace/bin/ghc-stage2' 
-fforce-recomp -dcore-lint -dcmm-lint -no-user-package-conf -optl-lz -dno-debug-output 
--make -o annrun01 annrun01 -O -dynamic -package ghc  >annrun01.comp.stderr 2>&1
Compile failed (status 256) errors were:
[1 of 3] Compiling Annrun01_Help    ( Annrun01_Help.hs, Annrun01_Help.o )
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
Loading package array-0.3.0.0 ... linking ... done.
Loading package containers-0.3.0.0 ... linking ... done.
Loading package filepath-1.1.0.3 ... linking ... done.
Loading package old-locale-1.0.0.2 ... linking ... done.
Loading package old-time-1.0.0.3 ... linking ... done.
Loading package unix-2.4.0.0 ... linking ... done.
Loading package directory-1.0.1.0 ... linking ... done.
Loading package pretty-1.0.1.1 ... linking ... done.
Loading package process-1.0.1.2 ... linking ... done.
Loading package Cabal-1.8.0.2 ... linking ... done.
Loading package bytestring-0.9.1.5 ... linking ... done.
Loading package ghc-binary-0.5.0.2 ... linking ... done.
Loading package bin-package-db-0.0.0.0 ... linking ... done.
Loading package hpc-0.5.0.4 ... linking ... done.
Loading package template-haskell ... linking ... done.
Loading package ghc-6.12.1 ... linking ... done.
Loading package ffi-1.0 ... linking ... done.
Loading object (dynamic) z ... done
final link ... done
Annrun01_Help.hs:26:0:
    Dynamic linking required, but this is a non-standard build (eg. prof).
    You need to build the program twice: once the normal way, and then
    in the desired way using -osuf to set the object file suffix.

*** unexpected failure for annrun01(dyn)

=====> barton-mangler-bug(profc)
cd ./programs/barton-mangler-bug && '/home/dafis/GHC121/ghc-6.12.1/inplace/bin/ghc-stage2' 
-fforce-recomp -dcore-lint -dcmm-lint -no-user-package-conf -optl-lz -dno-debug-output --make 
-o barton-mangler-bug Main -O -prof -auto-all -fvia-C   >barton-mangler-bug.comp.stderr 2>&1
Compile failed (status 25344) errors were:
[1 of 7] Compiling Plot             ( Plot.lhs, Plot.o )
[2 of 7] Compiling Expected         ( Expected.hs, Expected.o )

*** unexpected failure for barton-mangler-bug(profc)

=====> break024(ghci)
cd ./ghci.debugger/scripts && HC='/home/dafis/GHC121/ghc-6.12.1/inplace/bin/ghc-stage2' 
HC_OPTS='-dcore-lint -dcmm-lint -no-user-package-conf -optl-lz -dno-debug-output ' 
'/home/dafis/GHC121/ghc-6.12.1/inplace/bin/ghc-stage2' --interactive -v0 -ignore-dot-ghci 
-dcore-lint -dcmm-lint -no-user-package-conf -optl-lz -dno-debug-output  -ignore-dot-ghci   
<break024.script >break024.run.stdout 2>break024.run.stderr
Actual stdout output differs from expected:
--- ./ghci.debugger/scripts/break024.stdout.normalised	2009-12-15 15:05:51.000000000 +0100
+++ ./ghci.debugger/scripts/break024.run.stdout.normalised	2009-12-15 15:05:51.000000000 +0100
@@ -1,20 +1,26 @@
 Left user error (error)
 Stopped at <exception thrown>
-_exception :: e = _
+_exception :: e = SomeException (GHC.Exception.D:Exception _
+                                                           (GHC.Show.D:Show ...) ....)
+                                (GHC.IO.Exception.IOError Nothing GHC.IO.Exception.UserError ....)
 _exception = SomeException (GHC.Exception.D:Exception _
                                                       (GHC.Show.D:Show _ _ _) _ _)
                            (GHC.IO.Exception.IOError Nothing GHC.IO.Exception.UserError []
                                                      ['e','r','r','o','r'] Nothing Nothing)
 *** Exception: user error (error)
 Stopped at <exception thrown>
-_exception :: e = _
+_exception :: e = SomeException (GHC.Exception.D:Exception _
+                                                           (GHC.Show.D:Show ...) ....)
+                                (GHC.IO.Exception.IOError Nothing GHC.IO.Exception.UserError ....)
 _exception = SomeException (GHC.Exception.D:Exception _
                                                       (GHC.Show.D:Show _ _ _) _ _)
                            (GHC.IO.Exception.IOError Nothing GHC.IO.Exception.UserError []
                                                      ['e','r','r','o','r'] Nothing Nothing)
 *** Exception: user error (error)
 Stopped at <exception thrown>
-_exception :: e = _
+_exception :: e = SomeException (GHC.Exception.D:Exception _
+                                                           (GHC.Show.D:Show ...) ....)
+                                (GHC.IO.Exception.IOError Nothing GHC.IO.Exception.UserError ....)
 _exception = SomeException (GHC.Exception.D:Exception _
                                                       (GHC.Show.D:Show _ _ _) _ _)
                            (GHC.IO.Exception.IOError Nothing GHC.IO.Exception.UserError []
*** unexpected failure for break024(ghci)

=====> conc012(ghci)
cd ./concurrent/should_run && '/home/dafis/GHC121/ghc-6.12.1/inplace/bin/ghc-stage2' 
-fforce-recomp -dcore-lint -dcmm-lint -no-user-package-conf -optl-lz -dno-debug-output 
conc012.hs --interactive -v0 -ignore-dot-ghci +RTS -I0.1 -RTS   <conc012.genscript 
1>conc012.interp.stdout 2>conc012.interp.stderr
Wrong exit code (expected 0 , actual 99 )
Stdout:

Stderr:

*** unexpected failure for conc012(ghci)

=====> rtsflags001(normal)
cd ./rts && '/home/dafis/GHC121/ghc-6.12.1/inplace/bin/ghc-stage2' -fforce-recomp -dcore-lint 
-dcmm-lint -no-user-package-conf -optl-lz -dno-debug-output -o rtsflags001 rtsflags001.hs    
>rtsflags001.comp.stderr 2>&1
cd ./rts && ./rtsflags001 +RTS -H0m -RTS   </dev/null >rtsflags001.run.stdout 2>rtsflags001.run.stderr
Wrong exit code (expected 1 , actual 0 )
Stdout:

Stderr:

*** unexpected failure for rtsflags001(normal)

The overall result with -optl-lz is

OVERALL SUMMARY for test run started at Di 15. Dez 12:50:40 CET 2009
    2352 total tests, which gave rise to
   13034 test cases, of which
       0 caused framework failures
    2760 were skipped

    9940 expected passes
     296 expected failures
       0 unexpected passes
      38 unexpected failures

Unexpected failures:
   2317(normal,optc,hpc,optasm,profc,profasm,ghci,threaded1,threaded2,dyn,profthreaded)
   T1969(normal)
   annrun01(dyn)
   barton-mangler-bug(profc)
   break024(ghci)
   conc012(ghci)
   galois_raytrace(normal,optc,profc,ghci,threaded1,dyn)
   pkg02(normal,optc,hpc,optasm,profc,profasm)
   qq005(normal,optc,hpc,optasm,ghci,threaded1,threaded2,dyn)
   qq006(normal)
   rtsflags001(normal)

Only six of the unexpected failures are not due to an absent package.

[simonmar]

All of those failures that depend on external packages should be counted as "expected failures" by the testsuite framework, I'm not sure why they're not in your case. It certainly seems to be working here.

Could you fill in the Operating System / Architecture fields on the ticket please?

[daniel.is.fischer]

Sure, done.

[simonmar]

I'm guessing that you have a static libbfd.a on your system, but no shared version, correct? It looks like libbfd depends on libz. We link against libbfd in the DEBUG version of the RTS, to get access to the addresses of symbols, and we test that -lbfd works in the configure script, but I imagine that because you have a static libbfd that the test doesn't link in the bit that needs libz and thus succeeds.

The easiest workaround is for you to install a shared libbfd. We could fix the configure test, but I'm not sure how to do it properly.

[daniel.is.fischer]

I have a shared libbfd:

$ locate libbfd
/usr/lib/libbfd-2.19.so
/usr/lib/libbfd.a

The version number on the shared can't screw things up, can it?

[simonmar]

Perhaps in that case your libbfd requires libz, but it is not linked against it? Can you look for those inflate symbols? My libbfd.a here doesn't refer to them.

What Linux distro is this, BTW?

[daniel.is.fischer]

What nm spits out:

$ nm -D /usr/lib/libbfd-2.19.so | grep inflate
         U inflate
         U inflateEnd
         U inflateInit_
         U inflateReset
$ nm /usr/lib/libbfd.a | grep inflate
         U inflate
         U inflateEnd
         U inflateInit_
         U inflateReset

Does that help you? I guess it means you're right, it needs libz.

The distro is openSUSE 11.1.

[simonmar]

and can you try running ldd on your libbfd-2.19.so? Mine says:

$ ldd /usr/lib64/libbfd-2.18.50.0.6-2.so 
        linux-vdso.so.1 =>  (0x00007fffa61fe000)
        libc.so.6 => /lib64/libc.so.6 (0x0000003c09a00000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003c09600000)

[daniel.is.fischer]

Mine says

$ ldd /usr/lib/libbfd-2.19.so
        linux-gate.so.1 =>  (0xffffe000)
        /usr/lib/libv4l/v4l2convert.so (0xb7618000)
        libz.so.1 => /lib/libz.so.1 (0xb75da000)
        libc.so.6 => /lib/libc.so.6 (0xb747e000)
        libv4l2.so.0 => /usr/lib/libv4l2.so.0 (0xb7474000)
        /lib/ld-linux.so.2 (0xb789c000)
        libv4lconvert.so.0 => /usr/lib/libv4lconvert.so.0 (0xb740e000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb73f4000)

Yep,definitely depends on libz.

[igloo]

It sounds like installing something like a libbfd-dev package would work around this, as then we would link to the shared libbfd.

For a better fix, it would be useful if someone who can reproduce the problem can tweak the configure test to handle this situation properly.

[daniel.is.fischer]

What's the problem with adding '-optl-lz' to the command line of the pertinent tests or to EXTRA_HC_OPTS? Is it that the linker would/could pick the symbols from libz even when libbfd defines them too, and that would break things? So we'd need to determine whether libbfd depends on libz, and if it does, pass -lz. I assume testing whether

ldd `locate libbfd | grep so` | grep libz

returns a nonempty string is too naive/nonportable?

[igloo]

Just adding -optl-lz would needlessly make that a dependency for everyone.

The configure test should be based on trying to compile/run test programs.

[simonmar]

Replying to daniel.is.fischer:

What's the problem with adding '-optl-lz' to the command line of the pertinent tests or to EXTRA_HC_OPTS?

That would be the wrong place, the fact that you saw failures in the test suite is a symptom, not the problem. On your system libbfd depends on libz, and you're picking up a static libbfd, so hence there's a dependency from the debug variants of the RTS on libz. If you build any Haskell program with -debug you'll see a link failure. The right way to fix it is to add "z" to the extra-libraries of the RTS, but we need to figure out when to do this by adding a configure test.

I'd also like to understand what exactly is making this go wrong on your system, so we can figure out whether its a distro-specific thing or something else.

[daniel.is.fischer]

Ah, now I (begin to) understand. I had no idea it was connected to -debug per se, I thought it was something specific to the tests. Indeed, trying to compile something with -debug gives

Linking Trivial ...
/usr/lib/gcc/i586-suse-linux/4.3/../../../libbfd.a(compress.o):
         In function `bfd_uncompress_section_contents':

/usr/src/packages/BUILD/binutils-2.19/build-dir/bfd/../../bfd/compress.c:96:0:
     undefined reference to `inflateInit_'

/usr/src/packages/BUILD/binutils-2.19/build-dir/bfd/../../bfd/compress.c:103:0:
     undefined reference to `inflate'

/usr/src/packages/BUILD/binutils-2.19/build-dir/bfd/../../bfd/compress.c:106:0:
     undefined reference to `inflateReset'

/usr/src/packages/BUILD/binutils-2.19/build-dir/bfd/../../bfd/compress.c:108:0:
     undefined reference to `inflateEnd'
collect2: ld returned 1 exit status

unless I pass -optl-lz on the command line (it names libbfd.a even with -dynamic, is that necessarily a bug?).

Okay, I don't know whether that'll help diagnose the problem, but looking a bit further,

$ nm /usr/lib/libbfd.a | grep -B 8 inflate
compress.o:
00000000 r .LC0
00000005 r .LC1
         U _GLOBAL_OFFSET_TABLE_
00000000 T __i686.get_pc_thunk.bx
         U bfd_malloc
00000000 T bfd_uncompress_section_contents
         U free
         U inflate
         U inflateEnd
         U inflateInit_
         U inflateReset

and with a newly built binutils-2.20,

$ nm bfd/libbfd.a | grep -B 4 inflate
compress.o:
         U bfd_malloc
00000000 T bfd_uncompress_section_contents
         U free
         U inflate
         U inflateEnd
         U inflateInit_
         U inflateReset

and in bfd/configure, we find

# Link in zlib if we can.  This allows us to read compressed debug sections.
# This is used only by compress.c.

So it seems binutils inserts the dependency on libz if it finds one during configure. It's quite adamant about it, I didn't manage to turn it off by configure options, only by editing the code.

Makes me wonder, how come your libbfd.a doesn't mention inflate* ? What does nm libbfd.a | grep -A 4 compress.o return?

[simonmar]

Replying to daniel.is.fischer:

Makes me wonder, how come your libbfd.a doesn't mention inflate* ? What does nm libbfd.a | grep -A 4 compress.o return?

On my laptop (Ubuntu Karmic), I get the same as you. However, I also have a libbfd.so which gets picked up first, and it has the libz dependency built-in, so I don't run into problems.

[daniel.is.fischer]

Still four unexpected failures while running validate in a HEAD repo today.
Failing tests:

3231(threaded1)
ffi014(threaded1)
numsparks001(threaded1)
testwsdeque(threaded1)

[daniel.is.fischer]

Not setting EXTRA_HC_OPTS to -optl-lz, I still get lots of undefined references to inflateInit_ and friends, setting that option in mk/test.mk, I get that only for 4850(normal) (both, HEAD and 7.0.3), seems EXTRA_HC_OPTS doesn't get passed for that one.

[simonmar]

Fixed (or worked around) with changeset:ef062308fb38b2cbbf30a0057bd839b5382648a4