more object-code blow-up in ghc-6.8.3 vs. ghc-6.4.2 (both with optimization)

[maeder@…]

After upgrading from ghc-6.4.2 to ghc-6.6 our already large binary became noticeably larger.

I'll include a test case based on the programatica sources and our ATerm library. It's only necessary to compile the two top-level files ATC.hs and ATC2.hs and the files in Common/ATerm with optimization. This produces large object files (under i386 linux) of the following sizes:

using ghc-6.4.2: 160K ATC2.o 721K ATC.o (-O0) 2,1M ATC2.o 1,7M ATC.o (-O)

using ghc-6.6: 156K ATC2.o 729K ATC.o (-O0) 4,1M ATC2.o 1,8M ATC.o (-O)

After unpacking atc.tgz calling make should check out some programatica sources and compile everything with optimization up to ATC2.o. (programatica requires the convert.sh script for ghc-6.6)

Christian

P.S. I've check the old ticket  http://hackage.haskell.org/trac/ghc/ticket/490 (In that example the object code does no longer blow up.)

[maeder@…]

I'll update the attachment. Meanwhile latin1 characters have been removed from porgramatica. I've also tested to use a pragma in ATC2.hs in order to reduce the object code size.

{-# OPTIONS -O0 -fmax-simplifier-iterations2 #-} reduced the size of ATC2.o from 4,1M to 3,5M (which is still far to big).

Just using "-fmax-simplifier-iterations0" leaded to:

ghc-6.6: panic! (the 'impossible' happened)
  (GHC version 6.6 for i386-unknown-linux):
        simplCore/FloatOut.lhs:(393,0)-(394,45): Non-exhaustive patterns in function getBindLevel

However in conjunction with "-O0" it compiles nicely to a little object files but fails to link then, i.e. with {-# OPTIONS -O0 -fmax-simplifier-iterations0 #-} as first line of ATC2.hs:

Linking ATC3 ...
ATC2.o(.text+0x5214): In function `ATC2_zdf20_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0x5374): In function `ATC2_zdf19_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0x54d4): In function `ATC2_zdf18_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0x7bd4): In function `ATC2_zdf3_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0x7d34): In function `ATC2_zdf2_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0x8e2c): In function `s4RO_info':
ghc26101_0.hc: undefined reference to `TiTypes_zdWS_info'
ATC2.o(.text+0xcc98): In function `s5gf_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0xce14): In function `s5gM_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0xd094): In function `s5hC_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0xd210): In function `s5i9_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0xd490): In function `s5iZ_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0xddc8): In function `s5m3_info':
ghc26101_0.hc: undefined reference to `TiKinds_zdWKVar_info'
ATC2.o(.text+0xf3b8): In function `s5tY_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0xf534): In function `s5uv_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0xf6b0): In function `s5v2_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0xf82c): In function `s5vz_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0xf9a8): In function `s5w6_info':
ghc26101_0.hc: undefined reference to `base_DataziTypeable_zdWZCDTypeable_info'
ATC2.o(.text+0xfb08): In function `s5y3_info':
ghc26101_0.hc: undefined reference to `PropSyntaxRec_zdWDec_info'
ATC2.o(.text+0x10574): In function `s5Cw_info':
ghc26101_0.hc: undefined reference to `PropSyntaxRec_zdWPA_info'
ATC2.o(.text+0x10e10): In function `s5Gx_info':
ghc26101_0.hc: undefined reference to `PropSyntaxRec_zdWPP_info'
ATC2.o(.text+0x11724): In function `s5KC_info':
ghc26101_0.hc: undefined reference to `PropSyntaxRec_zdWExp_info'
collect2: ld returned 1 exit status

[igloo]

Christian says "[...] suggest to change the milestone to 6.8. Although the code-bloat is not nice, we could live with it."

[igloo]

Regression, so high priority

[igloo]

Christian,

Can you please tell me where I can get the other modules, e.g. SourceNames?, to reproduce this?

Thanks

[maeder]

SourceNames comes from programatica. It should be sufficient to check out:

cvs -d :pserver:anoncvs@cvs.haskell.org:/cvs co -P programatica/tools/base
cvs -d :pserver:anoncvs@cvs.haskell.org:/cvs co -P programatica/tools/property

I'll check if the attached atc.2.tgz behaves as described for ghc-6.8.2, too.

[maeder]

It compiles fine for me (110 files by calling "make") with ghc-6.8.2 and the object code sizes are much better than for ghc-6.6.1! So you may close this ticket.

ghc-6.6.1

340K ATC.hi
 88K ATC.hs
1.8M ATC.o
 12K ATC2.hi
 16K ATC2.hs
3.5M ATC2.o
8.0M ATC3
8.0K ATC3.hi
4.0K ATC3.hs
4.0K ATC3.o

ghc-6.8.2

maeder@leibniz:/local/maeder/atc> ls -sh1
total 7.1M
324K ATC.hi
 88K ATC.hs
968K ATC.o
 12K ATC2.hi
 16K ATC2.hs
896K ATC2.o
4.8M ATC3
8.0K ATC3.hi
4.0K ATC3.hs
4.0K ATC3.o
4.0K Common
4.0K Makefile
4.0K Makefile~
 12K atc.2.tgz
4.0K programatica

[maeder]

Even after removing the first line in ATC2.hs, {-# OPTIONS -O0 -fmax-simplifier-iterations2 #-}, the code only increases lightly.

maeder@leibniz:/local/maeder/atc> ls -sh1
total 7.8M
324K ATC.hi
 88K ATC.hs
968K ATC.o
124K ATC2.hi
 16K ATC2.hs
 16K ATC2.hs~
1.3M ATC2.o
5.1M ATC3
8.0K ATC3.hi
4.0K ATC3.hs
4.0K ATC3.o
4.0K Common
4.0K Makefile
4.0K Makefile~
 12K atc.2.tgz
4.0K programatica

[igloo]

OK, I'm closing this then. Please reopen if you think it's not fixed after all.

Thanks!

[maeder]

Rerunning the above example using ghc-6.8.3 the object file ATC2.o has now 3.5 MB instead of 1.3 MB for ghc-6.8.2

maeder@leibniz:/local/maeder/atc> ls -sh1
total 13M
308K ATC.hi
 88K ATC.hs
1.3M ATC.o
144K ATC2.hi
 16K ATC2.hs
 16K ATC2.hs~
3.5M ATC2.o
7.1M ATC3
8.0K ATC3.hi
4.0K ATC3.hs
4.0K ATC3.o
4.0K Common
4.0K Makefile
4.0K Makefile~
 12K atc.2.tgz
4.0K programatica

[maeder]

changing all instances like the following from

instance (ShATermConvertible c,
          ShATermConvertible t) => ShATermConvertible (Q c t) where
    toShATermAux att0 ((:=>) a b) = do
        (att1, a') <- toShATerm' att0 a
        (att2, b') <- toShATerm' att1 b
        return $ addATerm (ShAAppl "(:=>)" [a',b'] []) att2
    fromShATermAux ix att0 =
        case getShATerm ix att0 of
            ShAAppl "(:=>)" [a,b] _ ->
                    case fromShATerm' a att0 of { (att1, a') ->
                    case fromShATerm' b att1 of { (att2, b') ->
                    (att2, (:=>) a' b') }}
            u -> fromShATermError "Q" u

to

_toShATermAux_Q att0 ((:=>) a b) = do
    (att1, a') <- toShATerm' att0 a
    (att2, b') <- toShATerm' att1 b
    return $ addATerm (ShAAppl "(:=>)" [a',b'] []) att2
_fromShATermAux_Q ix att0 =
    case getShATerm ix att0 of
        ShAAppl "(:=>)" [a,b] _ ->
                case fromShATerm' a att0 of { (att1, a') ->
                case fromShATerm' b att1 of { (att2, b') ->
                (att2, (:=>) a' b') }}
        u -> fromShATermError "Q" u
instance (ShATermConvertible c,
          ShATermConvertible t) => ShATermConvertible (Q c t) where
    toShATermAux = _toShATermAux_Q
    fromShATermAux = _fromShATermAux_Q

avoids the blow up

[simonpj]

Adding data from Christian:

• For ghc-6.8.2 our compressed binary hets.bz2 shrunk from 7.2 MB to 6.6 MB

• For ghc-6.8.3 from 13 MB to 6.8 MB

(some manual instance are left in our code)

 http://www.informatik.uni-bremen.de/agbkb/forschung/formal_methods/CoFI/hets/linux/daily/  http://www.informatik.uni-bremen.de/agbkb/forschung/formal_methods/CoFI/hets/linux/versions/

[simonpj]

I believe this bug is fixed now, with the new HEAD. I've changed the compilation of instances so that it's just as if you'd written separate definitions for each method, much as you have now done by hand.

If you could test that claim, it'd be great.

Simon

[maeder]

Could you point me to a suitable snapshot distribution? The latest one from http://www.haskell.org/ghc/dist/current/dist/ seems to be ghc-6.9.20080822-src.tar.bz2

Christian

[maeder]

using atc.2.tgz the object code size for ghc-6.9.20080910-x86_64-unknown-linux is as follows:

 16K ATC2.hi
 16K ATC2.hs
324K ATC2.o
8,0K ATC3.hi
4,0K ATC3.hs
8,0K ATC3.o
248K ATC.hi
 88K ATC.hs
1,2M ATC.o

compared with our x64 ghc-6.8.3 installation (only consider ATC2.o!) that is a significant improvement.

 16K ATC2.hi
 16K ATC2.hs
9,2M ATC2.o
 12M ATC3
 12K ATC3.hi
4,0K ATC3.hs
8,0K ATC3.o
504K ATC.hi
 88K ATC.hs
2,2M ATC.o

Linking with ghc-6.9.20080910 did not work (and I also got more warnings)

...
WARNING: file simplCore/SimplCore.lhs line 501
Simplifier still going after 4 iterations; bailing out.  Size = 1248
...
[108 of 110] Compiling ATC              ( ATC.hs, ATC.o )
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6zB} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6zB} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6zB} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6zB} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6yT} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6yT} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6yT} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6yT} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6zB} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6zB} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6zB} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6zB} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6zB} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6zB} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6zB} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v s6zB} [lid] 0 0 [L, S, L]
[109 of 110] Compiling ATC2             ( ATC2.hs, ATC2.o )

ATC2.hs:13:15:
    Warning: Imported from `HsName' but not used:
               data constructor `UnQual'
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v X6J6} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v X6J6} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v X6Ja} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v X6Ja} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v X6Ko} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v X6Ko} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v X6Kd} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v X6Kd} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v X6L1} [lid] 0 0 [L, S, L]
WARNING: file simplCore/SimplUtils.lhs line 386
More demands than arity a9{v X6L1} [lid] 0 0 [L, S, L]
[110 of 110] Compiling Main             ( ATC3.hs, ATC3.o )

ATC3.hs:2:0:
    Warning: Module `ATC2' is imported, but nothing from it is used,
               except perhaps instances visible in `ATC2'
             To suppress this warning, use: import ATC2()
Linking ATC3 ...
/usr/lib64/gcc/x86_64-suse-linux/4.2.1/../../../../x86_64-suse-linux/bin/ld: cannot find -lffi
collect2: ld returned 1 exit status
make: *** [all] Fehler 1

[maeder]

just for the record I include the sizes for our oldest x64 ghc-6.6.1 installation that again show that they are particularly bad for ghc-6.8.3

 12K ATC2.hi
 16K ATC2.hs
3,5M ATC2.o
8,0M ATC3
8,0K ATC3.hi
4,0K ATC3.hs
4,0K ATC3.o
340K ATC.hi
 88K ATC.hs
1,8M ATC.o

[maeder]

Sorry, my previous comment is wrong. The above ghc-6.6.1 is a 32bit installation.

[maeder]

As shown in the above comment the ticket can be closed.

[simonpj]

Just to say that I've now fixed those odd WARNINGs too.

Wed Sep 17 17:29:10 BST 2008  simonpj@microsoft.com
  * Fix nasty infelicity: do not short-cut empty substitution in the simplifier
  
  I was perplexed about why an arity-related WARN was tripping. It took 
  me _day_ (sigh) to find that it was because SimplEnv.substExpr was taking
  a short cut when the substitution was empty, thereby not subsituting for
  Ids in scope, which must be done (CoreSubst Note [Extending the Subst]).
  
  The fix is a matter of deleting the "optimisation".  Same with
  CoreSubst.substSpec, although I don't know if that actually caused a
  problem.

Great to have this ticket closed at last.

Simon