literate markdown not handled correctly by unlit

[guest]

This simple program in literate haskell, using markdown in the comments gives unlit problems:

### Ok so lets try this again.

### A page that loads and compiles:

> myfact 0 = 1  
> myfact n = n * n-1

Lets see if it works!

If I run unlit and collect the output I can see where it went wrong:

$ ~/lib/ghc-7.0.1/unlit Main.lhs Main.lpp
$ cat Main.lpp
### Ok so lets try this again.

### A page that loads and compiles:

  myfact 0 = 1  
  myfact n = n * n-1

When I look through the source code of unlit.c I think the place to check for this would be here:

    if ( c == '#' ) {
      if ( ignore_shebang ) {
         c1 = egetc(istream);
         if ( c1 == '!' ) {
           while (c=egetc(istream), !isLineTerm(c)) ;
           return SHEBANG;
         }
         myputc(c, ostream);
         c=c1;
      }
      if ( leavecpp ) {
        myputc(c, ostream);
        while (c=egetc(istream), !isLineTerm(c))
            myputc(c,ostream);
        myputc('\n',ostream);
        return HASH;
      }
    }

It seems that cabal has a similar unlit function: http://www.haskell.org/ghc/docs/latest/html/libraries/Cabal-1.10.0.0/src/Distribution-Simple-PreProcess-Unlit.html#unlit

I haven't tested it but, I think the cabal version would handle this case correctly (or be easier to fix than a C program from 1990). Would it be possible/wise/feasible to extract the cabal version and make it a permanent replacement for the current unlit.c code?

[nalaurethsulfate]

In addition to the cabal version perhaps the perl script mentioned in the obscure unlit.c README reference ( http://www.desy.de/user/projects/LitProg/glasgow/programs-and-options.html, lit2stuff) could be called with the correct options to remove the comments from literate Haskell files.

[nalaurethsulfate]

while it might be easier to fix the cabal program also handles the same test case incorrectly:

GHCi, version 6.12.1: http://www.haskell.org/ghc/ :? for help

Loading package ghc-prim ... linking ... done.

Loading package integer-gmp ... linking ... done.

Loading package base ... linking ... done.

Prelude> :m Distribution.Simple.PreProcess?.Unlit

Prelude Distribution.Simple.PreProcess?.Unlit> f <- readFile "test.lhs"

Prelude Distribution.Simple.PreProcess?.Unlit> f

"### Ok so lets try this again.\n\n### A page that loads and compiles:\n\n> myfact 0

1 \n> myfact n

n * n-1\n\nLets see if it works!\n"

Prelude Distribution.Simple.PreProcess?.Unlit> unlit "log.txt" f

Loading package array-0.3.0.0 ... linking ... done.

Loading package containers-0.3.0.0 ... linking ... done.

Loading package filepath-1.1.0.3 ... linking ... done.

Loading package old-locale-1.0.0.2 ... linking ... done.

Loading package old-time-1.0.0.3 ... linking ... done.

Loading package unix-2.4.0.0 ... linking ... done.

Loading package directory-1.0.1.0 ... linking ... done.

Loading package pretty-1.0.1.1 ... linking ... done.

Loading package process-1.0.1.2 ... linking ... done.

Loading package Cabal-1.8.0.2 ... linking ... done.

Left "### Ok so lets try this again.\n\n### A page that loads and compiles:\n\n myfact 0 = 1 \n myfact n = n * n-1\n\n -- Lets see if it works!\n\n"

Prelude Distribution.Simple.PreProcess?.Unlit>

I don't think that this is terribly surprising though, and shouldn't be to difficult to fix. If someone could please explain why CPP lines wouldn't be in code blocks (no matter how they are delimited) that would help a lot.

[duncan]

So the problem here is that ghc does unlit before cpp and so it has to pass the #cpp directives through. It has to do unlit before cpp because in the worst case the only time ghc finds out cpp is needed is when it encounters a {-# LANGUAGE CPP #-} pragma.

In principle I suppose that ghc could unlit with cpp passthrough only for the pass where it reads the module head to find pragmas, and then if cpp is not required to re-unlit the file without the cpp passthrough mode.

Technically this probably does count as H98 non-compliance. The CPP extension interferes with the use of # in ordinary (non-cpp) lhs files.

[simonmar]

See also #4073 and #3719

Thanks Duncan for pointing out one good reason why we need to do unlit before CPP.

[jmg]

I've run into this problem when trying to use org-mode markup in a lhs file. This bug prevents me from using quite a lot of org-mode specific in-file settings. They all start with a '#' in the first column.

[elliottt]

I've implemented markdown processing in a branch, and just discovered this ticket. Is there a reason that CPP is preserved in the comment part of a literate file? Shouldn't it be that CPP is only preserved when it shows up in either a birdtrack or \begin{code} ... \end{code} block? Preserving it in the comment portion of a literate program seems akin to processing CPP that is in the comments of a non-literate file; I would normally expect commented-out CPP to not be run.