can't do I/O in the child of forkProcess with -threaded

[simonmar]

forkProcess kills all threads, as advertised. Unfortunately this includes the I/O manager thread, so trying to do I/O in the child of forkProcess leads to deadlock. It might be possible to fix this, but we'd need to add a way to restart the I/O manager thread.

[tibbe]

I need this to be able to do I/O in a forked process in order to implement a daemonized mode for my web server. I need to do something like:

server :: IO ()

main =
  if daemonize
  then forkProcess $ server
  else server

I also need to do some I/O before forking (i.e. call getArgs to get command line flags to pass to GetOpt) to figure out if I should daemonize the process or not. Note that my web server is a library which serve method will be called by user code so I can't provide a wrapper script that forks the process since I don't have a binry to call.

[simonmar]

I'd advise against this. forking is a total pain in a multi-threaded program, and I really don't want to commit to making it work for anything other than simple fork/exec type usage (and possibly not even that - doing the fork/exec in C is better).

Do you really need both -threaded and forkProcess? Can you fork in C before starting up the Haskell runtime?

[tibbe]

It doesn't work without -threaded either!

My library exports the following function:

serve :: Application -> IO ()

Where Application is a handler type function in the spirit of Request -> Response IO.

The intended usage is for the user who wants to build a web application to import my library and write a function e.g.

fileServer :: Application

and have a main that looks something like:

main = serve fileServer

The insides of the library are in principle those of HWS. serve reads lots of configuration flags such as --port and --daemonize from the command line so the user can get a functioning web application server that he or she can deploy in production with little fuzz.

I can't see how I can support this interface and still fork in C. It seems to me like the user needs to write the startup code in C and call main from it. Not a nice user experience.

If there is a way to write a C library that forks and include it in my library and still retain the same API I will of course use it as an intermediate solution but I still would prefer to be able to fork processes from inside Haskell. Forking, even in a restricted version, is something that would be useful in Haskell. I think daemonizing processes is a perfect example of where it's needed. I also think it is a necessary feature for server type applications.

I understand that this might be difficult from a technical perspective, yet it is something other languages have so it must be possible. If I understood the issue better I could try to help come up with a solution and also try to implement it.

[igloo]

Replying to simonmar:

I'd advise against this. forking is a total pain in a multi-threaded program, and I really don't want to commit to making it work for anything other than simple fork/exec type usage (and possibly not even that - doing the fork/exec in C is better). Do you really need both -threaded and forkProcess? Can you fork in C before starting up the Haskell runtime?

I really think we ought to be able to say something like

main = do args <- getArgs
          configFile <- readConfigFile
          let config = args .+. configFile
          if runAsDaemon config
              then daemonise realMain
              else realMain

(where daemonise does the standard double-forking thing) without having to resort to writing C code. Using forkIO, forkOS etc before daemonise wouldn't necessarily have to be supported.

[tibbe]

Implementing igloo's suggestion will solve my current problem. I'm still interested in workarounds that would allow me (by jumping through some hoops) to export the same API but internally daemonize the process using some C code. Note that I need to executed some Haskell code (that is single threaded) before I can invoke the C code.

[simonmar]

Replying to tibbe:

It doesn't work without -threaded either!

Then there could be a real bug here - can you give us complete reproduction instructions? We certainly intend to support forkProcess in the non-threaded RTS.

[simonmar]

Here's the sample code that demonstrates the problem, from  http://www.haskell.org/pipermail/glasgow-haskell-users/2007-March/012062.html, with an extra threadDelay inserted to make the hang show up more reliably:

module Main where

import Control.Concurrent
import System.Posix
import System.IO
import System.Exit

main =
    do putStrLn "running..."
       (stdinr, stdinw) <- createPipe
       (stdoutr, stdoutw) <- createPipe
       pid <- forkProcess $ do hw <- fdToHandle stdoutw
                               hr <- fdToHandle stdinr
                               closeFd stdinw
                               hGetContents hr >>= hPutStr hw
                               hClose hr
                               hClose hw
                               exitImmediately ExitSuccess
       threadDelay 100000
       closeFd stdoutw
       closeFd stdinw
       hr2 <- fdToHandle stdoutr
       hGetContents hr2 >>= putStr
       getProcessStatus True False pid >>= print

[bsdemon]

What is the status of this issue? I see it marked as new, but will it be fixed for 6.12 or later? I am interested in daemonizing threaded programs.

[simonmar]

It's not likely to be fixed in 6.12.1, I'm afraid. If there's a lot of interest we can schedule it for 6.12.2.

[bsdemon]

For me, as mostly being network programmer, it is "must have" feature.

[kazu-yamamoto]

This is a MUST. Without forkProcess which does not kill the IO thread, we cannot create even a simple deamon nor Apache's prefok model, etc...

[bsdemon]

Maybe a stupid question, but...

After reading sources, it seems that GHC.Conc.ensureIOManagerIsRunning is exported from module. Why we cannot use it for starting I/O Manager after forkProcess?

[simonmar]

Replying to bsdemon:

Maybe a stupid question, but... After reading sources, it seems that GHC.Conc.ensureIOManagerIsRunning is exported from module. Why we cannot use it for starting I/O Manager after forkProcess?

Unfortunately it's not that simple. The IO manager is started by an unsafePerformIO in the definition of pendingEvents, and ensureIOManagerIsRunning just seqs the thunk, so it won't do anything to call it after forking.

I think we probably have to keep track of the IO manager thread explicitly in the RTS.

[simonmar]

I have a patch that seems to work; will tidy up and commit soon.

[bsdemon]

It is awesome. I was trying to fix it by myself but failed, mostly due to the lack of knowledge of how RTS works. So, please, include revision number of that bug fix, when it will be applied, in comment to this ticket. I want to take a look. Thanks!

[simonmar]

Fixed, eventually. Patches to GHC:

Wed Nov 11 14:28:22 GMT 2009  Simon Marlow <marlowsd@gmail.com>
  * Second attempt to fix #1185 (forkProcess and -threaded)

  Patch 1/2: second part of the patch is to libraries/base
  
  This time without dynamic linker hacks, instead I've expanded the
  existing rts/Globals.c to cache more CAFs, specifically those in
  GHC.Conc.  We were already using this trick for signal handlers, I
  should have realised before.
  
  It's still quite unsavoury, but we can do away with rts/Globals.c in
  the future when we switch to a dynamically-linked GHCi.

Thu Nov 12 12:58:53 GMT 2009  Simon Marlow <marlowsd@gmail.com>
  * Windows-specific fix for #1185 patch

and for libraries/base:

Wed Nov 11 15:19:15 GMT 2009  Simon Marlow <marlowsd@gmail.com>
  * Second attempt to fix #1185 (forkProcess and -threaded)

We should merge this into 6.12.1, partly because there is a lot of demand for forkProcess, and also because it fixes an infelicity in GHCi whereby a new IO manager thread was being created for each :load/:reload.

[simonmar]

One more patch

Fri Nov 13 02:45:18 PST 2009  Simon Marlow <marlowsd@gmail.com>
  * The rest of the #1185 patch (forkProcess and -threaded)
  Ignore-this: fef0329743c91910dcd38734bb502649
  Due to darcs confusion, I managed to leave out part of the patch for
  #1185.  This should make 1185(threaded1) go through now.

[igloo]

The "One more patch" seems to already be applied, but I've merged the rest and the test now passes.