symlink binaries into ~/bin

We should do this soon.

Sat Aug  2 03:56:54 BST 2008  Duncan Coutts <duncan@haskell.org>
  * Add feature to symlink binaries into another dir
  This covers the major part of ticket #289.

Sat Aug  2 03:57:30 BST 2008  Duncan Coutts <duncan@haskell.org>
  * Add a symlink-bindir field to the config file
  and link it up with the symlink feature code.

Sat Aug  2 14:39:22 BST 2008  Duncan Coutts <duncan@haskell.org>
  * Warn about symlinks that could not be created

What remains to be done is to change the default config.

Replying to duncan:

Currently cabal-install uses --prefix=$HOME/.cabal and does not specify --bindir so we use $HOME/.cabal/bin as the bindir. Obviously this is not on the $PATH by default.

of course, neither is any other dir in $HOME in PATH by default, I just add $HOME/.cabal/bin to my PATH :-) ...in addition to using GNU Stow elsewhere to keep track of my non-Cabal stuff.

If the name does exist we could give a warning and let the user sort it out.

okay, by telling them (if they want it) what symlink to make? or having a cabal "update-bin-symlinks" maybe?

Of course we need to be able to overwrite our own symlinks. We know it's a symlink we're allowed to overwrite if it links into $HOME/.cabal/bin.

yep, that's the same heuristic that Stow uses -- pretty good. Hmm.. watch out, for people who use Stow itself, Stow might leave bin dir being a ("folded") symlink to somewhere you don't want to install to, such as $HOME/stow/darcs-2.0.2/bin ;-)

It would be nice to install into $HOME/.cabal/bin using versioned binary names (Cabal supports that now)

true!

and symlink the generic name into $HOME/bin.

I want the generic names somewhere well-defined under $HOME/.cabal too. And the versioned names should also be symlinked.

and there are no symlinks on windows.

:-( With greater hacks, we could generate wrapper binaries/scripts on Windows that just load the thing in $HOME/.cabal/bin, I suppose -- but maybe Windows users are used to the lack of symlinks?

I'd imagine we'd have it in the ~/.cabal/config file something like: {{{ prefix: $HOME/.cabal -- bindir: $HOME/.cabal/bin symlink-bindir: $HOME/bin }}}

I guess defaulting to $HOME/bin "if it exists as a non-symlink directory" would be a heuristic that's not too unreasonable if you want a silent default. (silence is what makes people like me not know about the existence of $HOME/.cabal/bin until we find out about it, too!) (Which reminds me, yet another reason not to put anything of my own making under $HOME, because programs might automatically trample it for their own heuristic purposes, thinking they know what's best for me and doing it behind my back without my permission -- I keep my $HOME in /Users/me/HOME, which might be called /home/isaac/HOME under more-normal circumstances)

-Isaac

Replying to Isaac Dupree:

Most distros set things up so that ~/bin is in the $PATH by default (if it exists).

Why would you want unversioned names under ~/.cabal/bin if that is not on your $PATH ?

I don't think we should ignore a ~/bin just because it is a symlink. Users who don't want the feature can easily turn it off in the ~/.cabal/config.

I'm not sure what you mean about trampling, we're being pretty careful not to trample anything. By default we only touch ~/.cabal and are careful with setting symlinks in ~/bin so that we do not overwrite anything created by the user or any other program.

Replying to duncan:

Replying to Isaac Dupree: Most distros set things up so that ~/bin is in the $PATH by default (if it exists).

hmm, I didn't realize. At least, I've never noticed that happening under Gentoo, GoboLinux? or Ubuntu (then again, I don't think I actually had a ~/bin when I was on Gentoo, I put things in ~/u/bin back then).

Why would you want unversioned names under ~/.cabal/bin if that is not on your $PATH ?

(1. It's on my $PATH)

2. so that I can explicitly invoke ~/.cabal/bin/alex without giving a version number, maybe? What I don't want is the "unversioned" feature to be combined in an essential way with another feature that I don't want. I suppose I could put my symlink-bindir as $HOME/.cabal/unversioned-bin, but that seems like it would be messing with cabal's area a bit without its permission.

I don't think we should ignore a ~/bin just because it is a symlink. Users who don't want the feature can easily turn it off in the ~/.cabal/config.

Only if you tell anyone who installs a binary that they can turn off the feature there, and how, and how to clean up the mess that cabal has already made... (unless you ask the user first)

I'm not sure what you mean about trampling, we're being pretty careful not to trample anything. By default we only touch ~/.cabal and are careful with setting symlinks in ~/bin so that we do not overwrite anything created by the user or any other program.

It's because I'm paranoid about that, because I prefer to have full control over what is in my home filesystem structure, separate from what programs want to automatically store. True, the proposed system won't delete any important data, so the worst that can happen is that I don't notice for a while, it causes a little havoc when some GHC build uses a version of Alex that I didn't expect (e.g. if I installed Alex through my system's package manager), and eventually I have to figure out what happened, clean up the mess, and if I'm feeling lucky, find out how I can turn it off in the cabal config file. This is no figment of my imagination -- I only went to such a weird-looking scheme because of repeated problems with miscellaneous programs that I maybe only used once (it became even worse on Ubuntu where things run even when you don't explicitly configure them to, AND make a mess in $HOME). By all means, go ahead and do things automatically in Cabal for the users that like that, because I've lost all trust in that sort of thing and I've found a way to avoid it (so far, I've never had a problem with different programs messing with $HOME in ways that they break each other automatically -- that would be more difficult to deal with :-). Grudging thanks for reminding me never to put anything of my own in $HOME, unless it's modifying a config-file in $HOME/.something that can't be stored elsewhere.

er, sorry for the rants, did it explain anything? I think if all programs had always been as polite as Cabal is planning to be, I might have appreciated it doing that automatically...

Anyway, which version are you planning to symlink as the unversioned one? The one with the greatest version? Is there a way to modify that, that'll last longer than just changing the symlink target?

-Isaac

Replying to Isaac Dupree:

(1. It's on my $PATH)

2. so that I can explicitly invoke ~/.cabal/bin/alex without giving a version number, maybe? What I don't want is the "unversioned" feature to be combined in an essential way with another feature that I don't want. I suppose I could put my symlink-bindir as $HOME/.cabal/unversioned-bin, but that seems like it would be messing with cabal's area a bit without its permission.

I guess you can tell cabal to put symlinks in ~/.cabal/bin instead of ~/bin and to use versioned binaries. Would that do?

The features are quite separate. You can install versioned binaries or not. You can add symlinks to some other dir or not. (You're not restricted to versioned binaries, you can specify arbitrary prefixes and suffixes on binaries and they can contain vars like the package name, version, compiler, etc).

I don't think we should ignore a ~/bin just because it is a symlink. Users who don't want the feature can easily turn it off in the ~/.cabal/config.

Only if you tell anyone who installs a binary that they can turn off the feature there, and how, and how to clean up the mess that cabal has already made... (unless you ask the user first)

Mess what mess? There will be some symlinks in ~/bin pointing to ~/.cabal/bin. Using ls -l they will be easily identifiable if you want to delete them.

I'm not sure what you mean about trampling, we're being pretty careful not to trample anything. By default we only touch ~/.cabal and are careful with setting symlinks in ~/bin so that we do not overwrite anything created by the user or any other program.

It's because I'm paranoid about that, because I prefer to have full control over what is in my home filesystem structure, separate from what programs want to automatically store. True, the proposed system won't delete any important data, so the worst that can happen is that I don't notice for a while, it causes a little havoc when some GHC build uses a version of Alex that I didn't expect (e.g. if I installed Alex through my system's package manager), and eventually I have to figure out what happened, clean up the mess, and if I'm feeling lucky, find out how I can turn it off in the cabal config file.

This should just work. You have /usr/bin in your $PATH before $HOME/bin so the alex you get will be the one installed by your system package manager, not the one installed by Cabal.

This is no figment of my imagination -- I only went to such a weird-looking scheme because of repeated problems with miscellaneous programs that I maybe only used once (it became even worse on Ubuntu where things run even when you don't explicitly configure them to, AND make a mess in $HOME). By all means, go ahead and do things automatically in Cabal for the users that like that, because I've lost all trust in that sort of thing and I've found a way to avoid it (so far, I've never had a problem with different programs messing with $HOME in ways that they break each other automatically -- that would be more difficult to deal with :-). Grudging thanks for reminding me never to put anything of my own in $HOME, unless it's modifying a config-file in $HOME/.something that can't be stored elsewhere.

Ok, but do you have any specific problems with what we're proposing? If so what is your suggestion for the best default behavior?

er, sorry for the rants, did it explain anything? I think if all programs had always been as polite as Cabal is planning to be, I might have appreciated it doing that automatically...

So you mean this sounds ok?

Anyway, which version are you planning to symlink as the unversioned one? The one with the greatest version? Is there a way to modify that, that'll last longer than just changing the symlink target?

The one that was just requested to be installed. No way to modify it yet, no, except by changing the symlink yourself. This is no worse than before where the binaries would just overwrite each other. Of course you can not symlink and just put $HOME/.cabal/bin on your $PATH and use versioned binaries in there in which case all versions will be available all the time.

We're not really so concerned with people who know what they're doing, except to the extent that we do not annoy them. What we want is for things to "Just Work"(tm) for users who don't know and don't care. The current behavior of installing just to ~/.cabal/bin fails that test.

Replying to duncan:

I guess you can tell cabal to put symlinks in ~/.cabal/bin instead of ~/bin and to use versioned binaries. Would that do?

if it works, it's fine with me

Mess what mess? There will be some symlinks in ~/bin pointing to ~/.cabal/bin. Using ls -l they will be easily identifiable if you want to delete them.

yes, that mess. And then there's always the risk that they come back next time (next time I use Cabal, that is), unless I found out how to disable it in the config, changed it, and didn't make any typoes or other mistakes. (Changing a config file is not the sort of thing you get instant feedback on.)

This should just work. You have /usr/bin in your $PATH before $HOME/bin

no I don't, I have $HOME/bin earlier. Usually when I install things locally it's because the package-manager version is older and I want to use the newer version! For completeness, here's my current $PATH:

$ echo $PATH
/Users/me/inst/bin:/Users/me/scripts:/Users/me/HOME/bin:/Users/me/HOME/.cabal/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/Users/me/inst/stow/fish-darcs20080706+open-help-fix/bin:/usr/X11R6/bin

Ok, but do you have any specific problems with what we're proposing? If so what is your suggestion for the best default behavior?

I suggest that if the config file is empty, then if anything looks suspicious ($HOME/bin is not in PATH or is a symlink), you ask the user what they want to do, and record that choice in the config file. Actually it'd probably be a service to ask even if they have no $HOME/bin, and create it if they want one, and otherwise record otherwise in the config file. Is it a problem, being interactive? Or simpler: just ask any time the config file doesn't say anything about it and you're installing a binary.

er, sorry for the rants, did it explain anything? I think if all programs had always been as polite as Cabal is planning to be, I might have appreciated it doing that automatically...

So you mean this sounds ok?

okay as long as no one else complains about it -- and certainly aids discoverability, because anyone who looks in their $HOME/bin will implicitly be able to find out about the existence of $HOME/.cabal!

We're not really so concerned with people who know what they're doing, except to the extent that we do not annoy them. What we want is for things to "Just Work"(tm) for users who don't know and don't care. The current behavior of installing just to ~/.cabal/bin fails that test.

Yeah, that's why we need to worry about what happens for users that don't have a $HOME/bin currently. And if we ever create $HOME/bin ourselves, about distros that don't add $HOME/bin to $PATH, because e.g. modifying files like .bashrc isn't going to be very reliable. I noticed I have a directory "$HOME/perl", probably from using CPAN sometime, but I don't remember if I chose that directory-name or it did:

> ls $HOME/perl/
bin/  lib/
> ls $HOME/perl/bin/
config_data*  ptar*  ptardiff*  pwhich*  shasum*  ysh*
# and $HOME/perl/lib/perl5/(lots of directories and stuff)

Probably the best we can do is output a message explaining the situation whenever (or the first time) we install a binary, possibly with the addition of some choices. Probably the message needs to be different on Windows too (If I was suddenly on Windows, I would need a helpful message like that :-)

-Isaac

For what it's worth, I agree with what I think Isaac is saying: Cabal should not put things in ~/bin unless I explicitly tell it to.

Anything under ~/.cabal/ is fine, so for example you could put things in ~/.cabal/bin/ and recommend that I add that to my path.

Replying to igloo:

For what it's worth, I agree with what I think Isaac is saying: Cabal should not put things in ~/bin unless I explicitly tell it to. Anything under ~/.cabal/ is fine, so for example you could put things in ~/.cabal/bin/ and recommend that I add that to my path.

For distros that put ~/.bin on the $PATH I think we're failing unless we can make things just work. For systems not set up that way we'll have to ask or tell the user what to do.

Remember most Ubuntu users don't even know what a $PATH is let along how to change one. Ubuntu sets things up such that if the ~/bin dir exists then it gets put on the $PATH however it doesn't create the dir by default for new accounts. For Fedora it's apparently always on the $PATH even if it does not exist.

So when we first set up the user's default ~/.cabal/config we should check if ~/bin is on the $PATH. If not we should tell the user that they need to take some action to make the executables available (either to put ~/bin or ~/.cabal/bin on the $PATH).

It should also print out what configuration its chosen the first time you run it, when it creates ~/.cabal/config so people know what it's going to do and what file to edit.

I know you think we should never touch ~/bin at all, but I think not working by default for standard distro setups is not a good option.

I rather suspect we'll have to do different things on different platforms. For example OSX does not help us with per-user bin dirs at all, though it does have /usr/local/bin and cabal-install can use a sudo style command, so perhaps that's a sensible default on OSX.

I'll try and get this issue of defaults discussed on some mailing list.

Does anything else automatically install into ~/bin? I always assumed that it being in the path by default was just to make things easier for users who wanted to put things there.

Incidentally, I don't think you shouldn't touch it /at all/, but that you shouldn't touch it without being explicitly told you can (e.g. "I see you haven't made a .cabal/config yet. Do you want me to make one for you? [Yn] Do you want me to automatically symlink binaries into ~/bin?" [Yn]).

It can now be set on the command line as well as the config file.

The remaining issue is just the question of defaults.

Replying to igloo:

Does anything else automatically install into ~/bin? I always assumed that it being in the path by default was just to make things easier for users who wanted to put things there. Incidentally, I don't think you shouldn't touch it /at all/, but that you shouldn't touch it without being explicitly told you can (e.g. "I see you haven't made a .cabal/config yet. Do you want me to make one for you? [Yn] Do you want me to automatically symlink binaries into ~/bin?" [Yn]).

As we've already seen, ~/bin is by no means a standard (I use ~/local/bin myself, so ~/bin would be pointless for me). But typically when you tell a build system to "install", you expect it's going to put its stuff where it needs to to be found. Fortunately for the Haskell libraries this can be in cabal's own private area, but binaries don't have that luxury. Compare GNU Automake: when you run ./configure without --prefix, it assumes /usr/local. When you run "sudo make install" you know full well it's putting stuff in system directories (polite makefiles will also uninstall, which is nice). Or if you don't have superuser access, it gives you an error message saying it can't write where it wants to, and the user needs to either run it again as su, or else re./configure with a new, writable, prefix. And this intentional re-prefixing means he'll pick somewhere that he *knows* has a bindir in the path.

I point all this out because as I see it, cabal-install is broken out of the box. A naive user (like myself a few weeks ago) will download Cabal and cabal-install, along with their dependencies, from Hackage, and install cabal-install (which ends up in /usr/local by default, a reasonably sane place). He'll be excited that packages now install painlessly and won't bother looking at any of the config files, because things basically seem to "just work". Then, whenever cabal-install attempts to meet a binary dependency, it appears to run in an endless loop, repeatedly installing the same software but with no effect. In my case, I didn't even realize Alex was a binary, so I installed it easily ten times before I realized what was happening - I imagine many others might take much longer. In any case, not one warning or error was raised during this time (other than "can't find Alex", which I thought I'd just installed), because the current warning is only issued when there's a symlink-bindir specified, when by default there is none.

I sent in a couple patches (one to Cabal and one to cabal-install) to address some of these issues, but I'm not sure if anyone saw them. I can resend them if anyone would like. But I'll summarize what I think should be done:

1. cabal-install needs to either

• be more pushy in the user's homedir -- people don't like this,
  
• be more stubborn with its default prefix, like automake, or
  
• issue clear warning messages *out of the box* when binaries are installed outside the path (this is what I implemented in the patch I sent).

2. Cabal should find some way to look inside cabal-install's bindir whether it's on the PATH or not. I implemented this in a somewhat clunky way in the other patch I sent (which I'm almost positive didn't make it onto cabal-devel), but it could be done better (at least on Posix systems - do Windows systems have similar issues, or is this a Unix-only problem?) with System.Posix.setEnv: cabal-install can just insert its bindir into the PATH before passing control along to Cabal.