🫸 Updo 🫷

A hairstyle and a way of maintaining and upgrading Haskell projects.

What is a Project?

Updo is good for project packages, constraints, source repository packages and stackage resolver¹. More than this can be added into a project template.

Goals

1. Map a single set of packages, constraints and source repository packages to projects for cabal and stack.
2. Minimize the effort of upgrading constraints, source repository packages and GHC versions.

We do this with configuration in .dhall files, makefiles and dhall text templates. We also have some simple Haskell scripts for operations not possible in Dhall like sorting a List Text².

Template Outputs

We generate projects from the configured inputs using templates. With the dhall text command we're able to mirror to both stack and cabal projects using text templates. See templates for more.

We provide one stack template dhall2stack. The dhall2cabal template for cabal is so similar that the generated outputs can be compared using file diff tooling. However, we prefer the dhall2config template because it retains package groups that can bring order to an upgrade if the cabal solving order is roughly captured by package group ordering.

We provide one level of nesting for grouping packages together as inputs. Groups are preserved or flattened depending on the template.

• dhall2stack - Stack projects cannot import so necessarily must lose the grouping and contain the list of packages.
• dhall2cabal - This template lists packages within the cabal project.
• dhall2config - When generating the cabal project we keep package groups, generating the same tree structure but replacing the leaves. From each group-name.dhall we generate a group-name.config file.

It is nice to be able to look at packages both grouped and as one sorted list. Find the flat sorted lists of packages used by some templates in the .updo folder.

.updo
├── pkgs-sorted.dhall        ▨ List Text
└── pkgs-upgrade-done.dhall  ▨ List Text

If you don't want groups then please put all packages in the one group.

Maintaining a Project

Maintaining a project involves configuring or reconfiguring inputs and then using make to generate the projects. What do we want to configure? We don't need much:

1. A stackage resolver¹.
2. A list of constraints (package dependencies with their versions).
3. A list of source repository packages, for unpublished³ packages or for unpublished versions of packages.
4. A list of packages.

All configuration goes into ./project-dhall (where . is the root folder for your Haskell project) except for the cabal.config that we'll need to download from stackage. Save this file as project-stackage/lts-m.n.config using the exact resolver name.

project-stackage
└── lts-m.n.config

For each ghc-x.y.z compiler version, create this set of inputs and templates:

project-dhall
└── ghc-x.y.z
    ├── constraints.dhall      ▨ List { dep : Text, ver : Text }
    ├── deps-external.dhall    ▨ List { loc : Text, tag : Text, sub : List Text }
    ├── deps-internal.dhall    ▨ List { loc : Text, tag : Text, sub : List Text }
    ├── forks-external.dhall   ▨ List { loc : Text, tag : Text, sub : List Text }
    ├── forks-internal.dhall   ▨ List { loc : Text, tag : Text, sub : List Text }
    └── text-templates
        ├── dhall2cabal.dhall  ▨ template for ghc-x.y.z.dhall2cabal.project
        ├── dhall2config.dhall ▨ template for ghc-x.y.z.dhall2config.project
        ├── dhall2stack.dhall  ▨ template for ghc-x.y.z.dhall2stack.yaml
        └── stacksnippet.dhall ▨ anything for ghc-x.y.z.dhall2stack.yaml

Anything in stacksnippet.dhall gets added to the top of the generated stack project⁴. The rest of the files are inputs.

1. Stackage Resolver and Constraints

The stackage resolver is able to provide us with a list of packages that work together for both stack and cabal. Any other dependencies as constraints or source packages are either not found on stackage or are different versions.

By specifying a resolver with stack we get a list of exact == versions for packages published to stackage. Stackage also publishes a cabal.config file that has these package versions as cabal constraints. As cabal constraints are additive, in practice importing directly from stackage often leads to conflicts between stackage's constraints and the project's own constraints and for that reason we'll usually have to download the cabal.config from stackage⁵ and comment out lines that are conflicted.

We don't expect any packages to impose constraints on their dependencies at the package level, in their .cabal files, but if they do then these constraints must fit with constraints at the project level.

2. Constraints

In constraints⁶ put published packages that you want to use that are not on stackage or if they are on stackage where you want to use a different version.

[ { dep = "diagrams-postscript", ver = "1.5" }
, { dep = "diagrams-svg", ver = "1.4.3" }
]

The type of constraints is a list of records with dependency and version fields:

List { dep : Text, ver : Text }

3. Source Repository Packages

There are various reasons to depend on source packages and forks of source packages; to keep a mirror within your organization, to add fixes that haven't been upstreamed yet, or to pick up a fix someone else has made but hasn't yet been merged upstream, released and published.

project-dhall
└── ghc-x.y.z
    ├── deps-external.dhall    ▨ List { loc : Text, tag : Text, sub : List Text }
    ├── deps-internal.dhall    ▨ List { loc : Text, tag : Text, sub : List Text }
    ├── forks-external.dhall   ▨ List { loc : Text, tag : Text, sub : List Text }
    └── forks-internal.dhall   ▨ List { loc : Text, tag : Text, sub : List Text }

The type of these files are all the same. They contain a list of records with fields for location, tag (commit SHA) and subdirectories:

List { loc : Text, tag : Text, sub : List Text }

Any empty sub field must be type annotated like this [] : List Text. Each source repository package record will bring in one dependency or as many as there are sub fields.

These source repository *.dhall files are for original work from your organisation (internal) and from third parties (external) and for internal and external forks. If using the empty list in any one of these files then use an explicit type annotation:

[] : List { loc : Text, tag : Text, sub : List Text }

4. Package Groups

In the pkgs folder, create one or more groups for related packages.

project-dhall
├── ghc-x.y.z
├── pkgs
│   ├── db.dhall      ▨ List Text
│   ├── server.dhall  ▨ List Text
│   └── tools.dhall   ▨ List Text
└── pkg-groups.dhall  ▨ List Text

The contents of each group is a List Text of relative paths to folders containing package .cabal files.

-- ./project-dhall/pkgs/tools.dhall
[ "tool/linter/"
, "tool/formatter/"
]

List the package groups in pkg-groups.dhall⁷. A good order to use is the order of cabal constraint solving that you may witness when upgrading a project and seeing solving failures.

-- ./project-dhall/pkg-groups.dhall
[ "tools"
, "db"
, "server"
]

This gives you control of the imports in the generated ./project-cabal/pkgs.config, itself imported into ghc-x.y.z.dhall2config.project:

• Use CABAL_RELATIVITY=ImportRelative to be import-compatible with cabal-3.10.

  -- ./project-cabal/pkgs.config
  import: ./pkgs/tools.config
  import: ./pkgs/db.config
  import: ./pkgs/server.config
  

• The default of CABAL_RELATIVITY=ProjectRelative is import-compatible with cabal-3.8.

  -- ./project-cabal/pkgs.config
  import: ./project-cabal/pkgs/tools.config
  import: ./project-cabal/pkgs/db.config
  import: ./project-cabal/pkgs/server.config
  

-- ./ghc-x.y.z.dhall2config.project
import: ./project-stackage/lts-m.n.config

import: ./project-cabal/pkgs.config

import: ./project-cabal/ghc-x.y.z/constraints.config
import: ./project-cabal/ghc-x.y.z/deps-external.config
import: ./project-cabal/ghc-x.y.z/deps-internal.config
import: ./project-cabal/ghc-x.y.z/forks-external.config
import: ./project-cabal/ghc-x.y.z/forks-internal.config

Upgrading a Project

GHC Upgrade

For a GHC compiler upgrade, add another folder to project-dhall for the upgrade from ghc-u.v.w to ghc-x.y.z:

project-dhall
├── ghc-u.v.w
├── ghc-x.y.z
├── pkgs
├── pkg-groups.dhall        ▨ List Text
└── pkgs-upgrade-todo.dhall ▨ List Text

At the start of a GHC upgrade put all packages into project-dhall/pkgs-upgrade-todo.dhall. As the upgrade progresses remove packages from this list as you work on them.

Constraint Upgrade

For a dependency version upgrade, add or change an entry in constraints.dhall and if that dependency is on stackage for the resolver we're using and if the version there differs then comment out that entry in the downloaded project-stackage/resolver-name.config file to avoid impossible constraint solving conflicts with cabal.

Source Repository Upgrade

For a source repository upgrade, bump the tag field if picking up a newer version.

If the tag you were on has been superseded by a published version then delete the entry and add a constraint unless the published version is already a match in the downloaded cabal.config file from stackage.

Adding a Fork

If you are forking then you'll want to add a record to fork-internal.dhall and then:

• If forking a stackage package: comment out the entry in the downloaded cabal.config
• If forking a hackage package: delete the matching entry in constraints.dhall

Using someone else's fork is the same except add the record to fork-external.dhall.

Unforking goes the other way, remove the fork-*.dhall entry and either fallback to the stackage version or another version for which you'll add an entry in constraints.dhall.

Getting Started

Dhall is a prerequisite.

As well as other make targets, updo can bootstrap itself by adding one of two recipes to project-files.mk:

UPDO_VERSION ?= 1.0.0
HACKAGE := http://hackage.haskell.org/package
UPDO_URL := ${HACKAGE}/updo-${UPDO_VERSION}/updo-${UPDO_VERSION}.tar.gz

1. When opting for the scripts:

updo/Makefile:
	rm -rf updo
	curl -sSL ${UPDO_URL} | tar -xz
	mv updo-* updo
	chmod +x $$(grep -RIl '^#!' updo)

2. When opting to install the included executables:

updo/Makefile:
	rm -rf updo
	curl -sSL ${UPDO_URL} | tar -xz
	cd updo-${UPDO_VERSION}
	stack install
	cd -
	mv updo-* updo

Instead of curl and tar, cabal get or stack unpack can unpack package sources.

Make Targets

In the root of your project, add two files.

.
├── project-files.mk
└── project-versions.mk

In project-versions.mk set variables for GHC and STACKAGE.

GHC_VERSION := u.v.w
GHC_UPGRADE := x.y.z
STACKAGE_VERSION := lts-j.k
STACKAGE_UPGRADE := lts-m.n

Here is a basic set up for project-files.mk:

include project-versions.mk
include updo/Makefile

Canonical Targets

We consider the following as canonical project names and no target is required to build them (they're built by the default all target):

$ make -f project-files.mk

• stack.yaml
• cabal.project
• stack.upgrade.yaml
• cabal.upgrade.project

We use CABAL_VIA and STACK_VIA makefile variables to decide which ghc-x.y.z prefixed projects will be copied to the default project names (cabal.project and stack.yaml). This way, Updo will create a single pair of projects for one version of GHC.

GHC-Prefixed Targets

We can build ghc-x.y.z projects explicitly:

$ make -f project-files.mk ghc-x.y.z.dhall2config.project
$ make -f project-files.mk ghc-x.y.z.dhall2cabal.project
$ make -f project-files.mk ghc-x.y.z.dhall2stack.yaml

Modified Time Travel

Be careful with modified time that make relies on. If you hand-modify a target so that it is newer than a prerequisite then make will not use its recipe to build the target. This is easy to do inadvertently with a git GUI tool when discarding changes or by doing a git restore <project-name>. As git advances the modified time, this is effectively the same as touch <project-name>. To recover from this situation, either;

• Remove the target with rm <project-name>.
• Run make with its --always-make flag but be careful with this if bootstrapping updo itself within the makefile.

Lock Files

For stack, the project and its lock are separate targets (.yaml and .yaml.lock).

$ make -f project-files.mk ghc-x.y.z.dhall2stack.yaml
dhall text --file project-dhall/ghc-x.y.z/project-stack.dhall > ghc-x.y.z.dhall2stack.yaml

$ make -f project-files.mk ghc-x.y.z.dhall2stack.yaml.lock
stack build --dry-run --stack-yaml ghc-x.y.z.dhall2stack.yaml

We don't provide a make target for cabal .project.freeze files, relying instead on a combination of constraints, those in the cabal.config downloaded from stackage and those we add ourselves to constraints.dhall. If you do want to freeze anyway there's a cabal command to generate a freeze file.

$ make -f project-files.mk ghc-x.y.z.dhall2cabal.project
dhall text --file project-dhall/ghc-x.y.z/project-cabal.dhall > ghc-x.y.z.dhall2cabal.project

Clean GHC-Prefixed Projects

There's a make target to remove all ghc-x.y.z prefixed projects:

$ make -f project-files.mk clean
rm -f ghc-*.stack.* ghc-*.dhall2config.* ghc-*.dhall2cabal.* ghc-*.dhall2stack.*
ghc-*.stack2cabal.* ghc-*.cabal2stack.* ghc-*.dhall2yaml2stack.* ghc-*.sha256map.nix

GHC-Prefixed Projects as Temporary

To treat ghc-x.y.z prefixed files as temporary, add lines like these to project-files.mk:

.INTERMEDIATE: ghc-$(GHC_VERSION).dhall2config.project
.INTERMEDIATE: ghc-$(GHC_VERSION).dhall2cabal.project
.INTERMEDIATE: ghc-$(GHC_VERSION).dhall2stack.yaml
.INTERMEDIATE: ghc-$(GHC_VERSION).sha256map.nix

SHA256 Map Generation Method

With this snippet from project-files.mk, we can switch between two methods of generating ghc-x.y.z.sha256map.nix.

# How to generate project-nix/ghc-$(GHC_VERSION)/sha256map.nix?
# This is copied from ghc-$(GHC_VERSION).sha256map.nix.
#  - false to generate from *.dhall inputs via sha256map.hs.
#  - true to generate from stack.yaml via sha256map.py.
SHA256MAP_VIA_PYTHON ?= false

# If true, generate the sha256map from the stack.yaml with python,
# overriding the recipe for this target.
ifeq ($(SHA256MAP_VIA_PYTHON), true)
ghc-$(GHC_VERSION).sha256map.nix: stack.yaml
	updo/project-nix/sha256map.py <$^ >$@
endif

The default is to use the sha256map.hs script to generate it⁸:

$ make -f project-files.mk ghc-x.y.z.sha256map.nix
echo \
  '[./project-dhall/ghc-x.y.z/deps-external.dhall
  , ./project-dhall/ghc-x.y.z/deps-internal.dhall
  , ./project-dhall/ghc-x.y.z/forks-external.dhall
  , ./project-dhall/ghc-x.y.z/forks-internal.dhall
  , ([] : List {loc : Text, tag : Text, sub : List Text})
  ]' \
  | ./updo/project-nix/sha256map.hs > ghc-x.y.z.sha256map.nix

To use the sha256map.py script instead that is much slower:

$ SHA256MAP_VIA_PYTHON=true \
  make -f project-files.mk project-nix/ghc-x.y.z/sha256map.nix
updo/project-nix/sha256map.py <stack.yaml >ghc-x.y.z.sha256map.nix
mkdir -p project-nix/ghc-x.y.z && cp ghc-x.y.z.sha256map.nix project-nix/ghc-x.y.z/sha256map.nix
rm ghc-x.y.z.sha256map.nix

You can read more about Updo Nix and its use with haskell.nix.

Installed Exes or Scripts

To use the executables of Updo's own package, by default, instead of its scripts, add these lines to project-files.mk:

SHA256MAP_HS_EXE ?= true
PKG_GROUPS_HS_EXE ?= true
PKGS_SORTED_HS_EXE ?= true
PKGS_UPGRADE_DONE_HS_EXE ?= true

Why Dhall for Configuration?

Dhall has excellent imports and is an intentionally limited typed programming language. We use dhall's text command to write cabal and stack projects using dhall text templating.

The dhall executable comes with a format command⁹. This helps in the maintenance of the various .dhall files.

An example of the limits of the language we encountered in Updo is not being able to compare anything but Natural values. Sorting something like List Text is not possible but there is great interop with Haskell so this can be done over there as we do with scripts.

Acknowledgements

I already knew Dhall well from maintaining hpack-dhall. Thanks Simon Hengel for suggesting I take over its maintenance.

FP Complete is the corporate sponsor of this tooling. Thanks to my collegues; Daniel Herrera Rendón for first seeing this could work, Neil Mayhew for makefile tips and tricks, Abel Sen for careful reviews and thoughtful suggestions and Eduardo Lemos Rocha for advocacy and enthusiasm.

Please contact me for any questions, problems, suggestions or the like. Maintenance of Updo will be coming from me and not FP Complete.

Updo grew to fill a need after every other way I could find to maintain projects was unsatisfactory for the specific problem I had, herding hundreds of packages and their dependencies into buildable projects over two compiler versions for both cabal and stack. The more I use it, the more I love it as a fast, simple and appropriate solution for maintaining Haskell projects.

Thanks FP Complete.

constraints:
    bson -_old-network
  , HaXml +splitbase