nvfetcher: Generate nix sources expr for the latest version of packages

[ library, mit, nix, program ] [ Propose Tags ]

Please see README

[Skip to Readme]
Versions [RSS] [faq],,
Change log CHANGELOG.md
Dependencies aeson (>=1.5.6 && <1.6), base (>=4.8 && <5), binary, binary-instances (>=1.0.1 && <1.1), bytestring, containers, data-default (>=0.7.1 && <0.8), extra (>=1.7.9 && <1.8), free (>=5.1.5 && <5.2), microlens, microlens-th, neat-interpolation (>=0.5.1 && <0.6), nvfetcher, optparse-simple (>=0.1.1 && <0.2), parsec, shake (>=0.19.4 && <0.20), text, tomland (>=1.3.2 && <1.4), transformers, unordered-containers, validation-selective [details]
License MIT
Copyright 2021 berberman
Author berberman
Maintainer berberman <berberman.yandex.com>
Category Nix
Home page https://github.com/berberman/nvfetcher
Bug tracker https://github.com/berberman/nvfetcher/issues
Source repo head: git clone https://github.com/berberman/nvfetcher.git
Uploaded by berberman at 2021-06-08T08:41:33Z
Distributions NixOS:
Executables example, nvfetcher
Downloads 123 total (10 in the last 30 days)
Rating (no votes yet) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Hackage Matrix CI
Docs uploaded by user [build log]
All reported builds failed as of 2021-06-08 [all 2 reports]


[Index] [Quick Jump]



Build example executable


Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info


Maintainer's Corner

For package maintainers and hackage trustees


Readme for nvfetcher-

[back to package description]


Hackage MIT license nix

nvfetcher is a tool to automate packages updates in flakes repos. It's built on top of shake, integrating nvchecker. nvfetcher cli program accepts a TOML file as config, which defines a set of package sources to run.


For example, given the following configuration file:

# nvfetcher.toml
src.pypi = "feeluown"
fetch.pypi = "feeluown"

src.github = "IsoaSFlus/QLivePlayer"
fetch.github = "IsoaSFlus/QLivePlayer"
git.fetchSubmodules = true

running nvfetcher build will create sources.nix like:

# sources.nix
{ fetchgit, fetchurl }:
  feeluown-core = {
    pname = "feeluown-core";
    version = "3.7.7";
    src = fetchurl {
      sha256 = "06d3j39ff9znqxkhp9ly81lcgajkhg30hyqxy2809yn23xixg3x2";
      url = "https://pypi.io/packages/source/f/feeluown/feeluown-3.7.7.tar.gz";
  qliveplayer = {
    pname = "qliveplayer";
    version = "3.22.1";
    src = fetchgit {
      url = "https://github.com/IsoaSFlus/QLivePlayer";
      rev = "3.22.1";
      fetchSubmodules = true;
      deepClone = false;
      leaveDotGit = false;
      sha256 = "00zqg28q5xrbgql0kclgkhd15fc02qzsrvi0qg8lg3qf8a53v263";

We tell nvfetcher how to get the latest version number of packages and how to fetch their sources given version numbers, and nvfetcher will help us keep their version and prefetched SHA256 sums up-to-date, stored in sources.nix. Shake will help us handle necessary rebuilds -- we check versions of packages during each run, but only prefetch them when needed.

sources.nix is a symlink of _build/generated.nix, which means that the "build" results produced by nvfetcher are under _build, so you have to keep this directory for further use.

Live examples

How to use the generated sources file? Here are some examples:


nvfetcher package is available in nixpkgs, so you can try it with:

$ nix-shell -p nvfetcher

This repo also has flakes support:

$ nix run github:berberman/nvfetcher

To use it as a Haskell library, the package is available on Hackage. If you want to use the Haskell library from flakes, there is also a shell ghcWithNvfetcher:

$ nix develop github:berberman/nvfetcher#ghcWithNvfetcher
$ runghc Main.hs

where you can define packages in Main.hs. See Haskell library for details.


Basically, there are two ways to use nvfetcher, where the difference is how we provide package sources definitions to it.


To run nvfetcher as a CLI program, you'll need to provide package sources defined in TOML.

Available options:
  --version                Show version
  --help                   Show this help text
  -c,--config FILE         Path to nvfetcher TOML config
                           (default: "nvfetcher.toml")
  -o,--output FILE         Path to output nix file (default: "sources.nix")
  -l,--changelog FILE      Dump version changes to a file
  -j NUM                   Number of threads (0: detected number of processors)
                           (default: 0)
  -r,--retry NUM           Times to retry of some rules (nvchecker, prefetch,
                           nix-instantiate, etc.) (default: 3)
  -t,--timing              Show build time
  -v,--verbose             Verbose mode
  TARGET                   Two targets are available: 1.build 2.clean
                           (default: "build")

Each package corresponds to a TOML table, whose name is encoded as table key, with two required fields and three optional fields in each table. You can find an example of the configuration file, see nvfetcher_example.toml.


Version source -- how do we track upstream version updates?

  • src.github = owner/repo - the latest gituhb release
  • src.github_tag = owner/repo - the max github tag, usually used with list options (see below)
  • src.pypi = pypi_name - the latest pypi release
  • src.git = git_url (and an optional src.branch = git_branch) - the latest commit of a repo
  • src.archpkg = archlinux_pkg_name -- the latest version of an archlinux package
  • src.aur = aur_pkg_name -- the latest version of an aur package
  • src.manual = v -- a fixed version, which never updates
  • src.repology = project:repo -- the latest version from repology
  • src.webpage = web_url and src.regex -- a string in webpage that matches with regex
  • src.httpheader = request_url and src.regex -- a string in http header that matches with regex

Optional list options for some version sources (src.github_tag, src.webpage, and src.httpheader), see the corresponding nvchecker documentation for details.

  • src.include_regex
  • src.exclude_regex
  • src.sort_version_key
  • src.ignored

Optional global options for all kinds of version sources, see the corresponding nvchecker documentation for details. You can tweak obtained version number using this option, e.g. stripping the prefix v or transforming the result by regex.

  • src.prefix
  • src.from_pattern
  • src.to_pattern

Nix fetcher

How do we fetch the package source if we have the target version number? $ver is available in string, which will be set to the result of nvchecker.

  • fetch.github = owner/repo
  • fetch.pypi = pypi_name
  • fetch.git = git_url
  • fetch.url = url

Optional nix-prefetch-git config, which make sense only when the fetcher equals to fetch.github or fetch.git. They can exist simultanesouly.

  • git.deepClone
  • git.fetchSubmodules
  • git.leaveDotGit

Extract src

Optional extract src config, files are extracted into build directory, and then read by readFile in generated nix expr.

  • extract = [ "file_1", "file_2", ...] - file paths are relative to the source root

Rust support

rustPlatform.buildRustPackage now accepts an attribute cargoLock to vendor dependencies from Cargo.lock, so we can use this instead TOFU cargoSha256 for Rust packageing. nvfetcher supports automating this process, extracting the lock file to build and calculating cargoLock.outputHashes, as long as you set the config

  • cargo_lock = cargo_lock_path - relative to the source root

Haskell library

nvfetcher itsetlf is a Haskell library as well, whereas the CLI program is just a trivial wrapper of the library. You can create a Haskell program depending on it directly, by using the runNvFetcher entry point. In this case, we can define packages in Haskell language, getting rid of TOML constraints.

You can find an example of using nvfetcher in the library way, see Main_example.hs.


For details of the library, documentation of released versions is available on Hackage, and of master is on our github pages.


There is no way to check the equality over version sources and fetchers, so If you change either of them in a package, you will need to rebuild everything, i.e. run nvfetcher clean to remove shake databsae, to make sure that our build system works correctly. We could automate this process, for example, calculate the hash of the configuration file and bump shakeVersion to trigger the rebuild. However, this shouldn't happen frequently and we want to minimize the changes, so it's left for you to do manually.

Adding or removing a package doesn't require such rebuild


Issues and PRs are always welcome. _(:з」∠)_

Building from source:

$ git clone https://github.com/berberman/nvfetcher
$ nix develop
$ cabal build