Bazel is a great tool for describing monorepo builds. This is especially the case when one has to deal with more than one technology or language within the same repository. In this blog post I am going to demonstrate how to convert the Makefile build of example-servant-elm to Bazel. Well… maybe not all at once. In the first post of this series, I’d like to focus only on the Haskell components of the example-servant-elm project. The following post will cover the Bazel build for the front-end (Elm) component.
This blog post is, after a fashion, an extension of the talk given by Andreas Herrmann during ZuriHac 2021. I want to slightly modify Andreas’ approach here, especially the part which relates to the Haskell component (backend server), and utilize a feature that was recently added to the gazelle_cabal project—support for the Cabal sub-libraries. All the code from this blog post can be found here.
example-servant-elm project structure
Before we start, it will be helpful to analyze the actual structure of the example-servant-elm project:
$ tree
.
├── assets
├── client
│ ├── elm.json
│ ├── GenerateElm.hs
│ ├── Makefile
│ ├── src
│ │ └── Main.elm
│ └── tests
│ └── Tests.elm
├── example-servant-elm.cabal
├── Makefile
├── package.yaml
├── README.md
├── server
│ ├── src
│ │ ├── Api.hs
│ │ ├── App.hs
│ │ └── Main.hs
│ └── test
│ ├── AppSpec.hs
│ └── Spec.hs
├── stack.yaml
└── stack.yaml.lockpackage.yamlfile - hpack description for Haskell componentsexample-servant-elm.cabalfile - generated byhpackfrom the package.yaml fileserverdirectory - backend code written in Haskellclientdirectory - frontend code written in Elm with an additional Haskell fileassetsdirectory - a place where all the web components end up (e.g. theindex.htmlfile)
It is clear that the Haskell code is split into two logically separated components—
the server and the client—but there is only a single .cabal file in this
project. The Haskell code inside the client directory generates Elm bindings
that call the Haskell API code from the Elm frontend application. Therefore,
server/src/Api.hs is a dependency for both client/GenerateElm.hs and
server/src/Main.hs.
With this in mind, the Makefile-to-Bazel conversion process will involve first creating
a sub-library for server/src/Api.hs in the server component,
and subsequently reusing it for both the Main.hs and GenerateElm.hs binaries.
We can automatically generate the Bazel build definition by utilizing the
gazelle_cabal extension, which will create all the
rules_haskell entries for us.
Haskell components—The server and… the client
Adjusting hpack/Cabal definitions
In order to use gazelle_cabal, the client/GenerateElm.hs component must be
taken into account by Cabal, which is not yet the case. Since the Cabal file is
generated by hpack, package.yaml must be extended. This is achieved by adding a shared
library for the API, and another executable for the Elm generation, resulting
in:
name: example-servant-elm
dependencies:
- ...
internal-libraries:
api:
source-dirs:
- server/src
exposed-modules: Api
other-modules: []
executables:
server:
main: Main.hs
source-dirs:
- server/src
dependencies: api
other-modules: [App]
generate-elm:
main: GenerateElm.hs
source-dirs:
- client/
dependencies: api
other-modules: []Running hpack against the updated package.yaml gives us what we expect—an
example-servant-elm.cabal file with two executables depending on a single internal
library:
$ hpack && cat example-servant-elm.cabal
cabal-version: 2.0
-- This file has been generated from package.yaml by hpack version 0.34.4.
--
-- see: https://github.com/sol/hpack
name: example-servant-elm
version: 0.0.0
build-type: Simple
library api
exposed-modules:
Api
hs-source-dirs:
server/src
build-depends:
...
default-language: Haskell2010
executable generate-elm
main-is: GenerateElm.hs
hs-source-dirs:
client/
build-depends:
...
, api
...
default-language: Haskell2010
executable server
main-is: Main.hs
other-modules:
App
hs-source-dirs:
server/src
build-depends:
...
, api
...
default-language: Haskell2010
...The api component is called an internal library (or sub-library) in Cabal’s
nomenclature, and the relevant section of the cabal file can be additionally annotated with the visibility parameter, indicating whether
the library can be used only in this package, or also in other Cabal packages.
By default the visibility is private. This is taken into account by
gazelle_cabal, which will generate suitable rules_haskell entries.
The generation of BUILD.bazel files will have the following flow:
We can alias this build command as follows:
$ alias gen-build-files="hpack && bazel run :gazelle"Playing with gazelle_cabal
Now we can create a WORKSPACE file in the example-servant-elm root directory,
and add all the preambles required for gazelle_cabal, rules_haskell
and other dependencies.
workspace(name = "example-servant-elm")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "io_tweag_gazelle_cabal",
strip_prefix = "gazelle_cabal-main",
url = "https://github.com/tweag/gazelle_cabal/archive/main.zip",
sha256 ="49c9bda91ae2867fa45dbc6f5c04d0ee0146dd433ba7ab9694ed2914a092e817",
)
...
http_archive(
name = "rules_haskell",
sha256 = "851e16edc7c33b977649d66f2f587071dde178a6e5bcfeca5fe9ebbe81924334",
strip_prefix = "rules_haskell-0.14",
urls = ["https://github.com/tweag/rules_haskell/archive/v0.14.tar.gz"],
)
load("@rules_haskell//haskell:repositories.bzl", "rules_haskell_dependencies")
load("@rules_haskell//haskell:cabal.bzl", "stack_snapshot")
rules_haskell_dependencies()
...
stack_snapshot(
name = "stackage",
packages = [
"path", #keep
"path-io", #keep
"aeson" #keep
],
snapshot = "lts-18.12",
)
Following gazelle_cabal’s README, we add the required Gazelle targets to the build file at the root of the example-servant-elm directory:
$ cat BUILD.bazel
load("@bazel_gazelle//:def.bzl", "gazelle")
load(
"@bazel_gazelle//:def.bzl",
"DEFAULT_LANGUAGES",
"gazelle_binary",
)
gazelle(
name = "gazelle",
data = ["@io_tweag_gazelle_cabal//cabalscan"],
gazelle = ":gazelle_binary",
)
gazelle_binary(
name = "gazelle_binary",
languages = DEFAULT_LANGUAGES + ["@io_tweag_gazelle_cabal//gazelle_cabal"],
)
gazelle(
name = "gazelle-update-repos",
command = "update-repos",
data = ["@io_tweag_gazelle_cabal//cabalscan"],
extra_args = [
"-lang",
"gazelle_cabal",
"stackage",
],
gazelle = ":gazelle_binary",
)Finally, it’s time to use gazelle_cabal to automatically generate all the rules_haskell
targets in our repository. On the very first run it will fail due to
of Paths_* modules being required by the spec test suite. This
requirement is automatically inserted by hpack (and the Paths_*
files are automatically generated by Cabal). In this project, we don’t
actually need the Paths_* files. We can get rid of the requirement by setting
other-modules in the spec target section of package.yaml explicitly.
$ cat package.yaml
...
tests:
spec:
...
other-modules: [AppSpec, Api, App]
...
$ #################################################################
$ gen-build-files
generated example-servant-elm.cabal
INFO: Analyzed target //:gazelle (1 packages loaded, 3 targets configured).
INFO: Found 1 target...
Target //:gazelle up-to-date:
bazel-bin/gazelle-runner.bash
bazel-bin/gazelle
INFO: Elapsed time: 0.161s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
$ #################################################################
$ cat BUILD.bazel
...
# rule generated from example-servant-elm.cabal by gazelle_cabal
haskell_library(
name = "api",
srcs = ["server/src/Api.hs"],
compiler_flags = ["-DVERSION_example_servant_elm=\"0.0.0\""],
version = "0.0.0",
visibility = ["//visibility:public"],
deps = [
"@stackage//:aeson",
...
"@stackage//:warp",
],
)
# rule generated from example-servant-elm.cabal by gazelle_cabal
haskell_binary(
name = "generate-elm",
srcs = ["client/GenerateElm.hs"],
compiler_flags = ["-DVERSION_example_servant_elm=\"0.0.0\""],
version = "0.0.0",
visibility = ["//visibility:public"],
deps = [
":api",
...
],
)
# rule generated from example-servant-elm.cabal by gazelle_cabal
haskell_binary(
name = "server",
srcs = [
"server/src/App.hs",
"server/src/Main.hs",
],
compiler_flags = ["-DVERSION_example_servant_elm=\"0.0.0\""],
version = "0.0.0",
visibility = ["//visibility:public"],
deps = [
":api",
...
],
)
# rule generated from example-servant-elm.cabal by gazelle_cabal
haskell_test(
name = "spec",
srcs = [
"server/src/Api.hs",
"server/src/App.hs",
"server/src/Main.hs",
"server/test/AppSpec.hs",
"server/test/Spec.hs",
],
compiler_flags = ["-DVERSION_example_servant_elm=\"0.0.0\""],
version = "0.0.0",
visibility = ["//visibility:public"],
deps = [
...
],
)So far, so good. It turns out that gazelle_cabal has generated a bunch of rules_haskell
targets within the BUILD.bazel file. It created exactly what was
expected: two binary targets—generate-elm and server—which depend on the api target.
Moreover, there’s also the spec target, which is responsible for test execution.
Building the :api, :generate-elm and :server targets
Our next step is to run/build each one of the generated targets to verify that
all of them work as expected. We will start with the :api target. It fails on the
first run, so we have to iterate and fix the errors.
$ bazel build :api
ERROR: example-servant-elm/BUILD.bazel:33:16: no such target '@stackage//:elm-bridge': target 'elm-bridge' not declared in package '' defined by /home/kczulko/.cache/bazel/_bazel_kczulko/0544c71977bc4fd7972aa4a6b74ea529/external/stackage/BUILD.bazel and referenced by '//:api'
ERROR: example-servant-elm/BUILD.bazel:33:16: no such target '@stackage//:servant': target 'servant' not declared in package '' defined by /home/kczulko/.cache/bazel/_bazel_kczulko/0544c71977bc4fd7972aa4a6b74ea529/external/stackage/BUILD.bazel and referenced by '//:api'
...
INFO: Elapsed time: 0.579s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (3 packages loaded, 0 targets configured)Essentially, we need to add all the dependencies from package.yaml under the
dependencies section of the stack_snapshot repository defined in the
WORKSPACE file. Fortunately, we don’t have to add those manually but can instead
invoke gazelle_update_repos to generate the imports. Running the
command below automatically populates stack_snapshot with all the required
dependencies like wai, warp, aeson etc:
$ bazel run :gazelle-update-repos && cat WORKSPACE
...
stack_snapshot(
name = "stackage",
...
packages = [
"aeson", #keep
"base",
"containers",
"elm-bridge",
"hspec",
"hspec-discover",
"http-client",
"http-types",
"path", #keep
"path-io", #keep
"servant",
"servant-client",
"servant-elm",
"servant-server",
"text",
"transformers",
"wai",
"warp",
],
snapshot = "lts-18.12",
)
...At this point we can update the gen-build-files alias to include
:gazelle-update-repos as the last step of BUILD.bazel files generation:
$ alias gen-build-files="hpack && \
bazel run :gazelle && \
bazel run :gazelle-update-repos"The next step is to add precise versions to the dependencies which require them. All of these dependencies
can be found in the stack.yaml file - elm-bridge, servant-elm, wai-make-assets.
To prevent gazelle-update-repos from removing them, we have to mark those with gazelle’s
built-in #keep directive. This is basically a comment that will be interpreted by the
gazelle engine, which we have to add next to the dependency name inside the stack_snapshot
bazel workspace rule. Those two actions should populate the packages array in
stack_snapshot as follows:
stack_snapshot(
name = "stackage",
packages = [
"aeson", #keep
...
"elm-bridge",
"elm-bridge-0.6.0", #keep
...
"servant-elm",
"servant-elm-0.7.2", #keep
...
"wai-make-assets-0.2", #keep
"warp",
],
snapshot = "lts-18.12",
)Evidently, some of those dependencies are quasi-duplicated (e.g. elm-bridge and elm-bridge-0.6.0).
In this case, rules_haskell will unify all quasi-duplicates and fetch the specified version from Hackage. We can use unversioned labels in
rules_haskell targets, like haskell_library or haskell_binary, to refer to these libraries. So, instead of having to use elm-bridge-0.6.0
in the build definition, we can simply refer to elm-bridge.
The next problem we encounter is quite typical for Haskell builds:
$ bazel build :api
...
Setup.hs: Missing dependency on a foreign library:
* Missing (or bad) header file: zlib.h
* Missing (or bad) C library: z
...We need zlib. Since this setup is using nix, the easiest way to include
zlib in this build is to copy the configuration from the gazelle_cabal
WORKSPACE file (for example), and insert it under the stack_snapshot repository rule.
stack_snapshot(
...
extra_deps = {"zlib" : ["@zlib.dev//:zlib"]},
...
)
...
nixpkgs_package(
name = "nixpkgs_zlib",
attribute_path = "zlib",
repository = "@nixpkgs",
)
nixpkgs_package(
name = "zlib.dev",
build_file_content = """
...
""",
repository = "@nixpkgs",
)Now the :api target should build without errors:
$ bazel build :api
/nix/store/7whk47dml5f1dpvy6cgvaf80ll1h7pkd-zlib-1.2.11-dev
/nix/store/bl1kc9hw119ly7if07m2pbqak2yhars8-zlib-1.2.11
INFO: Analyzed target //:api (6 packages loaded, 4303 targets configured).
INFO: Found 1 target...
Target //:api up-to-date:
bazel-bin/libHSapi.a
bazel-bin/libHSapi-ghc8.10.4.so
INFO: Elapsed time: 611.149s, Critical Path: 551.46s
INFO: 196 processes: 125 internal, 71 linux-sandbox.
INFO: Build completed successfully, 196 total actionsThe next target we’ll work on is :generate-elm, which should build successfully
after removing the redundant imports from GenerateElm.hs.
The :server target builds successfully without any manual intervention.
Thank you gazelle_cabal! Almost
all automatically generated targets work without manual intervention. However,
when we run this target, it immediately terminates. We’ll return to that problem later.
$ bazel run :server
...
INFO: Build completed successfully, 1 total action
server: missing directory: 'client/'
Please create 'client/'.
(You should put sources for assets in there.)Building the :spec target
The last Bazel target to evaluate is :spec. On the first run
it’s going to fail with the following error:
Use --sandbox_debug to see verbose messages from the sandbox
ghc: could not execute: hspec-discover
Target //:spec failed to buildThis means that we have to add hspec-discover to the build-tool-depends section of
example-servant-elm.cabal file, just like in this example.
Since the Cabal file is automatically generated, we are required to add a build-tools
section to package.yaml, and to regenerate BUILD.bazel.
$ cat package.yaml
...
tests:
spec:
...
build-tools:
- hspec-discover
...
$ gen-build-files && cat BUILD.bazel
...
haskell_test(
name = "spec",
...
compiler_flags = [
"-DVERSION_example_servant_elm=\"0.0.0\"",
"-DHSPEC_DISCOVER_HSPEC_DISCOVER_PATH=$(location @stackage-exe//hspec-discover)",
],
tools = ["@stackage-exe//hspec-discover"],
...
)gazelle_cabal has introduced the HSPEC_DISCOVER_HSPEC_DISCOVER_PATH variable, so we can reuse
it in the server/Spec.hs file to let GHC know where to find hspec-discover.
This is a good step toward hermeticity in comparison to what the server/Spec.hs file contained previously.
Finally, server/Spec.hs should contain the following:
{-# LANGUAGE CPP #-}
{-# OPTIONS_GHC -F -pgmF HSPEC_DISCOVER_HSPEC_DISCOVER_PATH #-}There is also some additional cleaning up required in server/test/AppSpec.hs,
otherwise compilation will fail due to name shadowing and unused-imports warnings.
After that, we can finally try to run the :spec target:
$ gen-build-files && bazel run :spec
...
Executing tests from //:spec
-----------------------------------------------------------------------------
App
app
/api/item
returns an empty list FAILED [1]
/api/item/:id
returns a 404 for missing items FAILED [2]
POST
allows to create an item FAILED [3]
lists created items FAILED [4]
lists 2 created items FAILED [5]
DELETE
allows to delete items FAILED [6]
Failures:
server/test/AppSpec.hs:30:7:
1) App.app./api/item returns an empty list
uncaught exception: ErrorCall
missing directory: 'client/'
Please create 'client/'.
(You should put sources for assets in there.)
...There are two things worth observing now:
- The target compiles—that’s obviously great!
- All the tests fail—let’s search for where “client” is used.
With regard to the second observation, it turns out that the string “client” is
used in the server/src/App.hs options method:
$ bat server/src/App.hs
...
18 │
19 │ options :: Options
20 │ options = Options "client"
21 │
...Indeed, options is used by the server function, and the server
function is used by the app function, and the app function is imported
and used in server/test/AppSpec.hs. Whew… we’ve got it! It turns
out to be used to generate and load assets with the
serveAssets function. The serveAssets function
calls make to generate those assets, but our assets are actually
static, so let’s use the simpler wai-app-static package instead:
$ git diff server/src/App.hs
diff --git a/server/src/App.hs b/server/src/App.hs
index 203a6c7..e0a674e 100644
--- a/server/src/App.hs
+++ b/server/src/App.hs
@@ -7,6 +7,7 @@ import Control.Monad.IO.Class
import Data.Map
import Network.Wai
-import Network.Wai.MakeAssets
+import Network.Wai.Application.Static (staticApp, defaultFileServerSettings)
import Servant
import Api
@@ -16,15 +17,14 @@ type WithAssets = Api :<|> Raw
withAssets :: Proxy WithAssets
withAssets = Proxy
-options :: Options
-options = Options "client"
+assets :: Application
+assets = staticApp $ defaultFileServerSettings "assets"
app :: IO Application
app = serve withAssets <$> server
server :: IO (Server WithAssets)
server = do
- assets <- serveAssets options
db <- mkDB
return (apiServer db :<|> Tagged assets)Of course, we also need to add wai-app-static under the dependencies
list within the package.yaml file.
...
tests:
spec:
...
dependencies:
...
- http-types
- wai-app-staticThis also means that we can remove the wai-make-assets-0.2 #keep entry from the
stack_snapshot packages list, as well as from the package.yaml file.
Let’s regenerate the build definitions and check whether the tests pass…
$ gen-build-files && bazel test :spec
...
INFO: Build completed successfully, 2 total actions
//:spec PASSED in 0.0s
Executed 1 out of 1 test: 1 test passes.The “client” error sounds familiar—we encountered it before in the case of the :server target.
Let’s check whether :server still terminates:
$ bazel run :server
...
INFO: Build completed successfully, 10 total actions
listening on port 3000...We’re home and dry. It looks like getting rid of wai-make-asset did the job. The :spec and :server
targets work, and all the tests passed.
Summary
We’ve shown how quickly you can switch your build from the combination of
Cabal/make/hpack to Bazel. We now have one tool, and one approach to maintain the
build process of example-servant-elm. Thanks to gazelle_cabal it was almost fully
automated—especially for a few simple Cabal build definitions.
In the next post we’re going to “bazelify” the Elm part of this repository. Stay tuned!
About the author
If you enjoyed this article, you might be interested in joining the Tweag team.
