If you run HashiCorp Nomad in production, you already know the storage story is thinner than the one Kubernetes gets. There are fewer drivers, fewer people writing about the edge cases, and a lot of the tooling assumes a Kubernetes control plane sitting underneath. We hit the wall on this ourselves, so we built a way through it and we are open-sourcing the result: nomad-csi-driver.
It is a Container Storage Interface (CSI) driver built for Nomad, with no Kubernetes anywhere in the picture. This post covers why it exists, what it does, how we chose to build it, the gotchas worth knowing before you deploy it, and a short end-to-end walkthrough.
We were running democratic-csi for persistent storage on Nomad. It is a capable project, but it is written for a Kubernetes-first world, and using it on Nomad meant living with translation layers and assumptions that did not quite fit. When something broke, debugging it meant reasoning about a stack that was never really designed for the orchestrator we actually run.
We wanted something that was Nomad-native from the first line, spoke plain CSI, and covered the two storage backends we actually use every day:
So we wrote a single Go binary that does both, and nothing we do not need.
nomad-csi-driver is one binary and one container image. You pick the backend at launch with a --driver flag:
--driver=qnap (SAN) | --driver=local (ZFS) | |
|---|---|---|
| Storage | iSCSI block LUNs on a QNAP appliance | node-local thick ZFS zvols |
| Placement | reachable from any node | pinned to the node that owns the zvol |
| Snapshots | yes (QNAP LUN snapshot) | yes (ZFS @snap) |
| Clones | yes (full independent copy) | yes (send | recv) |
| Filesystems | ext4 / xfs (or raw block) | ext4 / xfs (or raw block) |
Both backends do the same core job: provision storage, attach it to a node, and mount it for your workload. The difference is only in how they get a block device in front of the kernel. QNAP does it over iSCSI and multipath; local does it with a ZFS zvol. After that point, the exact same format and mount code runs for both. You provision with nomad volume create, you snapshot and clone through nomad volume snapshot, and you get per-volume usage stats and Prometheus metrics out of the box.
A few decisions shaped the whole thing, and they are worth calling out because they are the reason it behaves the way it does.
One interface: CSI, and only CSI. Nomad has more than one way to attach storage now, including host volumes and the newer dynamic host volumes. We deliberately picked CSI as the single path. It is the orchestrator-standard interface, it works all the way back to our Nomad floor of v1.6.3, and it keeps us from coupling to Nomad internals that shift between releases. One interface means one mental model when something goes wrong at 2am.
The plugin is a control-plane shim, never in the data path. This is the part we are most happy with. The driver issues storage operations (provision, attach, mount, unmount) and then gets out of the way. The bytes your workload reads and writes never pass through the plugin process. The data plane lives in the host kernel: the kernel mount, the host’s iscsid, the host’s multipathd, the ZFS layer. Kill the plugin container and the iSCSI sessions, the multipath maps, and the mounts all keep running on the host.
The practical payoff is that you can upgrade, restart, or reschedule the plugin with running workloads untouched. No remounts, no I/O interruption, no forced restart of the jobs consuming the volumes. A rolling image bump is a non-event for anything already running. That is very different from drivers whose data plane lives inside the plugin process, where a restart leaves you with stale handles and forces every consumer to restart with it.
A shared format and mount layer. Because both backends converge on “here is a block device,” we wrote the format, mount, and idempotency logic once. An existing filesystem is never reformatted, a ZFS destroy is leaf-only and never recursive, and expansion is grow-only. Volumes are destroyed only when you explicitly run nomad volume delete. There is no background reconciler that can decide on its own to delete storage you did not ask it to touch.
None of these are bugs. They are design choices, and knowing them up front saves you a confused afternoon.
capacity_min = "1GiB", not "1000MB". Local ZFS is finer-grained (it rounds up to the zvol block size).zfs rollback or a QNAP revert still exists at the storage layer, but you do that out of band, outside Nomad.nomad volume delete is refused until you delete the snapshots first. Delete children, then the parent.host = "auto" for the least-provisioned capacity node, or an explicit node name). A consumer that mounts it gets scheduled onto that node automatically.--driver=local, controllers forward create and delete operations to the owning node over plain HTTP, authenticated by a shared secret in cleartext. There is no TLS in this release. Run that forwarding port on a trusted, isolated network segment and firewall it to the peer controllers. Do not expose it on a shared network. Anyone who can reach it can issue storage operations against any node.To keep this honest: the driver manages CSI volumes, not the storage systems underneath them. A couple of things are firmly on your side of the line.
If you already run QNAP and ZFS, none of this is new work. The driver just assumes that foundation exists and builds on top of it.
Here is the local (ZFS) backend end to end. The QNAP flow is nearly identical, it just uses the controller and node jobspecs instead of the single monolith job. The repo ships a complete, copy-pasteable examples/ set for both backends.
1. Deploy the plugin on every node.
nomad job run examples/local-monolith.nomad.hclFor the local backend this runs as a system job so it lands on every node, all under one plugin id. Before you run it, edit the jobspec to point at your image, your zpool name, and your cluster’s specifics.
2. Provision a volume.
The volume spec is small and every parameter has a sensible default:
id = "local-data"
name = "local-data"
type = "csi"
plugin_id = "local"
capacity_min = "1GiB"
capacity_max = "1GiB"
capability {
access_mode = "single-node-writer"
attachment_mode = "file-system"
}
# Params below are all optional--defaults come from controller job
parameters {
pool = "tank" # a zpool from your config allowlist
host = "auto" # "auto" = fewest-volumes node, or an explicit node name
fsType = "ext4" # ext4 | xfs
volblocksize = "16K" # set at create, immutable after
}nomad volume create examples/local-volume.hcl3. Run a workload that mounts it.
nomad job run examples/local-consumer.nomad.hclThe consumer mounts the volume at /data and stays up, so you can exec in and look around.
4. Verify the mount and check usage.
nomad alloc exec -job local-consumer df -h /data
curl -s localhost:9610/v1/volume-stats/local-data | jqThat stats endpoint reports per-volume usage (bytes, inodes, file and directory counts) keyed by the Nomad volume id you actually manage, and the same data is exposed as Prometheus gauges if you want it on a dashboard.
5. Tear down in reverse.
Stop the consumer first so the volume claim releases, then delete the volume:
nomad job stop -purge local-consumer && nomad volume delete local-dataThat is the whole loop: deploy, create, mount, verify, delete.
This is another piece of the same promise we make with everything at HonestHosting: no upsells, no games, just honest and simple hosting that agencies do not have to think about. Reliable persistent storage is part of that foundation, and we would rather run something we understand end to end than glue ourselves to a stack built for a different world. Open-sourcing it is the natural next step, because tools like this get better in the open.
The code, the full examples, and the deeper docs on the control-plane architecture, the metrics, and the security model all live in the repo:
github.com/honest-hosting/nomad-csi-driver
If you run Nomad and you have been quietly wishing the storage story were simpler, give it a look. And if you break it in an interesting way, we would genuinely like to hear about it.
We continue to work hard to bring you the best value for all of your WordPress hosting needs. If you have any questions, concerns, or would generally like to chat, feel free to contact us anytime. We want to make sure you get the most out of HonestHosting.io!