A CSI Storage Driver for Nomad (Because We Got Tired of Fighting Ours)

If you run HashiCorp Nomad in production, you already know the storage story is thinner than the one Kubernetes gets. There are fewer drivers, fewer people writing about the edge cases, and a lot of the tooling assumes a Kubernetes control plane sitting underneath. We hit the wall on this ourselves, so we built a way through it and we are open-sourcing the result: nomad-csi-driver.

It is a Container Storage Interface (CSI) driver built for Nomad, with no Kubernetes anywhere in the picture. This post covers why it exists, what it does, how we chose to build it, the gotchas worth knowing before you deploy it, and a short end-to-end walkthrough.

Why we built it

We were running democratic-csi for persistent storage on Nomad. It is a capable project, but it is written for a Kubernetes-first world, and using it on Nomad meant living with translation layers and assumptions that did not quite fit. When something broke, debugging it meant reasoning about a stack that was never really designed for the orchestrator we actually run.

We wanted something that was Nomad-native from the first line, spoke plain CSI, and covered the two storage backends we actually use every day:

• A QNAP appliance serving iSCSI block LUNs to the cluster.
• Node-local ZFS, for workloads that want fast local disk without a SAN in the path.

So we wrote a single Go binary that does both, and nothing we do not need.

What it does

nomad-csi-driver is one binary and one container image. You pick the backend at launch with a --driver flag:

	--driver=qnap (SAN)	--driver=local (ZFS)
Storage	iSCSI block LUNs on a QNAP appliance	node-local thick ZFS zvols
Placement	reachable from any node	pinned to the node that owns the zvol
Snapshots	yes (QNAP LUN snapshot)	yes (ZFS @snap)
Clones	yes (full independent copy)	yes (send | recv)
Filesystems	ext4 / xfs (or raw block)	ext4 / xfs (or raw block)

Both backends do the same core job: provision storage, attach it to a node, and mount it for your workload. The difference is only in how they get a block device in front of the kernel. QNAP does it over iSCSI and multipath; local does it with a ZFS zvol. After that point, the exact same format and mount code runs for both. You provision with nomad volume create, you snapshot and clone through nomad volume snapshot, and you get per-volume usage stats and Prometheus metrics out of the box.

How we chose to solve it

A few decisions shaped the whole thing, and they are worth calling out because they are the reason it behaves the way it does.

One interface: CSI, and only CSI. Nomad has more than one way to attach storage now, including host volumes and the newer dynamic host volumes. We deliberately picked CSI as the single path. It is the orchestrator-standard interface, it works all the way back to our Nomad floor of v1.6.3, and it keeps us from coupling to Nomad internals that shift between releases. One interface means one mental model when something goes wrong at 2am.

The plugin is a control-plane shim, never in the data path. This is the part we are most happy with. The driver issues storage operations (provision, attach, mount, unmount) and then gets out of the way. The bytes your workload reads and writes never pass through the plugin process. The data plane lives in the host kernel: the kernel mount, the host’s iscsid, the host’s multipathd, the ZFS layer. Kill the plugin container and the iSCSI sessions, the multipath maps, and the mounts all keep running on the host.

The practical payoff is that you can upgrade, restart, or reschedule the plugin with running workloads untouched. No remounts, no I/O interruption, no forced restart of the jobs consuming the volumes. A rolling image bump is a non-event for anything already running. That is very different from drivers whose data plane lives inside the plugin process, where a restart leaves you with stale handles and forces every consumer to restart with it.

A shared format and mount layer. Because both backends converge on “here is a block device,” we wrote the format, mount, and idempotency logic once. An existing filesystem is never reformatted, a ZFS destroy is leaf-only and never recursive, and expansion is grow-only. Volumes are destroyed only when you explicitly run nomad volume delete. There is no background reconciler that can decide on its own to delete storage you did not ask it to touch.

Gotchas worth knowing before you deploy

None of these are bugs. They are design choices, and knowing them up front saves you a confused afternoon.

• Single-node access only. The driver accepts every single-node CSI access mode and rejects all multi-node modes. ext4 and xfs are not cluster filesystems, so this is intentional. If you need shared read-write across nodes, this is not the tool for that volume.
• QNAP volumes are whole-GiB, exactly. QNAP sizes LUNs in whole binary GiB, so the driver rejects any capacity that is not GiB-aligned rather than silently rounding it. Write capacity_min = "1GiB", not "1000MB". Local ZFS is finer-grained (it rounds up to the zvol block size).
• There is no in-place rollback. CSI has no revert operation, so “restore a snapshot” means provisioning a new volume from that snapshot and repointing your job at it. True in-place zfs rollback or a QNAP revert still exists at the storage layer, but you do that out of band, outside Nomad.
• You cannot delete a volume that still has snapshots. Snapshots depend on their source volume, so nomad volume delete is refused until you delete the snapshots first. Delete children, then the parent.
• The local backend picks a node at create time. A local ZFS volume is pinned to one node’s pool for its whole life. You choose that node when you create it (host = "auto" for the least-provisioned capacity node, or an explicit node name). A consumer that mounts it gets scheduled onto that node automatically.
• The local forwarding transport trusts the network. For --driver=local, controllers forward create and delete operations to the owning node over plain HTTP, authenticated by a shared secret in cleartext. There is no TLS in this release. Run that forwarding port on a trusted, isolated network segment and firewall it to the peer controllers. Do not expose it on a shared network. Anyone who can reach it can issue storage operations against any node.
• A host reboot is different from a plugin restart. The zero-disruption property is about restarting the plugin. A full host reboot tears down the in-kernel iSCSI sessions and mounts, and everything re-stages on boot. That is expected, it is just a different kind of event.

What is out of scope

To keep this honest: the driver manages CSI volumes, not the storage systems underneath them. A couple of things are firmly on your side of the line.

• Setting up the QNAP. Provisioning the appliance, creating storage pools, configuring the iSCSI portal, and creating the credentials the driver uses are all QNAP administration tasks you do before the driver ever runs.
• Creating and managing zpools. For the local backend, ZFS is provided by the host. You create the zpools, keep them healthy, and decide their layout and redundancy. The driver will refuse to start on a node where its allowlisted pool is not imported and online, but it will never create or repair a pool for you. It carves zvols out of pools you already run.

If you already run QNAP and ZFS, none of this is new work. The driver just assumes that foundation exists and builds on top of it.

A short walkthrough

Here is the local (ZFS) backend end to end. The QNAP flow is nearly identical, it just uses the controller and node jobspecs instead of the single monolith job. The repo ships a complete, copy-pasteable examples/ set for both backends.

1. Deploy the plugin on every node.

nomad job run examples/local-monolith.nomad.hcl

For the local backend this runs as a system job so it lands on every node, all under one plugin id. Before you run it, edit the jobspec to point at your image, your zpool name, and your cluster’s specifics.

2. Provision a volume.

The volume spec is small and every parameter has a sensible default:

id           = "local-data"
name         = "local-data"
type         = "csi"
plugin_id    = "local"

capacity_min = "1GiB"
capacity_max = "1GiB"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

# Params below are all optional--defaults come from controller job
parameters {
  pool         = "tank"   # a zpool from your config allowlist
  host         = "auto"   # "auto" = fewest-volumes node, or an explicit node name
  fsType       = "ext4"   # ext4 | xfs
  volblocksize = "16K"    # set at create, immutable after
}

nomad volume create examples/local-volume.hcl

3. Run a workload that mounts it.

nomad job run examples/local-consumer.nomad.hcl

The consumer mounts the volume at /data and stays up, so you can exec in and look around.

4. Verify the mount and check usage.

nomad alloc exec -job local-consumer df -h /data
curl -s localhost:9610/v1/volume-stats/local-data | jq

That stats endpoint reports per-volume usage (bytes, inodes, file and directory counts) keyed by the Nomad volume id you actually manage, and the same data is exposed as Prometheus gauges if you want it on a dashboard.

5. Tear down in reverse.

Stop the consumer first so the volume claim releases, then delete the volume:

nomad job stop -purge local-consumer && nomad volume delete local-data

That is the whole loop: deploy, create, mount, verify, delete.

Why we are sharing it

This is another piece of the same promise we make with everything at HonestHosting: no upsells, no games, just honest and simple hosting that agencies do not have to think about. Reliable persistent storage is part of that foundation, and we would rather run something we understand end to end than glue ourselves to a stack built for a different world. Open-sourcing it is the natural next step, because tools like this get better in the open.

The code, the full examples, and the deeper docs on the control-plane architecture, the metrics, and the security model all live in the repo:

github.com/honest-hosting/nomad-csi-driver

If you run Nomad and you have been quietly wishing the storage story were simpler, give it a look. And if you break it in an interesting way, we would genuinely like to hear about it.

NEXT STEPS

We continue to work hard to bring you the best value for all of your WordPress hosting needs.  If you have any questions, concerns, or would generally like to chat, feel free to contact us anytime.  We want to make sure you get the most out of HonestHosting.io!

• App Login: https://app.honesthosting.io/login
• Step-by-step Documentation: https://docs.honesthosting.io
• Live Chat & Discord Community: https://discord.gg/H5mWqsktj6 
• Personalized Help: https://cal.com/kmott/hh-intro