Blockstor

Building

Mon, 01 Jan 0001 00:00:00 +0000

Binaries

Blockstor is a Go module, github.com/cozystack/blockstor. make build regenerates the manifests and deep-copy code, runs go fmt and go vet, and builds the controller, the satellite, the CLI and the migration tool into bin/. The API server builds the usual Go way:

make build
go build -o bin/apiserver ./cmd/apiserver

Binary	Source	What it is
`controller`	`cmd/controller`	controller-runtime manager with the controller-side reconcilers.
`apiserver`	`cmd/apiserver`	the stateless LINSTOR-compatible REST front end.
`satellite`	`cmd/satellite`	the per-node manager that drives DRBD, LUKS and storage.
`blockstor`	`cmd/blockstor`	the operator CLI.
`linstor-migrate`	`cmd/linstor-migrate`	the LINSTOR database converter.

Other generators: make manifests regenerates the CRDs and RBAC from the kubebuilder markers in api/v1alpha1, and make generate regenerates the deep-copy code.

Overview

Mon, 01 Jan 0001 00:00:00 +0000

Blockstor turns the local disks of your Kubernetes nodes into replicated block storage. Each volume is a logical volume (LVM) or a zvol (ZFS) on a node, and DRBD keeps copies of it on other nodes in sync, so a pod can lose its node and come back elsewhere with its data.

Blockstor speaks the LINSTOR REST API. The clients built for LINSTOR, linstor-csi, piraeus-operator, ha-controller and golinstor, work against it unchanged. What is different is how the cluster keeps its state: everything lives in Kubernetes custom resources, and controllers converge the cluster toward it. There is no separate database to back up and no in-memory state to lose when a pod restarts.

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

This page is the shortest path from a Kubernetes cluster with DRBD-capable nodes to a pod writing to a replicated volume. It leaves out the choices. Installation and Provisioning volumes with CSI explain them.

Before you start

The blockstor CLI this page uses is in no release up to v0.1.17. It ships in the release after that, so pick that one or a newer one for <VERSION>.

You need:

Three or more worker nodes with the DRBD 9 kernel module loaded, and the ZFS module if you want ZFS pools. On Talos Linux both come from the siderolabs/drbd and siderolabs/zfs system extensions.
One spare block device on each of those nodes. The example below uses /dev/sdb.
cert-manager in the cluster. It issues the certificates for the API.
kubectl with cluster-admin access, and Go to install the blockstor CLI.

The satellite image carries drbd-utils, lvm2, cryptsetup and zfsutils. Only the kernel modules come from the host.

Installation

Mon, 01 Jan 0001 00:00:00 +0000

Blockstor installs onto an existing Kubernetes cluster. You need the storage nodes prepared, cert-manager in the cluster, and the blockstor manifests from the release you want to run.

Prerequisites

On every storage node that will run the satellite:

The DRBD 9 kernel module, loaded (modprobe drbd).
For ZFS pools, the ZFS kernel module. On Talos Linux both modules come from the siderolabs/drbd and siderolabs/zfs system extensions.
At least one spare block device, or an existing LVM volume group or ZFS pool to back the storage pools.

The satellite image already carries the userspace tools (drbd-utils, lvm2, cryptsetup, zfsutils); only the kernel modules come from the host.

Testing

Mon, 01 Jan 0001 00:00:00 +0000

Each tier answers a different question, and a change is expected to come with tests at the lowest tier that can catch the bug.

Tier	What it covers	Where
L1 unit	Logic, run on every commit	`*_test.go` next to the code
L2 contract, golden	Recorded `golinstor` responses replayed against the server, byte diff	`tests/contract/`
L3 contract, oracle	The same requests to a real LINSTOR and to blockstor, JSON diff	`tests/contract/`
Integration	Controllers and REST against a real Kubernetes API server (envtest), satellites faked	`tests/integration/`
L4 integration on DRBD	`make smoke` on the stand	`tests/smoke*.sh`
L5 e2e	csi-sanity and piraeus-operator e2e on the stand	`tests/e2e/`
L6 operator CLI	The real CLI, through REST and the satellite, down to the DRBD kernel state	`tests/e2e/cli-matrix/`
L7 replay	Operator workflows replayed, and the CLI parity diff against upstream	`tests/operator-harness/`

Unit tests and lint

go test ./...
make lint

Tests that need envtest skip themselves when KUBEBUILDER_ASSETS is not set. make test downloads the envtest binaries and runs the suite with them. make lint must be clean before a push.

Provisioning volumes with CSI

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes provisions blockstor volumes through the LINSTOR CSI driver, linstor-csi. The driver speaks the LINSTOR REST API, and blockstor serves that API, so you point the driver at the blockstor API server instead of a LINSTOR controller.

What to deploy

Piraeus normally ships a whole LINSTOR distribution. With blockstor you take only the client side:

Piraeus component	With blockstor
`linstor-controller`	Skip. The blockstor controller and API server replace it.
`linstor-satellite`	Skip. Blockstor runs its own satellites.
`linstor-csi` (controller and node plugin)	Deploy, pointed at the blockstor API server.
HA controller, `drbd-reactor`	Optional. Point them at the API server the same way as CSI.

Point the driver at blockstor

The API server listens with mutual TLS on port 3371 of the blockstor-apiserver Service in blockstor-system. The installation created a client certificate for in-cluster clients: the Secret blockstor-apiserver-client-tls carries tls.crt, tls.key and ca.crt. A pod in blockstor-system can mount that Secret directly; for a driver in another namespace, see Client certificates in another namespace first. Mount the Secret into both the CSI controller and the CSI node plugin, and set the endpoint and certificate paths:

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Blockstor is built like a Kubernetes operator. The desired state of every node, pool and volume is a custom resource, and controllers keep working until the cluster matches it. Nothing important lives only in a process’s memory, so any blockstor pod can restart without losing state.

Components

flowchart TB
 subgraph clients["Clients"]
 csi["linstor-csi"]
 other["piraeus-operator, ha-controller, golinstor"]
 cli["blockstor CLI"]
 end

 subgraph k8s["Kubernetes API"]
 crds[("blockstor custom resources")]
 end

 subgraph cp["blockstor-system"]
 api["blockstor-apiserver<br/>LINSTOR-compatible REST, mTLS"]
 ctrl["blockstor-controller<br/>placement and allocation"]
 end

 subgraph nodes["Storage nodes"]
 sat1["satellite on node 1"]
 sat2["satellite on node 2"]
 sat3["satellite on node 3"]
 end

 csi -->|REST| api
 other -->|REST| api
 cli -->|kubeconfig| crds
 api <--> crds
 ctrl <--> crds
 crds <--> sat1
 crds <--> sat2
 crds <--> sat3

blockstor-apiserver serves the LINSTOR REST API over mutual TLS. It does not keep state of its own: every request becomes a read or a write of custom resources. It runs as several replicas behind one Service.
blockstor-controller watches the custom resources and fills in what users do not choose: which nodes get replicas, the DRBD port, minor numbers and node IDs, and tiebreakers and quorum settings.
blockstor-satellite runs on every storage node. It creates the logical volumes or zvols, sets up LUKS and DRBD, and reports what it sees back into the custom resources.
The blockstor CLI reads and writes the custom resources directly with your kubeconfig, without going through the REST API.

There is no direct connection between the controller and the satellites. Everything goes through the Kubernetes API.

Repository layout

Mon, 01 Jan 0001 00:00:00 +0000

api/v1alpha1/ CRD types and kubebuilder markers
cmd/ controller, apiserver, satellite, blockstor CLI, linstor-migrate
internal/controller/ controller-side reconcilers
internal/cli/ the blockstor CLI
pkg/ REST, stores, satellite, storage, DRBD, LUKS, placer, dispatcher
config/ generated CRDs, RBAC and kustomize bases
stand/ the Talos and QEMU dev stand and the reference manifests
tests/ contract, integration, e2e, cli-matrix and replay suites
docs/ design notes, parity deltas and audit history
site/ this website

Path	What lives there
`api/v1alpha1/`	The CRD types. `make manifests` turns their markers into `config/crd/bases/`.
`pkg/api/v1/`	The REST wire types, shaped like upstream LINSTOR’s, and the layer-stack resolver.
`pkg/rest/`	The LINSTOR-compatible REST handlers.
`pkg/store/`, `pkg/store/k8s/`	The in-memory and CRD-backed stores behind one `store.Store` interface, and the translation between wire and CRD shapes.
`internal/controller/`	Controller-side reconcilers: resources, definitions, groups, pools, snapshots, nodes, plus tiebreakers, auto-diskful, eviction, rebalancing and migration.
`pkg/satellite/controllers/`	Satellite-side reconcilers for resources, pools, snapshots and physical devices, and the `events2` observer.
`pkg/satellite/`	The DRBD, LUKS and storage layer reconciler and snapshot shipping.
`pkg/storage/`	Storage providers: LVM, LVM-thin, ZFS, ZFS-thin, loopfile and file.
`pkg/drbd/`	`drbdadm` and `drbdsetup` wrappers, the `.res` renderer, the `events2` parser and the options resolver.
`pkg/luks/`	The `cryptsetup` wrapper.
`pkg/placer/`	The autoplacer: capacity-weighted, anti-affinity aware.
`pkg/dispatcher/`	Translates CRDs into the desired state a satellite applies: layers, options, passphrases.
`internal/cli/`	The `blockstor` CLI. Its design is in `docs/cli-design.md`.
`cmd/linstor-migrate/`	The LINSTOR database converter.

Networking and ports

Mon, 01 Jan 0001 00:00:00 +0000

Blockstor’s components do not talk to each other directly. They read and write the custom resources through the Kubernetes API, and the Kubernetes API is the only thing every component needs to reach. The listeners below exist for clients, for kubelet probes, and for the data path between storage nodes.

Listeners

Component	Port	Protocol	Who connects	Reachable from
blockstor-apiserver	3371	HTTPS with mutual TLS	`linstor-csi`, piraeus-operator, `ha-controller`, the `linstor` client	The Services `blockstor-apiserver` and `blockstor-controller` in `blockstor-system`
blockstor-apiserver	3370	Plain HTTP, no authentication	`kubectl port-forward` for debugging	The pod’s loopback address only. Not in any Service.
blockstor-apiserver	8081	HTTP `/healthz`, `/readyz`	kubelet	The pod
blockstor-controller	8081	HTTP `/healthz`, `/readyz`	kubelet	The pod
blockstor-satellite	3372	HTTPS with mutual TLS	Other satellites, to fetch a snapshot stream	The node’s own address
blockstor-satellite	9101	HTTP `/healthz`, `/readyz`	kubelet	The node’s own address
DRBD (kernel)	20000 to 20999	DRBD replication, TCP	The other replicas of the same volume	The node’s own address

The satellite runs with host networking, so its ports are open on the node itself, not on a pod IP. The manifests set hostPort 3372 and 9101 to make this explicit. Both have to be free on every storage node. Port 3372 is outside the DRBD port range and clear of the LINSTOR ports and of port 9100, where node-exporter usually listens on the host.

Internals

Mon, 01 Jan 0001 00:00:00 +0000

This page is about the invariants whose violation would quietly corrupt cluster state. For the user-level picture, see Architecture. For a tour of the code, follow the call graph from cmd/controller/main.go and cmd/satellite/main.go.

Spec and Status

Every blockstor CRD has two halves:

Spec is desired state. Users, REST handlers and the controller’s placement write it. The satellite writes Spec in two places: Resource.Spec.Flags (and ToggleDiskCancel) while it carries out a toggle-disk, and the discovered volume groups and ZFS pools in Node.Spec.Props.
Status is observed state. The satellite writes what it reads from the kernel (drbdsetup events2, drbdadm status, lvs), and the controller’s allocators write values derived from Spec. The REST API writes Status in one place: the resource activate endpoint with ?reallocate-port=true clears Resource.Status.DRBDPort through a Status patch. The allocator only fills an empty Spec.DRBDPort, and the controller copies the spec value back into Status, so the port does not change.

A whole-object Update is unsafe whenever both halves have different writers: a Spec write in flight would overwrite a concurrent Status write, and the other way round. Status goes through the Status subresource. Fields with more than one writer, such as Resource.Status.DRBDPort from the controller next to DiskState and CurrentGi from the satellite, go through Server-Side Apply with separate field managers, blockstor-controller and blockstor-satellite, so each side owns only its own fields.

Resources

Mon, 01 Jan 0001 00:00:00 +0000

Everything blockstor manages is a cluster-scoped custom resource in the blockstor.cozystack.io/v1alpha1 API group. You can create and change them with kubectl or GitOps, with the blockstor CLI, or through the REST API. All three edit the same objects.

Kind	What it describes	Name
`Node`	A storage node and its replication address	the Kubernetes node name
`StoragePool`	A pool of space on one node	`<pool>.<node>`
`PhysicalDevice`	A raw disk the satellite found on a node	set by the satellite
`ResourceGroup`	A placement and replication policy	any
`ResourceDefinition`	One volume set: sizes, layers, policy	any
`Resource`	One replica of a definition on one node	`<definition>.<node>`
`Snapshot`	A snapshot of a definition	`<definition>.<snapshot>`
`ControllerConfig`	Cluster-wide settings	always `default`

The naming rules are enforced by the API server, so a Resource or StoragePool with a name that does not match its spec is refused. Each kind has printer columns, so kubectl get resources or kubectl get storagepools gives a useful table.

Contributing

Mon, 01 Jan 0001 00:00:00 +0000

Licensing: Apache 2.0, clean room

Blockstor is Apache 2.0. Do not copy structure or code from GPL sources, in particular linstor-server and drbd-utils. You may read them to understand behaviour, never to reproduce how they are written.

Hard rules:

No GPL-licensed spec files in the tree. LINSTOR’s rest_v1_openapi.yaml declares GPLv3, and code generated from it is a derivative work.
No code generated from a GPL source by any tool.
No Go dependency under GPL, AGPL, LGPL or SSPL. CI enforces this through license-check.yml, which runs go-licenses check --disallowed_types=forbidden,restricted over ./cmd/.... That is a denylist: it fails on licenses go-licenses classifies as forbidden or restricted, and lets everything else through, so check a new dependency’s license yourself.
Every Go source file carries an SPDX-License-Identifier: Apache-2.0 header.

Apache-2.0 projects that are safe to study and depend on: golinstor, the authoritative description of the REST wire shape; piraeus-operator; linstor-csi; drbd-reactor; and the public DRBD 9 documentation and man pages.

Layer stack

Mon, 01 Jan 0001 00:00:00 +0000

Each volume is built from layers. The bottom layer allocates space on the node, and each layer above adds something on top of the one below. The topmost layer is the device the pod uses.

Layer	What it does
`STORAGE`	Allocates the block device: an LVM or LVM-thin logical volume, a ZFS zvol, or a file. Every stack ends with it.
`LUKS`	Encrypts the device below it with `cryptsetup`.
`DRBD`	Replicates the device to the other nodes over the network.

Supported stacks

Stack	Use it for
`DRBD`, `STORAGE`	Replicated volumes. This is the default.
`DRBD`, `LUKS`, `STORAGE`	Replicated and encrypted at rest. Each replica encrypts its own copy.
`LUKS`, `STORAGE`	Encrypted volume with a single replica and no DRBD.
`STORAGE`	Plain local volume with a single replica, for caches and scratch space.

These four are the only orders blockstor accepts. Other orders, such as LUKS on top of DRBD, are refused when the definition is created.

Using the CLI

Mon, 01 Jan 0001 00:00:00 +0000

blockstor is the command-line tool for blockstor. It reads your kubeconfig and works on the blockstor custom resources directly, so it needs no port-forward and no API certificate: if kubectl can reach the cluster, so can blockstor.

The commands follow the linstor client’s noun-verb grammar, with the same short aliases, so runbooks written for LINSTOR mostly carry over by changing the command name. Tools that speak the LINSTOR REST API, such as linstor-csi, keep working against the blockstor API server. The examples on this page use blockstor.

Day-2 operations

Mon, 01 Jan 0001 00:00:00 +0000

Everything on this page can be done with the blockstor CLI or by editing the custom resources with kubectl. The CLI commands are shown; Resources has the matching YAML.

Reading health

Start with the replicas that are not healthy:

blockstor resource list --faulty

A replica counts as faulty when one of its volumes has a disk state that has not settled, or when one of its DRBD connections is anything other than Connected. A replica the satellite has not reported on yet is not listed, so an empty result means “nothing broken that we can see”, not “everything verified”.

Migrating from LINSTOR

Mon, 01 Jan 0001 00:00:00 +0000

Blockstor can take over a LINSTOR cluster that runs on Kubernetes. The linstor-migrate tool converts LINSTOR’s database into blockstor resources, and blockstor then adopts the volumes and DRBD metadata that are already on disk. No data is copied or resynced, and replica placement does not change.

Do this in a maintenance window, after a full rehearsal on a staging cluster.

What moves and what does not

Converted:

nodes, storage pools, resource groups, resource definitions with their volumes, replicas, and successful snapshots;
controller-wide DRBD options;
DRBD minor numbers, node IDs and the shared secret, unchanged;
DRBD ports, if you capture them from the running nodes (step 3 below).

Not converted, and listed in the tool’s report:

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Each section below is one symptom. Day-2 operations explains the commands used to look at state.

Where a fix needs the DRBD tools, run them inside the satellite pod of the node in question:

POD=$(kubectl -n blockstor-system get pod -l app=blockstor-satellite --field-selector spec.nodeName=<node> -o name)
kubectl -n blockstor-system exec $POD -c satellite -- drbdadm status <definition>

Replicas stay in Connecting

Look at the connection state of the replica, and at the node addresses:

kubectl get resource <definition>.<node> -o jsonpath='{range .status.connections[*]}{.peerNodeName}={.message} {end}'
kubectl get nodes.blockstor.cozystack.io -o wide

A peer stuck in Connecting cannot reach the other replica’s address and port. The two causes are a wrong address and a blocked port.

Compatibility with LINSTOR

Mon, 01 Jan 0001 00:00:00 +0000

Blockstor implements the LINSTOR REST API, so linstor-csi, piraeus-operator, ha-controller, golinstor and the linstor client work against it. This page lists the differences an operator can notice. The full list, kept for the project’s parity tests, is in docs/cli-parity-known-deltas.md.

Not implemented

Feature	What happens
Backups, backup shipping, remotes such as S3	The API answers `501 Not Implemented`. Use snapshots and restore inside the cluster.
Schedules	The API answers `501 Not Implemented`.
`advise`	Not available.
In-place snapshot rollback	Refused on purpose, because it destroys every newer snapshot. Restore the snapshot into a new volume instead.
Per-connection DRBD peer options (`resource-connection drbd-peer-options`)	Not available.
External DRBD metadata (`StorPoolNameDrbdMeta`)	Not supported yet.
Other providers: SPDK, NVMe-oF, OpenFlex, Exos	Not supported.

Behaves differently

Area	LINSTOR	Blockstor
Active network interface	`linstor node interface modify --active` switches the satellite’s connection to another interface.	Has no effect. DRBD uses the address picked in the order described in Networking and ports, not an active flag. A change of address reaches DRBD after the node’s satellite pod restarts.
Default action on lost quorum	`on-no-quorum io-error`	`on-no-quorum suspend-io`: I/O blocks until quorum returns and then resumes, so a replica that was cut off does not get stuck after the network heals.
Shrinking a volume	Allowed for stacks that support it.	Refused. `--force` lowers the size in the definition only; the backing devices and DRBD keep their size.
DRBD ports and minors	Ports from 7000, minors from 1000.	Ports from 20000 to 20999, minors from 20000. Blockstor can run on the same nodes as LINSTOR without collisions. Change the port range per node with the `DrbdOptions/TcpPortRange` property.
Spawning more replicas than there are nodes	Fails and places nothing.	Places as many replicas as it can, reports the shortfall, and adds the rest when more nodes join.
Turning a tiebreaker into a replica with data	Can skip the initial sync.	Runs a full sync, to avoid a DRBD split where both sides refuse to connect.
Removing a replica below quorum	Removes the quorum options from the definition and prints a note.	Sets `quorum off` explicitly, with the same effect on DRBD.
DRBD option values	Validated against the DRBD schema.	Not validated: a wrong value is stored and fails later on the node. Check values before you set them.
Autoplacer weights	`MaxFreeSpace` is 1, the other weights 0.	All four weights default to 1.
`controller version`	Reports a git commit hash.	Through the REST API, release images report the commit they were built from. A development build that was not given one reports a placeholder instead.
`node info`	Shows OS, CPU and memory.	These fields are empty.

Using the `linstor` client

The linstor client works against blockstor’s API server. Port-forward the plain HTTP port and point the client at it. That port has no authentication. It listens on the pod’s loopback address, so a port-forward reaches it and other pods do not; see Networking and ports.

Blockstor

Building

Binaries

Overview

Quickstart

Before you start

Installation

Prerequisites

Testing

Unit tests and lint

Provisioning volumes with CSI

What to deploy

Point the driver at blockstor

Architecture

Components

Repository layout

Networking and ports

Listeners

Internals

Spec and Status

Resources

Contributing

Licensing: Apache 2.0, clean room

Layer stack

Supported stacks

Using the CLI

Day-2 operations

Reading health

Migrating from LINSTOR

What moves and what does not

Troubleshooting

Replicas stay in Connecting

Compatibility with LINSTOR

Not implemented

Behaves differently

Using the linstor client

Using the `linstor` client