Networking and ports

Blockstor’s components do not talk to each other directly. They read and write the custom resources through the Kubernetes API, and the Kubernetes API is the only thing every component needs to reach. The listeners below exist for clients, for kubelet probes, and for the data path between storage nodes.

Listeners

The satellite runs with host networking, so its ports are open on the node itself, not on a pod IP. The manifests set hostPort 3372 and 9101 to make this explicit. Both have to be free on every storage node. Port 3372 is outside the DRBD port range and clear of the LINSTOR ports and of port 9100, where node-exporter usually listens on the host.

The satellite opens no metrics listener. Its code turns off controller-runtime’s metrics server, which would otherwise listen on port 8080 of the node.

The controller has no REST listener. The manifests start it with --enable-rest-api=false, and the REST API is served by the separate blockstor-apiserver Deployment. Both the controller and the API server are started with --metrics-bind-address=0, so they open no metrics port.

Satellites do not dial the controller, and the controller does not dial satellites. Both go through the Kubernetes API.

The API server

Port 3371 requires a client certificate and verifies it against the CA bundle given by --tls-client-ca-file. The manifests create that CA with cert-manager as the blockstor-api-ca Issuer, and issue one client certificate into the Secret blockstor-apiserver-client-tls. The serving certificate covers both Service names, blockstor-apiserver and blockstor-controller, with and without the .blockstor-system.svc.cluster.local suffix. The API server reloads certificates on rotation without a restart.

Port 3370 serves the same API without TLS and without authentication. It exists for debugging, and it listens only on the pod’s loopback address: --rest-bind-address defaults to 127.0.0.1:3370, and the manifests pass that address. Other pods cannot connect to it. kubectl port-forward can, because it connects from inside the pod’s network namespace, so whoever may port-forward to the API server pods gets full read and write access to the API through it. That is the way to reach port 3370 when you need it, for example for the linstor client. See Compatibility with LINSTOR.

An API server started with --rest-bind-address set to an address off loopback, such as the :3370 that the manifests of v0.1.17 and earlier pass, serves the full API without a certificate to every pod that can reach that address. It logs a warning saying so at start. An empty value is refused. If you do bind the port off loopback, fence it with a NetworkPolicy. This one admits only the TLS port and the probe port to the API server pods, so port 3370 is refused from other pods:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: blockstor-apiserver
  namespace: blockstor-system
spec:
  podSelector:
    matchLabels:
      app: blockstor-apiserver
  policyTypes: [Ingress]
  ingress:
    - ports:
        - {protocol: TCP, port: 3371}
        - {protocol: TCP, port: 8081}

A NetworkPolicy only takes effect if your CNI enforces it. kubectl port-forward still works with this policy in place.

DRBD replication

Each volume gets one TCP port, and all of its replicas use that port to connect to each other. The controller allocates the port from a range, per node, and the port appears in the Port column of kubectl get resources and of blockstor resource list.

The range is resolved in this order for every node that hosts a replica:

1. spec.drbdPortRange on the blockstor Node, with min and max.
2. The node property DrbdOptions/TcpPortRange, written as min-max, kept for LINSTOR compatibility.
3. TcpPortAutoRange in spec.extraProps of the ControllerConfig.
4. The built-in default, 20000 to 20999.

Minor numbers for the /dev/drbd<N> devices follow the same pattern with spec.drbdMinorRange and DrbdOptions/MinorNrRange, with a default of 20000 to 65535. These defaults are above what LINSTOR uses, so blockstor can share nodes with a LINSTOR installation.

A firewall between storage nodes must allow the whole port range you use, in both directions. A replica whose peer is blocked stays in Connecting.

Which address replication uses

Each blockstor Node lists its addresses in spec.netInterfaces. blockstor node create <name> <address> writes one interface named default. DRBD peers connect to that address when nothing earlier in the order below applies, so it has to be the node’s own IP as seen by the other nodes. A pod IP does not work, because it is only routable inside the CNI.

Each satellite renders the DRBD configuration of its own node, peer addresses included. Before it renders, it adds the Kubernetes node’s InternalIP to every node’s interface list as an interface named k8s-internal. That interface exists only in the satellite’s memory and never shows up on the Node object. It needs the satellite’s permission to read Kubernetes nodes, which the manifests grant.

The address of a peer is picked in this order:

1. spec.satelliteEndpoint on the blockstor Node, if set.
2. The interface named k8s-internal.
3. The interface named default.
4. The first interface with an address.

This order is what keeps replication working when another tool, such as piraeus-operator, overwrites the first interface with a pod IP.

Snapshot stream between satellites

When a clone or a snapshot restore places a replica on a node that does not hold the snapshot, the receiving satellite fetches the raw snapshot stream from a satellite that does, with an HTTPS GET on port 3372.

The stream runs over mutual TLS. Each satellite presents a certificate, and accepts only a peer whose certificate is signed by the stream CA and issued for the name blockstor-satellite. The manifests create that CA with cert-manager, in blockstor-apiserver-tls.yaml: the CA Certificate and Issuer blockstor-stream-ca, kept apart from blockstor-api-ca so that a satellite’s certificate is not a client certificate for the API, and one Certificate, blockstor-satellite-stream, shared by every satellite and stored in the Secret blockstor-satellite-stream-tls. The DaemonSet mounts that Secret at /etc/blockstor/stream-tls and passes its files with --stream-tls-cert-file, --stream-tls-key-file and --stream-tls-ca-file. The satellite reads the files on every handshake, so it uses a renewed certificate without a restart.

The Secret is a required volume of the DaemonSet, so a satellite pod does not start until cert-manager has issued it. A satellite with only some of the three flags set, or with files that do not load, exits at start. A satellite with none of them set runs with the stream off: it logs snapshot stream disabled: no stream TLS configured, serves nothing on port 3372, and when it needs a snapshot from another node it creates the replica empty, and DRBD fills it with a full sync.

The receiving satellite dials the address of the peer’s interface named default, or the first interface with an address if there is no default. It does not use spec.satelliteEndpoint or the k8s-internal interface, so the default address must be the node’s own IP and reachable on port 3372 from the other storage nodes. A pod IP written there by another tool breaks the stream even when DRBD still connects.

The satellite tries every node that holds the snapshot. A peer that answers that it has no such snapshot is skipped, and if no peer has it, the replica is created empty and DRBD fills it with a full sync. A peer that cannot be reached, or that refuses the TLS handshake, for example because its certificate comes from another CA, is different: if no other peer delivers the stream, the volume is not created, the satellite fails the reconcile with a cross-node fetch error that names the peer’s address, and retries, and the replica does not come up until the port is reachable.

The satellite keeps running if it cannot bind port 3372, for example because another program on the node holds it. It logs snapshot stream server exited with the address, and other nodes can no longer fetch snapshots from it.

What a firewall must allow

Between storage nodes, in both directions:

• The DRBD port range, 20000 to 20999 by default.
• Port 3372 for the snapshot stream.

Inside the cluster:

• From the CSI pods and other API clients to the blockstor-apiserver Service on port 3371.

On each node, for kubelet:

• Port 9101 on the node for the satellite’s probes.
• Port 8081 inside the controller and API server pods for theirs.

Nothing needs to reach port 3370, and with the default bind address nothing outside the pod can.

Ports without authentication

• 3370 on the API server, on the pod’s loopback address. Full read and write access to the API for anyone who can port-forward to an API server pod. Bound off loopback, for any pod that can reach it; see The API server.

The snapshot stream on port 3372 requires a certificate from the stream CA. Port 3371 is the only API entry point meant for clients.