Why your 2-node Proxmox cluster freezes

Two Proxmox nodes feel like redundancy. One box dies, the other carries on. That is the picture most people have when they join a second machine to the cluster.

Out of the box, it is the opposite: two machines that can each stop the other from working. This post covers why that happens, the small fix that solves it, and the two related traps that make a cluster look haunted.

How quorum works, in one paragraph

A Proxmox cluster uses Corosync to agree on who is alive. Each node gets one vote, and the cluster is quorate when more than half the votes are present. Without quorum, the cluster filesystem at /etc/pve goes read-only. You can't start guests, change configs or create backups, and nothing in the UI saves.

The trap: 1 vote out of 2 is not a majority

Now do the arithmetic for two nodes. Lose either one, and the survivor holds 1 vote out of 2. That is exactly half, and half is not more than half. So the survivor freezes.

That is the part that catches people. The node that is still up, still powered and still on the network is the one that stops working, because it has no way to know whether the other node is dead or just unreachable. Refusing to write is the safe choice for the cluster, and it is a very annoying choice for you.

You can see the vote count at any time:

pvecm status

The fix: a third vote that isn't a third server

A QDevice is a tiny external voter. It can run on a Raspberry Pi, a NAS container or any always-on Linux box that isn't one of the cluster nodes. It gives the cluster a third vote without a third hypervisor.

With three votes, losing one node leaves 2 out of 3. That is more than half, so the survivor stays quorate and keeps working.

The setup is three installs and one command:

# On the external box (Debian/Ubuntu/Raspberry Pi OS):
apt install corosync-qnetd

# On EVERY cluster node:
apt install corosync-qdevice

# On ONE cluster node (root SSH to the QDevice box must work):
pvecm qdevice setup 192.168.10.5

# Check: you should see 3 total votes and "Qdevice" listed
pvecm status

Replace 192.168.10.5 with the address of your own external box.

Where the QDevice must not live

Don't run the QDevice as a VM or container on one of the cluster nodes. It is tempting, because the hardware is already there. But if that node goes down, it takes its own vote and the QDevice's vote with it, and you are back to 1 vote out of 3 on the survivor. Separate hardware is the whole point.

The related trap: a node on Wi-Fi that keeps flapping

Even with enough votes, a cluster can look broken if one member keeps dropping in and out. The usual cause is Wi-Fi.

Corosync is extremely sensitive to latency and jitter. It wants a stable, low-latency link: single-digit milliseconds, consistently. Wi-Fi delivers great average speeds and terrible consistency. A microwave, a neighbor's channel change or a roaming decision adds 50 to 300 ms of jitter for a few seconds. Corosync sees missed tokens, decides the node is gone, and the cluster reshuffles. A few seconds later it is back. Then it is gone again.

The symptoms look random:

• The node shows a red X, then turns green again
• /etc/pve is briefly read-only
• Backup jobs fail with "cluster not ready"
• The logs fill with lines like [TOTEM] A processor failed, forming new configuration

Check for it:

journalctl -u corosync --since "24 hours ago" | \
  grep -Ei "failed|new membership|retransmit"
corosync-cfgtool -s        # link status per node
ping -c 100 -i 0.2 <other-node> | tail -2   # look at mdev/max, not avg

Look at the mdev and max numbers from the ping, not the average. A good average with a big max is exactly the jitter Corosync hates.

The fix isn't a Corosync tuning trick. It's a cable. If the node physically can't be wired, use a powerline or MoCA adapter (still better than Wi-Fi), or take it out of the cluster and run it standalone. A standalone node you manage separately is far less trouble than a cluster member that drops out twice an hour.

If your nodes have two network ports, give Corosync a second link (link1) on a separate network or a direct cable between the nodes. Corosync fails over between links automatically.

When you're stuck without quorum right now

If one node of two is dead and you need the survivor working immediately, there is an emergency lever:

pvecm expected 1

This tells the survivor to accept being alone. It is an emergency lever, not a fix. Only use it when you are sure the other node is truly off, because two nodes that each think they are the whole cluster can write conflicting configs. Once the crisis is over, the real fix is still the QDevice.

Scripts: wait for quorum instead of failing halfway

If you script changes on a cluster with a flaky member, check quorum before each write and retry, instead of letting the script die partway through a batch:

until pvecm status | grep -q "Quorate:.*Yes"; do
  echo "waiting for quorum..."; sleep 10
done
pct destroy 123 --purge

The loop waits until pvecm status reports the cluster as quorate, then runs the write. Wrap each destructive or config-changing step this way and a brief membership blip costs you a pause, not a half-finished job.

Checklist

• Every cluster node is wired, ideally on the same switch
• Two-node clusters have a QDevice on separate hardware, never on a cluster node
• pvecm status shows the expected vote count (3 for two nodes plus a QDevice)
• Corosync logs show no membership changes over the past day
• Nodes with two ports give Corosync a second link (link1)
• Scripts check for quorum before each write
• You know the pvecm expected 1 lever, and when not to pull it

Found this useful? Subscribe on YouTube for the daily video, or follow the RSS feed.