Proxmox VE 9 high availability: HA rules, fencing and a failover you have actually tested

High availability is the feature most people build a Proxmox VE cluster for: a node dies at 3 a.m. and its important guests are running again on another node before anyone wakes up. Proxmox VE does this with two daemons, a watchdog and a set of rules. This post explains how those pieces fit together, adds guests to HA, writes placement rules (which changed in Proxmox VE 9), walks through a node failure step by step, and covers planned shutdowns and maintenance.

The trap

HA gets switched on for a cluster that is not ready for it: two nodes and no QDevice, or guests whose disks live on local storage with no replica. When a node fails, nothing restarts, or the surviving node fences itself.

The second trap is using HA as a substitute for backups. HA restarts the guest. It does not undo whatever was wrong with it.

The documented requirements are short: at least three cluster nodes (or two plus a QDevice) for reliable quorum, shared storage for the guests (or replicated storage, with a data-loss window back to the last replica), hardware redundancy, and a watchdog.

The HA stack in two daemons

Every node runs:

• pve-ha-lrm, the local resource manager. It starts and stops HA guests on its own node, following the state the CRM requests.
• pve-ha-crm, the cluster resource manager. One node's CRM is the active master at any time. It tells the LRMs what to run, moves guests when something fails and handles fencing.

Both rely on locks in the cluster file system and on a watchdog. While a node is healthy and quorate, its daemons keep resetting the watchdog timer. If the node loses quorum or the HA daemons stop working, the timer runs out and the watchdog resets the node.

Why fencing matters

Before a guest from a failed node can start somewhere else, the cluster must be certain the failed node is not still running it, perhaps cut off from the network but still writing to shared storage. Starting a second copy would corrupt the disk. Making sure the node is really off is called fencing.

Proxmox VE fences with watchdog timers. A node outside the quorate part of the cluster cannot reset its watchdog, so the watchdog resets it. Once the CRM master can take the failed node's lock, it knows the node has fenced itself and can recover its guests.

Without a configured hardware watchdog, Proxmox VE uses the Linux kernel's softdog, which the docs describe as reliable but not independent of the server's hardware. Hardware watchdog modules are blocked by default; to use one, name its module and reboot:

# /etc/default/pve-ha-manager
WATCHDOG_MODULE=iTCO_wdt

One warning from the docs worth repeating: never kill pve-ha-crm, pve-ha-lrm or watchdog-mux by hand. They manage the watchdog, and killing them can cause an immediate reset of the node.

Adding guests to HA

An HA resource is a type and an ID: vm:100, ct:110.

ha-manager add vm:100
ha-manager add ct:110 --state started --max_restart 1 --max_relocate 1
ha-manager config
ha-manager status

In the web interface it is Datacenter → HA → Resources → Add, or More → Manage HA on a guest.

--state is the requested state: started, stopped, disabled or ignored. Once HA manages a guest, ordinary qm start and qm stop simply change its requested state. max_restart is how often HA retries starting it on the same node after a failed start, and max_relocate how often it tries another node; both default to 1.

ha-manager set vm:100 --state stopped     # stop and keep stopped
ha-manager migrate vm:100 pve2            # live migration
ha-manager relocate vm:100 pve2           # stop, move, start
ha-manager remove vm:100                  # stop managing (does not stop it)

Before you add a guest, check it can run on every node HA might pick: disks on shared or replicated storage, bridges with the same names on every node, and no passthrough devices. If a dependency exists on only some nodes, restrict the guest to them with a strict node affinity rule.

HA groups became HA rules in Proxmox VE 9

Earlier releases used HA groups to say which nodes a guest prefers. Proxmox VE 9.0 deprecated them in favour of HA rules, stored in /etc/pve/ha/rules.cfg. Existing groups are migrated to node affinity rules automatically once every node runs Proxmox VE 9; if anything looks wrong afterwards, the upgrade guide points to the active CRM's log, journalctl -eu pve-ha-crm.

Node affinity: which nodes a guest runs on

# Prefer pve1 for the firewall VM; any node is fine if pve1 is down
ha-manager rules add node-affinity fw-on-pve1 --resources vm:100 \
  --nodes pve1
# Make it a hard requirement instead
ha-manager rules set node-affinity fw-on-pve1 --strict 1
# Ordered preference: pve1 first, then pve2 or pve3 equally
ha-manager rules add node-affinity ha-core \
  --resources vm:101,ct:110 --nodes "pve1:2,pve2:1,pve3:1"
# Keep these guests off pve3 (the small node)
ha-manager rules add node-affinity not-pve3 --affinity negative \
  --resources ct:200,vm:300 --nodes pve3

Node affinity rules are not strict by default: if none of the listed nodes is available, the guest may run elsewhere. With --strict 1 it is restricted to the listed nodes and stopped if none of them is up. Priorities are relative; higher numbers win, and nodes without a number get 0. failback, a per-resource property that defaults to on, moves a guest back to its highest-priority node when that node returns.

Resource affinity: guests together or apart

# Two DNS servers should never share a node
ha-manager rules add resource-affinity dns-apart --affinity negative \
  --resources ct:120,ct:121
# An app and its database should stay together
ha-manager rules add resource-affinity app-with-db --affinity positive \
  --resources vm:130,vm:131

Resource affinity rules are the opposite: strict by default. A negative rule cannot list more resources than there are nodes.

Rules are checked for conflicts before they apply. A resource can be in only one node affinity rule, for example, and two resources cannot be kept both together and apart. A conflicting rule is disabled until you resolve it, so run ha-manager rules config after every edit.

What happens when a node dies

1. Node pve2 loses power. Corosync on pve1 and pve3 stops hearing from it and forms a new membership without it. Two of three votes: still quorate.
2. The CRM master marks pve2's HA guests as waiting for fencing (state fence).
3. If pve2 were still running but cut off, it would be in a minority without quorum and unable to reset its watchdog, and the watchdog (60 seconds, per the docs) would reset it. A node with no power is already fenced in practice.
4. Once the CRM can take pve2's lock, the node counts as fenced and its guests move to recovery.
5. The CRM picks a node for each guest from the online, quorate nodes, narrowed by the affinity rules, and chooses the one with the lowest load. The guests start there.
6. When pve2 comes back it rejoins the cluster, and guests with failback move back if a rule prefers it.

With Ceph or shared storage the guests start with every completed write intact. With ZFS replication they start from the last replica. And every guest that ran on the failed node now runs on fewer nodes, which the docs warn can overload small clusters: size each node so the others can carry its HA guests, or mark only what truly matters as HA.

Test it with the power cord

Before you trust HA with the firewall, test it with a disposable VM: add it to HA, then pull the power cord (not a clean shutdown) of the node running it. Watch ha-manager status on another node until the VM shows as started on its new home. A clean shutdown tests something else, the shutdown policy.

Planned shutdowns: pick a shutdown policy

A planned shutdown is not a failure, and HA handles it according to the cluster's shutdown policy, set in Datacenter → Options → HA Settings or in datacenter.cfg:

ha: shutdown_policy=migrate

• conditional (the default): on shutdown, stop the guests and let other nodes take them over; on reboot, freeze them and start them again on the same node.
• migrate: mark the node unavailable and live-migrate its HA guests away first; they return when it is back.
• failover: stop the guests and recover them elsewhere if the node is not back soon.
• freeze: stop and freeze the guests; nothing moves until the node returns.

The docs note that conditional is the default for backward compatibility and that some users find migrate behaves more as expected. One catch with migrate: a guest restricted by a strict node affinity rule to unavailable nodes makes the shutdown hang until you step in.

Maintenance mode for one node

For work on a single node, maintenance mode is the cleanest tool. It migrates the node's HA guests away and brings them back when you disable it:

ha-manager crm-command node-maintenance enable pve2
ha-manager status          # wait until no HA guests remain on pve2
# ... patch, reboot, replace hardware ...
ha-manager crm-command node-maintenance disable pve2

It can run from any node. The LRM shows maintenance as soon as the request is picked up, before every guest has moved, so watch the guest list rather than the status word. Maintenance mode survives a reboot and ends only when you disable it.

For work that could briefly break quorum everywhere, such as re-cabling the corosync network, Proxmox VE 9.2 added a cluster-wide switch: ha-manager crm-command disarm-ha freeze, then ha-manager crm-command arm-ha when you are done. While HA is disarmed it protects nothing, so keep the window short.

Checklist

• Quorum survives one node down (three nodes, or two plus a QDevice).
• Every HA guest's disks are on shared, Ceph or replicated storage.
• Every rule in ha-manager rules config is enabled (no conflicts).
• The shutdown policy is a deliberate choice, not the default by accident.
• A pulled-power failover test of a disposable VM succeeded.
• Each node can carry the HA guests of any one failed node.

Found this useful? Subscribe on YouTube for the daily video, or follow the RSS feed.