Homelab the Right Way

Home › Posts

A backup job that never runs never fails

Failure alerts can't warn you about a backup job that stopped running. A vzdump hook that pings a dead-man's switch, plus a weekly backup-age report, turns that silence into a phone notification.

· 4 min read · proxmoxbackupsvzdumpmonitoringalerts

A backup job that never runs never fails

Backups fail in two ways. Loudly, when a job runs and reports an error. And quietly, when a job stops running at all: the timer was disabled during an upgrade, the node was off, the cron line was lost in a rebuild, the API key expired.

Loud failures are easy, because every backup tool can email you. Quiet failures need a different shape of alert, and they are the ones that catch people out.

The trap: alerting only on failure

A job that never runs never fails. An off-site rclone sync that stopped in February will not send a single error between then and the day you need it. Your inbox stays clean the whole time, and a clean inbox looks exactly like a healthy backup.

So the alert you need is not "something failed". It is "nothing succeeded recently". That takes three pieces: notifications for the loud failures, a dead-man's switch for the quiet ones, and a report that checks every guest is actually covered.

Step 1: route the loud failures to your phone

Since Proxmox VE 8.1 there is a proper notification system under Datacenter → Notifications. It has two parts.

  • Targets are where notifications go: SMTP (your own mail account), Gotify, a generic webhook, or the local sendmail. A webhook to a self-hosted ntfy server pushes straight to your phone, which matters because email from a homelab tends to land in spam exactly when it counts.
  • Matchers decide which events go to which target. The default matcher sends everything to the mail-to-root target. Add one for backups: match field type equals vzdump, severity error and warning, target: your phone. A second matcher can send info events to email if you like a morning receipt.

One setting is easy to miss: every backup job's Notification mode has to be set to the notification system, or the matchers never see it. On the command line that is:

vzdump --all 1 --exclude 900,901 --mode snapshot --compress zstd \
  --notification-mode notification-system

The older --mailto option still works, but it is deprecated; the notification system replaces it.

Proxmox Backup Server has the same system under Configuration → Notifications for its own verify, prune, GC and sync jobs. Wire the sync job's failures to the same phone target, because the sync job is your off-site copy.

Step 2: a dead-man's switch for the quiet failures

For "nothing succeeded recently", use a check that expects a ping. A dead-man's switch service such as healthchecks.io, or a self-hosted Uptime Kuma push monitor, gives you a URL. Your job pings it on success. If the pings stop, you get a message.

vzdump has hook scripts for exactly this. Add one line to /etc/vzdump.conf:

script: /usr/local/bin/vzdump-hook.sh

Then the hook itself:

#!/bin/bash
# vzdump calls this with the phase as $1
URL=https://hc-ping.com/your-uuid-here
case "$1" in
  job-end)   curl -fsS -m 10 --retry 3 "$URL" >/dev/null ;;
  job-abort) curl -fsS -m 10 --retry 3 "$URL/fail" >/dev/null ;;
esac
exit 0

A clean finish pings the check; an aborted job pings its /fail URL so you hear about it straight away.

Set the check's period to a day and its grace to a few hours. A night the job does not run, for any reason, becomes a phone notification by breakfast.

Do the same for everything else on a timer: the rclone sync (a && curl at the end of the cron line) and the database dump timer.

Step 3: a weekly report of each guest's newest backup

Hooks tell you a job ran. They do not tell you it covered every guest. A job built for a hand-picked list of guests keeps pinging happily while every VM you created after it goes unprotected.

So once a week, a script on the node lists the newest backup of every guest and complains about any older than two days:

#!/bin/bash
# newest backup per guest on a storage; flag any older than MAXAGE days
STORE=pbs01; MAXAGE=2; now=$(date +%s)
API=/nodes/$(hostname)/storage/$STORE/content
pvesh get $API --content backup --output-format json \
 | jq -r '.[] | "\(.vmid) \(.ctime)"' | sort -k1,1n -k2,2nr | sort -u -k1,1n \
 | while read -r vmid ctime; do
     age=$(( (now - ctime) / 86400 ))
     [ "$age" -gt "$MAXAGE" ] && echo "STALE: guest $vmid, last backup ${age}d"
   done
ids="$(qm list | awk 'NR>1{print $1}') $(pct list | awk 'NR>1{print $1}')"
for id in $ids; do
  pvesh get $API --content backup --vmid "$id" --output-format json \
    | jq -e 'length > 0' >/dev/null || echo "NONE: guest $id has no backup"
done

Change STORE to your backup storage's name. Send the output to ntfy or email only when it is not empty, so a quiet week stays quiet.

The second loop is the important one. It catches a guest that has never been backed up at all, which is exactly the gap a hand-picked job list leaves.

Step 4: let Home Assistant watch its own backups

Home Assistant's Backup integration exposes sensors for the backup manager's state and the last successful automatic backup. An automation that fires when the last success is more than two days old closes the loop for the one system in the house that already knows how to notify your phone:

alias: Backup is stale
triggers:
  - trigger: time
    at: "09:00:00"
conditions:
  - condition: template
    value_template: >
      {% set last = states('sensor.backup_last_successful_automatic_backup') %}
      {{ (now() - last | as_datetime) > timedelta(days=2) }}
actions:
  - action: notify.mobile_app_your_phone
    data:
      message: "Home Assistant has not completed a backup in over two days."

Check the sensor's exact entity ID in Developer Tools → States. The Backup integration creates it once automatic backups are configured.

Step 5: watch the disk before it fails the job

Backup storage filling up is a backup failure in waiting, and a common cause of failed jobs. Proxmox shows storage usage on the Datacenter summary; PBS shows datastore usage and has its own notification for it.

A simple rule: when the backup target passes 80 percent, either prune harder or add a disk, and do it that week.

Step 6: break something on purpose

An alert you have never seen fire is in the same category as a backup you have never restored. Test every path once:

  • Disable the backup job for a day and confirm the dead-man's switch fires.
  • Point rclone at a bucket name that does not exist and confirm you hear about it.

If nothing arrives, fix the alert before you trust anything else in this post.

Checklist: monitoring

  • Notification targets set up (phone push plus email), with a matcher for vzdump warnings and errors
  • Every backup job on the notification system, on Proxmox and on PBS
  • A dead-man's switch pinged by the vzdump hook, the rclone sync and the database dump timer
  • A weekly backup-age report that also lists guests with no backup at all
  • A Home Assistant automation that complains when its own backup is stale
  • Backup storage usage watched, with 80 percent as the act-now line
  • Every alert path deliberately triggered once

More articles