Free weekly email: the week's new posts, videos and one tip from the books, every Sunday. Get it free →
Homelab the Right Way

Home › Posts

Promtail is dead: ship your homelab logs to Loki with Grafana Alloy

Promtail reached end of life on March 2, 2026. Here is the replacement pipeline from Watch Your Homelab, with Loki 3.7, retention that actually deletes, Grafana Alloy reading the journal and Docker logs, and LogQL queries worth saving.

· 4 min read · lokigrafana alloypromtailloggingmonitoringhomelab

Promtail is dead: ship your homelab logs to Loki with Grafana Alloy

Metrics tell you that something changed. Logs tell you why. Watch Your Homelab gives logs their own chapter because the usual homelab approach only works until the day you actually need it.

The trap: SSH, journalctl, and a dead agent

The default homelab log strategy is SSH and journalctl, one machine at a time, after the fact. By the time you look, the container was recreated and its logs went with it, or the journal on a small root disk has already rotated.

The second trap is building a log stack from an old tutorial. Many of them still use Promtail, which reached end of life on March 2, 2026. Grafana's documentation states that it receives no further updates and that future development happens in Grafana Alloy. New setups should start with Alloy.

How the pieces fit

  • Loki stores logs. Unlike a full-text search engine, it indexes only a small set of labels (host, unit, container) and stores the log lines compressed in chunks. That keeps it light enough for a homelab.
  • Alloy is the agent. It reads the systemd journal and Docker's container logs on each host, attaches labels, and pushes the lines to Loki.
  • Grafana queries Loki with LogQL, from the same place that shows your dashboards.

Run Loki with retention switched on

At the time the book was written (October 2026), Loki is at 3.7. A single Loki process with local filesystem storage is the right size for a homelab. This config is based on the project's own local example, with retention turned on:

# /opt/stacks/logs/loki/config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2026-10-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  delete_request_store: filesystem

limits_config:
  retention_period: 30d

analytics:
  reporting_enabled: false
# /opt/stacks/logs/compose.yaml
services:
  loki:
    image: grafana/loki:3.7.8
    restart: unless-stopped
    command: ["-config.file=/etc/loki/config.yaml"]
    ports: ["3100:3100"]
    volumes:
      - ./loki/config.yaml:/etc/loki/config.yaml:ro
      - loki-data:/loki
volumes:
  loki-data:

Retention is the part people get wrong. In Loki, the compactor does the deleting, and Loki's documentation is explicit on three points:

  1. retention_enabled must be true. Otherwise the compactor only compacts.
  2. delete_request_store must be set when retention is on.
  3. Retention needs the 24-hour index period used in the schema above.

retention_period in limits_config then applies to every log stream. The last block turns off Loki's anonymous usage reporting, which the example config leaves on by default.

Install Alloy on each host

Alloy runs as a native service, which makes reading the journal straightforward. These steps follow Grafana's Debian instructions (Alloy 1.20 when the book was written). Run them on the Proxmox host and on each Docker host:

apt install gpg
mkdir -p /etc/apt/keyrings
wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] \
https://apt.grafana.com stable main" > /etc/apt/sources.list.d/grafana.list
apt-get update && apt-get install alloy
usermod -aG adm,systemd-journal alloy

Don't skip the last line. Alloy's documentation warns that without membership in adm and systemd-journal, the journal source starts without an error and collects nothing. That's a silent failure, the worst kind for a logging system.

On Docker hosts, also add the alloy user to the docker group so it can read container logs through the socket. That comes with a caveat: access to the Docker socket is full control of Docker on that host.

The Alloy config

The config lives in /etc/alloy/config.alloy. It's a set of components wired together: sources forward to a writer. This is the book's example for a Docker host named docker1, with Loki on the monitoring guest at 192.168.10.20:

// /etc/alloy/config.alloy on docker1
loki.write "default" {
  endpoint {
    url = "http://192.168.10.20:3100/loki/api/v1/push"
  }
}

// systemd journal, with unit and level as labels
loki.relabel "journal" {
  forward_to = []
  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }
  rule {
    source_labels = ["__journal_priority_keyword"]
    target_label  = "level"
  }
}

loki.source.journal "host" {
  forward_to    = [loki.write.default.receiver]
  relabel_rules = loki.relabel.journal.rules
  labels        = {host = "docker1", job = "journal"}
}

// Docker containers, labelled by container name
discovery.docker "local" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "containers" {
  targets = discovery.docker.local.targets
  rule {
    source_labels = ["__meta_docker_container_name"]
    regex         = "/(.*)"
    target_label  = "container"
  }
}

loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.containers.output
  forward_to = [loki.write.default.receiver]
  labels     = {host = "docker1", job = "docker"}
}

On the Proxmox host, keep the loki.write and journal blocks and drop the Docker ones. The journal there carries pvedaemon, pveproxy, pvestatd, backup jobs and the kernel. Restart with systemctl restart alloy and check it with systemctl status alloy. When started with a listen address, Alloy also serves a debugging UI on port 12345 that shows each component and whether it's healthy.

Labels: few and boring

Loki is fast because its index is small. Every distinct combination of label values creates a new stream, and too many streams make Loki slow and heavy. Keep labels to things with only a handful of values that you filter by: host, job, unit, container, level. Never turn request IDs, user names, IP addresses or full paths into labels. Search for those in the log line instead.

Four LogQL queries worth saving

In Grafana, open Explore, pick the Loki data source, start with a label selector, then filter lines:

{host="pve1", unit="pvedaemon.service"}
{job="docker", container="immich_server"} |= "error"
{job="journal", level=~"err|crit|alert|emerg"}
sum by (host) (count_over_time({level="err"}[1h]))

In order, these show:

  1. The Proxmox API daemon's log.
  2. Only the lines containing "error" from one container.
  3. Every high-priority journal line across the lab.
  4. Errors per host per hour. This turns logs into a metric, and it makes a good panel on an overview dashboard.

Grafana alert rules can run LogQL too. The book's advice is to use that sparingly, for lines that only appear when something is wrong, like a backup job's failure message or repeated authentication failures on a service you expose.

Don't forget the local logs

Loki gives you a searchable copy, but Docker still keeps container logs on the host. With the default json-file logging driver, they grow without limit unless you configure them. Set limits in /etc/docker/daemon.json on each Docker host and restart Docker. Existing containers pick up the change when they're recreated.

{
    "log-driver": "json-file",
    "log-opts": { "max-size": "10m", "max-file": "3" }
}

Cap the journal too: set SystemMaxUse=500M in /etc/systemd/journald.conf, then run systemctl restart systemd-journald. Loki is now the long memory, so local logs only need to cover a few days.

Checklist

  • No Promtail in new setups: Alloy on every host
  • Loki 3.7 with retention_enabled: true, delete_request_store set, a 24h index period and retention_period: 30d
  • The alloy user in adm and systemd-journal (plus docker on Docker hosts)
  • Journal on every host, container logs on every Docker host, systemctl status alloy clean
  • Labels limited to host, job, unit, container and level
  • The four LogQL queries saved, with the errors-per-host query as a dashboard panel
  • Docker log rotation in daemon.json and SystemMaxUse=500M for the journal

More articles