15 Prometheus alert rules for Linux servers (copy-paste, with thresholds explained)

Every team writes the same fifteen node_exporter alert rules. Here they are. Each rule includes the threshold reasoning and the for: duration, because a rule without a tuned for: is a pager-noise generator. Drop them into /etc/prometheus/rules/node.yml, run promtool check rules, reload Prometheus, and Alertmanager does the rest. (Need the stack first? Prometheus install -> node_exporter setup.)

The rules file skeleton

# /etc/prometheus/rules/node.yml
groups:
  - name: node
    rules:
      # rules from every section below go here

The one rule that guards all the others

      - alert: InstanceDown
        expr: up{job="node"} == 0
        for: 2m
        labels: {severity: critical}
        annotations:
          summary: "{{ $labels.instance }} is not being scraped - host, exporter, or network is down"

Without this, a dead host produces fewer alerts, not more. Every other rule goes silent exactly when things are worst. for: 2m rides out a single failed scrape.

Disk: trajectory, capacity, inodes, read-only

      - alert: DiskFillPredicted
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[1h], 4*3600) < 0
              and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} < 0.25 * node_filesystem_size_bytes
        for: 15m
        labels: {severity: critical}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} full within 4h at current rate"

      - alert: DiskAlmostFull
        expr: node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes < 0.10
        for: 30m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} under 10% free"

      - alert: InodesExhausted
        expr: node_filesystem_files_free{fstype!~"tmpfs|overlay"} / node_filesystem_files < 0.10
        for: 30m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} under 10% inodes free - deletes fail with disk space still showing"

      - alert: FilesystemReadOnly
        expr: node_filesystem_readonly{fstype!~"tmpfs|overlay"} == 1
        for: 1m
        labels: {severity: critical}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} remounted read-only - usually a dying disk"

DiskFillPredicted is the star: trajectory, not percentage. A runaway log writer kills a disk from 40% in an hour; a stable 85% partition is fine forever. The extra and clause stops it firing on huge, mostly-empty volumes where a linear fit extrapolates nonsense. FilesystemReadOnly gets for: 1m because the kernel only does that remount when something is genuinely wrong underneath. It is never noise.

Memory: pressure and the OOM killer

      - alert: OutOfMemory
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} under 10% memory available"

      - alert: OomKillDetected
        expr: increase(node_vmstat_oom_kill[10m]) > 0
        labels: {severity: critical}
        annotations:
          summary: "{{ $labels.instance }} kernel OOM-killed a process - something died and it wasn't chosen by you"

      - alert: SwapThrashing
        expr: rate(node_vmstat_pswpin[5m]) + rate(node_vmstat_pswpout[5m]) > 1000
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} is swapping heavily - performance is already degraded"

Use MemAvailable, not MemFree. Linux keeps free memory low to use it for page cache. Alerting on MemFree fires on healthy systems. OomKillDetected has no for:. It records a past event, not a current state.

CPU: saturation, steal, load

      - alert: HighCpu
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) > 0.90
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} CPU above 90% for 15m"

      - alert: CpuStealHigh
        expr: avg by (instance) (rate(node_cpu_seconds_total{mode="steal"}[5m])) > 0.10
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} losing {{ $value | humanizePercentage }} to CPU steal - noisy neighbour or oversold host"

      - alert: HighLoad
        expr: node_load15 / count by (instance) (node_cpu_seconds_total{mode="idle"}) > 2
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} load is 2x core count - work is queueing"

All three use for: 15m. Short CPU spikes are normal. Sustained saturation is a problem. Steal deserves its own rule because it looks like generic slowness but has exactly one fix: move the VM. Knowing that during an incident saves an hour of profiling the wrong thing.

The quiet failures: units, clocks, network

      - alert: SystemdUnitFailed
        expr: node_systemd_unit_state{state="failed"} == 1
        for: 5m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} unit {{ $labels.name }} is in failed state"

      - alert: ClockSkew
        expr: abs(node_timex_offset_seconds) > 0.05
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} clock off by {{ $value }}s - TLS, auth tokens, and log ordering all suffer"

      - alert: NetworkInterfaceErrors
        expr: rate(node_network_receive_errs_total[5m]) + rate(node_network_transmit_errs_total[5m]) > 0.01
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.device }} showing interface errors - cable, NIC, or driver"

      - alert: DiskIoSaturated
        expr: rate(node_disk_io_time_seconds_total[5m]) > 0.90
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.device }} at IO saturation - everything on this disk is waiting"

SystemdUnitFailed requires --collector.systemd on node_exporter (covered in the setup guide) and is the cheapest coverage you will ever add: one rule alerts on any service that dies, including the ones you forgot you were running.

promtool check rules /etc/prometheus/rules/node.yml && sudo systemctl reload prometheus

What no internal rule can catch

Every rule above shares one blind spot: it runs inside the network, on data the host reports about itself. Four incident classes are invisible from there - the site unreachable from the internet while every internal check is green, the SSL certificate expiring, the domain quietly lapsing, and the death of the Prometheus/Alertmanager stack itself, which by definition cannot page about its own absence. The fix costs nothing: AlertKick’s free external monitors check HTTP, SSL, DNS, and domain expiry from outside with real escalation (Slack, Telegram, SMS, phone, on-call rosters), and a one-line heartbeat from the Prometheus host alerts if the watcher itself stops watching. Run the internal rules for depth and external checks for the view from outside.