15 Prometheus alert rules for Linux servers (copy-paste, with thresholds explained)

Every team ends up writing the same fifteen node_exporter alert rules; here they are, so you can skip to the incidents. Each rule below includes the threshold reasoning and the for: duration, because a rule without a tuned for: is a pager-noise generator. Drop them into /etc/prometheus/rules/node.yml, run promtool check rules, reload Prometheus, and Alertmanager does the rest. (Need the stack first? Prometheus install -> node_exporter setup.)

The rules file skeleton

# /etc/prometheus/rules/node.yml
groups:
  - name: node
    rules:
      # rules from every section below go here

The one rule that guards all the others

      - alert: InstanceDown
        expr: up{job="node"} == 0
        for: 2m
        labels: {severity: critical}
        annotations:
          summary: "{{ $labels.instance }} is not being scraped - host, exporter, or network is down"

Without this, a dead host produces fewer alerts, not more - every other rule goes silent exactly when things are worst. for: 2m rides out a single failed scrape.

Disk: trajectory, capacity, inodes, read-only

      - alert: DiskFillPredicted
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[1h], 4*3600) < 0
              and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} < 0.25 * node_filesystem_size_bytes
        for: 15m
        labels: {severity: critical}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} full within 4h at current rate"

      - alert: DiskAlmostFull
        expr: node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes < 0.10
        for: 30m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} under 10% free"

      - alert: InodesExhausted
        expr: node_filesystem_files_free{fstype!~"tmpfs|overlay"} / node_filesystem_files < 0.10
        for: 30m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} under 10% inodes free - deletes fail with disk space still showing"

      - alert: FilesystemReadOnly
        expr: node_filesystem_readonly{fstype!~"tmpfs|overlay"} == 1
        for: 1m
        labels: {severity: critical}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} remounted read-only - usually a dying disk"

DiskFillPredicted is the star: trajectory, not percentage (a runaway log writer kills a disk from 40% in an hour; a stable 85% partition is fine forever). The extra and clause stops it firing on huge, mostly-empty volumes where a linear fit extrapolates nonsense. FilesystemReadOnly gets for: 1m because the kernel only does that remount when something is genuinely wrong underneath - it is never noise.

Memory: pressure and the OOM killer

      - alert: OutOfMemory
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} under 10% memory available"

      - alert: OomKillDetected
        expr: increase(node_vmstat_oom_kill[10m]) > 0
        labels: {severity: critical}
        annotations:
          summary: "{{ $labels.instance }} kernel OOM-killed a process - something died and it wasn't chosen by you"

      - alert: SwapThrashing
        expr: rate(node_vmstat_pswpin[5m]) + rate(node_vmstat_pswpout[5m]) > 1000
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} is swapping heavily - performance is already degraded"

Use MemAvailable, never MemFree - Linux deliberately keeps free memory low and MemFree alerts on healthy page-cache behaviour. OomKillDetected has no for: on purpose: it is a fact about the past, not a state that might pass.

CPU: saturation, steal, load

      - alert: HighCpu
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) > 0.90
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} CPU above 90% for 15m"

      - alert: CpuStealHigh
        expr: avg by (instance) (rate(node_cpu_seconds_total{mode="steal"}[5m])) > 0.10
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} losing {{ $value | humanizePercentage }} to CPU steal - noisy neighbour or oversold host"

      - alert: HighLoad
        expr: node_load15 / count by (instance) (node_cpu_seconds_total{mode="idle"}) > 2
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} load is 2x core count - work is queueing"

All three use for: 15m - short CPU spikes are life, sustained saturation is a problem. Steal deserves its own rule because it looks like generic slowness but has exactly one fix (move the VM), and knowing that during an incident saves an hour of profiling the wrong thing.

The quiet failures: units, clocks, network

      - alert: SystemdUnitFailed
        expr: node_systemd_unit_state{state="failed"} == 1
        for: 5m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} unit {{ $labels.name }} is in failed state"

      - alert: ClockSkew
        expr: abs(node_timex_offset_seconds) > 0.05
        for: 10m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} clock off by {{ $value }}s - TLS, auth tokens, and log ordering all suffer"

      - alert: NetworkInterfaceErrors
        expr: rate(node_network_receive_errs_total[5m]) + rate(node_network_transmit_errs_total[5m]) > 0.01
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.device }} showing interface errors - cable, NIC, or driver"

      - alert: DiskIoSaturated
        expr: rate(node_disk_io_time_seconds_total[5m]) > 0.90
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.device }} at IO saturation - everything on this disk is waiting"

SystemdUnitFailed requires --collector.systemd on node_exporter (covered in the setup guide) and is the cheapest coverage you will ever add: one rule alerts on any service that dies, including the ones you forgot you were running. That is 15 - validate and load:

promtool check rules /etc/prometheus/rules/node.yml && sudo systemctl reload prometheus

What no internal rule can catch

Every rule above shares one blind spot: it runs inside the network, on data the host reports about itself. Four incident classes are invisible from there - the site unreachable from the internet while every internal check is green, the SSL certificate expiring, the domain quietly lapsing, and the death of the Prometheus/Alertmanager stack itself, which by definition cannot page about its own absence. The fix costs nothing: AlertKick’s free external monitors check HTTP, SSL, DNS, and domain expiry from outside with real escalation (Slack, Telegram, SMS, phone, on-call rosters), and a one-line heartbeat from the Prometheus host alerts if the watcher itself stops watching. Internal rules for depth, external checks for truth - run both.