Alertmanager routing tree examples: matchers, nesting, grouping, and continue

Most Alertmanager problems are not delivery problems. The webhook works, Slack is connected, the alert is firing in Prometheus - and it still went to the wrong channel, or arrived forty times, or never arrived at all. That is nearly always the routing tree.

This post is the deep version of the route block. If you need to get Alertmanager installed and talking to Prometheus first, start with the Alertmanager setup guide and come back here when the basic config is running.

How matching actually works

The route block is a tree. The top-level route is the root, and it must have a receiver - that is the fallback that catches anything the children do not.

Alertmanager walks the tree depth-first. At each level it checks child routes in order and takes the first one whose matchers all pass. If that child has children of its own, it repeats the process one level down. If no child matches, the alert is delivered by the current node’s receiver.

Two rules cause most surprises:

1. First match wins among siblings. Order matters. A broad matcher placed above a narrow one will swallow everything below it.
2. A matched child ends the search at that level unless it sets continue: true.

That second one is the escape hatch for “I want this alert to go to two places”, covered further down.

Matchers: the four operators

matchers is the current syntax. The older match (exact) and match_re (regex) fields are deprecated and should be migrated - one list now handles both:

matchers:
  - severity = "critical"        # equals
  - environment != "dev"         # not equals
  - service =~ "api|web|worker"  # regex matches (fully anchored)
  - team !~ "test-.*"            # regex does not match

Three things worth knowing:

• Regexes are fully anchored. service =~ "api" matches the label value api and nothing else. If you want substring behaviour, write service =~ ".*api.*".
• All matchers in a list must pass. It is AND, never OR. To express OR, use a regex or two sibling routes.
• A missing label is an empty string. matchers: [team = ""] is a legitimate way to catch alerts that forgot to set team, which is a genuinely useful safety net.

Example 1: route by severity

The most common starting shape. Criticals page, warnings go to chat, info is recorded but never announced.

route:
  receiver: team-slack           # the catch-all, always required
  group_by: [alertname, cluster]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity = "critical"]
      receiver: oncall-pager
      group_wait: 10s            # criticals wait less before the first page
      repeat_interval: 1h        # and nag harder while unresolved

    - matchers: [severity = "warning"]
      receiver: team-slack

    - matchers: [severity = "info"]
      receiver: "null"           # deliberately swallowed

The "null" receiver is a real receiver with no configs in it:

receivers:
  - name: "null"

Being explicit about what you are discarding is better than leaving it to the catch-all, because six months later the config still says out loud that info alerts are intentionally silent.

Example 2: route by team, then by severity

Nesting is where the tree earns its name. Route to the owning team first, then decide urgency inside that team’s subtree - so each team controls its own escalation without touching anyone else’s.

route:
  receiver: fallback-slack
  group_by: [alertname, cluster, service]
  routes:
    - matchers: [team = "platform"]
      receiver: platform-slack       # platform's default
      routes:
        - matchers: [severity = "critical"]
          receiver: platform-pager

    - matchers: [team = "data"]
      receiver: data-slack
      routes:
        - matchers: [severity = "critical"]
          receiver: data-pager

Note what the child routes inherit. group_by, the timers, and anything else not overridden come from the parent, so you only write what differs. A platform alert with severity: warning matches the team = "platform" route, finds no matching child, and is delivered by platform-slack.

The fallback-slack receiver at the root is not decoration. Any alert with a team label you did not anticipate lands there, which is how you find out a new service is emitting alerts nobody owns.

Example 3: silence an environment without silencing the rules

Staging alerts firing into the production channel is the fastest way to train a team to ignore the channel. Route on environment above everything else:

route:
  receiver: prod-slack
  group_by: [alertname, cluster]
  routes:
    - matchers: [environment =~ "dev|staging"]
      receiver: nonprod-slack
      repeat_interval: 24h       # visible, but not nagging
      routes:
        - matchers: [severity = "critical"]
          receiver: nonprod-slack   # still not a page, on purpose

Put this route first. Because first match wins, an environment = staging alert with severity = critical hits the non-prod subtree and never reaches the production paging routes below it.

Example 4: continue, for when one alert needs two destinations

By default a matched route ends the search. continue: true tells Alertmanager to keep evaluating the remaining siblings after this one matches, so a single alert can reach several receivers.

The usual case is an audit or ticketing sink that should see everything without interfering with normal routing:

route:
  receiver: team-slack
  routes:
    - matchers: [severity =~ "critical|warning"]
      receiver: incident-webhook
      continue: true             # <- keep going, do not stop here

    - matchers: [severity = "critical"]
      receiver: oncall-pager

    - matchers: [severity = "warning"]
      receiver: team-slack

Without continue: true, the first route would capture every critical and warning, and the pager route below it would be dead code that never fires. This is a genuinely common way to break a routing tree, and it fails silently - nothing errors, the pages just stop.

Grouping: the three timers, in plain terms

group_by decides what counts as “the same problem”. The three timers decide when you hear about it.

group_by: [alertname, cluster]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h

• group_wait (default 30s) - a new group has appeared. Wait this long before the first notification, so the other forty servers with the same problem land in the same message. Too short and you get one ping per host; too long and you delay the page.
• group_interval (default 5m) - a group you have already notified about has gained new alerts. Wait this long before sending an update. This is the anti-chatter timer.
• repeat_interval (default 4h) - nothing changed, the group is still firing. Re-send this often so an unresolved problem does not quietly fall out of memory.

Two special values of group_by are worth knowing:

group_by: ['...']   # do not group at all - one notification per alert
group_by: []        # group everything into a single group

['...'] (a literal three-dot string) is the escape hatch when you have a downstream system that wants one event per alert, such as a ticketing webhook. Do not use it for a human channel - that is how storms happen.

The most common grouping mistake is putting a high-cardinality label in group_by. Adding instance means a 40-server disk problem produces 40 groups and therefore 40 notifications, which is exactly the storm grouping exists to prevent. Group by what identifies the problem (alertname, cluster, service), not what identifies the machine.

Test the tree before it wakes anyone

amtool will tell you exactly where a set of labels ends up, and sends nothing while doing it:

# Print the whole tree as Alertmanager understands it
amtool config routes show

# Where would this specific alert land?
amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  severity=critical team=platform environment=production

# Catch typos and bad matchers before a reload
amtool check-config /etc/alertmanager/alertmanager.yml

Run routes test against the four or five label combinations you actually care about and put them in a shell script. It takes ten minutes, it is the only way to verify continue behaves as you think, and it turns “I believe criticals page the on-call” into something you have checked.

The failure modes worth knowing

• Broad route above narrow route. The narrow one becomes unreachable. routes show makes this obvious.
• Missing continue: true on a catch-all sink. Everything downstream silently stops.
• instance in group_by. Grouping stops working, storms return.
• No root receiver that a human reads. Unowned alerts vanish into a channel nobody has open.
• Regex assumed to be a substring match. service =~ "api" does not match api-gateway.
• Editing the config without reloading. amtool check-config validates the file, but Alertmanager still needs systemctl reload alertmanager or a POST /-/reload.

Where this stops being worth it

A well-built routing tree is genuinely good engineering, and the config above will serve a small team for years. What it will not do is get someone out of bed. Alertmanager delivers a payload to a receiver and considers the job done - it has no concept of an on-call rotation, no acknowledgement, and no escalation to a second person when the first does not respond. That is the gap every team eventually hits, and it is why a webhook receiver named oncall-pager usually points at something else.

AlertKick fills that side: on-call rotations with overrides, escalation chains that keep climbing until somebody acknowledges, and notification channels ordered least to most disruptive. Point an Alertmanager webhook receiver at it and keep the routing tree you have already tuned, or skip the stack entirely and let the agent do the detection too.

Either way: build the tree, then test it with amtool before you trust it.