Alert escalation
Escalation fans an alert out through channels in sequence — notify the team chat first, then the on-call pager 15 minutes later if nothing has been acknowledged. In PathWatch, escalation is built from the per-channel delay on an alert rule’s channel bindings.
Escalation makes sense once you have a real on-call rotation — it’s overkill for a personal homelab. It pairs naturally with the Pro-gated channels (PagerDuty, SMS, Teams), which is where tiered paging starts to matter.
How steps work
Every channel bound to a rule has a delay_minutes value
(default 0):
| Delay | Behaviour |
|---|---|
0 | Notified immediately when the alert fires. |
N > 0 | Becomes an escalation step: notified N minutes after the alert started, only if the alert is still firing and nobody has acknowledged it. |
A rule with bindings at 0, 15, and 30 minutes is a three-step
chain. The delays are all measured from the moment the alert fires
(after confirmation is satisfied), not
from the previous step — so “Slack now, lead after 15, whole team
after 30” is written as delays of 0, 15, and 30.
Escalation steps are evaluated about once a minute, so a step fires within a minute of its delay elapsing. Each step fires at most once per outage.
What stops an escalation
A pending step is skipped — permanently, for the current outage — when either of these happens before its delay elapses:
- The alert recovers. A passing check transitions the alert
back to healthy. Steps that hadn’t fired yet never fire. If the
rule has recovery notices enabled (
on_recovery, the default), the recovery notice goes to every channel on the rule — including escalation steps that never sent an alert. A channel that only ever receives a recovery notice means the outage resolved before its step came due. - Someone acknowledges the alert. Acknowledging (from the Active alerts panel at Alerts, or via the API) freezes the chain: no further steps fire and recurring reminders stop. Acknowledgement is “stop paging me”, not “this is fixed” — the alert stays open until the monitor actually recovers, and the acknowledgement is recorded in alert history with who clicked it.
If the monitor fails again after a recovery, that’s a new alert: the chain starts over from step 0.
Escalation vs recurring reminders
The two are independent and combine well:
| Configured on | Fires | Stops when | |
|---|---|---|---|
| Escalation step | Channel binding (delay_minutes) | Once, after the delay | Acknowledged or recovered before it’s due |
| Recurring reminder | Rule (recurring + recurring_interval_minutes) | Repeatedly, every interval | Acknowledged or recovered |
Recurring reminders re-fire to every channel bound to the rule, regardless of each binding’s delay. A common combination: Slack at 0 minutes, PagerDuty at 15, and a 30-minute recurring reminder so a long outage never goes quiet.
Worked example
Rule: “Production outage”, channels Slack (delay_minutes: 0) and
PagerDuty (delay_minutes: 15), on_recovery: true.
| Time | Outage A (recovers fast) | Outage B (drags on) |
|---|---|---|
| 00:00 | Alert fires → Slack notified | Alert fires → Slack notified |
| 00:09 | Monitor recovers → recovery notice to Slack and PagerDuty; the PagerDuty alert step never fires | Still down |
| 00:15 | — | Still unacknowledged → PagerDuty incident created |
| 00:21 | — | Engineer acknowledges → no further steps or reminders |
| 01:05 | — | Monitor recovers → recovery notice to both channels; PagerDuty incident auto-resolves |
Tips
- Keep chains short. Two or three steps is plenty for most teams; more than five usually means the wrong people are at the top of the rotation.
- Put the noisy channel (PagerDuty, SMS) on the delayed step — the quiet channel at 0 minutes gives whoever is at a keyboard a chance to acknowledge before phones start ringing.
- Size the delay against your confirmation settings: confirmation already adds detection latency before step 0, so a 15-minute step 2 on a monitor with a 5-minute interval and 2 consecutive failures means the pager fires ~25 minutes after the first failed check.
- Use one rule per severity tier, not one per service. A service maps to a monitor or tag; severity maps to “who do I wake up”.