How alarms work
How an alarm decides between OK, ALARM and INSUFFICIENT_DATA every minute, and when it tells anyone.
An alarm watches one metric of one resource, or one custom series. Its rule is a statistic, a
comparison (>, >=, <, <=), a threshold in the metric's unit, a period and a number of
evaluation periods.
Evaluation
Every alarm is evaluated every minute over its last evaluationPeriods whole periods, aligned to
the period and ending two minutes before now, so a minute still being collected is not judged.
Each period's datapoint is the statistic of the points in it; a period without points has none.
| Datapoints in the evaluated periods | New state |
|---|---|
| None | INSUFFICIENT_DATA |
| At least one does not cross the threshold | OK |
| Every period has one, and every one crosses | ALARM |
| Some periods are empty, and every one present crosses | unchanged |
A new alarm starts in INSUFFICIENT_DATA. A missing datapoint never crosses a threshold, so an
alarm on a series that stops arriving goes to INSUFFICIENT_DATA, not to ALARM. To hear about
something that stops, publish a zero while there is nothing to count and alarm on sum < 1.
The state carries a reason, for example "Threshold crossed: 3 datapoint(s) [91.20, 95.00, 97.40] were greater than the threshold (80)." or "No datapoints in the last 3 period(s) of 300s.".
Notifications
An alarm notifies when it enters ALARM and when it leaves it: one mail to each of its addresses
(at most ten) and one POST to its webhook. Moving between OK and INSUFFICIENT_DATA notifies
nobody. Leaving ALARM for INSUFFICIENT_DATA, for example because the machine stopped, notifies
like leaving it for OK.
With notifications switched off (enabled: false), the alarm is still evaluated and its state kept
current; a change that would have notified says "Notifications are switched off for this alarm."
in its history. For the webhook's
body and what counts as delivered, see Notification webhooks.
Every state change, notified or not, is recorded in the alarm's history and sent to the event bus
as Alarm State Change from ews.monitoring; see Monitoring alarms and checks.
Limits
- The period is a whole number of minutes from 1 minute to one day, and period × evaluation periods is at most one day.
- Evaluation periods: 1 to 100. The alarm goes to
ALARMonly when every one of them crosses; "M of N" is not supported. - Alarm names are unique in the organization, up to 128 characters.