# How alarms work

How an alarm decides between OK, ALARM and INSUFFICIENT_DATA every minute, and when it tells anyone.

An alarm watches one metric of one resource, or one custom series. Its rule is a statistic, a
comparison (`>`, `>=`, `<`, `<=`), a threshold in the metric's unit, a period and a number of
evaluation periods.

## Evaluation

Every alarm is evaluated every minute over its last `evaluationPeriods` whole periods, aligned to
the period and ending two minutes before now, so a minute still being collected is not judged.
Each period's datapoint is the statistic of the points in it; a period without points has none.

| Datapoints in the evaluated periods                   | New state           |
| ----------------------------------------------------- | ------------------- |
| None                                                  | `INSUFFICIENT_DATA` |
| At least one does not cross the threshold             | `OK`                |
| Every period has one, and every one crosses           | `ALARM`             |
| Some periods are empty, and every one present crosses | unchanged           |

A new alarm starts in `INSUFFICIENT_DATA`. A missing datapoint never crosses a threshold, so an
alarm on a series that stops arriving goes to `INSUFFICIENT_DATA`, not to `ALARM`. To hear about
something that stops, publish a zero while there is nothing to count and alarm on `sum < 1`.

The state carries a reason, for example "Threshold crossed: 3 datapoint(s) [91.20, 95.00, 97.40]
were greater than the threshold (80)." or "No datapoints in the last 3 period(s) of 300s.".

## Notifications

An alarm notifies when it enters `ALARM` and when it leaves it: one mail to each of its addresses
(at most ten) and one `POST` to its webhook. Moving between `OK` and `INSUFFICIENT_DATA` notifies
nobody. Leaving `ALARM` for `INSUFFICIENT_DATA`, for example because the machine stopped, notifies
like leaving it for `OK`.

With notifications switched off (`enabled: false`), the alarm is still evaluated and its state kept
current; a change that would have notified says "Notifications are switched off for this alarm."
in its history. For the webhook's
body and what counts as delivered, see [Notification webhooks](notification-webhooks.md).

Every state change, notified or not, is recorded in the alarm's history and sent to the event bus
as `Alarm State Change` from `ews.monitoring`; see [Monitoring alarms and checks](monitoring.md).

## Limits

- The period is a whole number of minutes from 1 minute to one day, and period × evaluation periods
  is at most one day.
- Evaluation periods: 1 to 100. The alarm goes to `ALARM` only when every one of them crosses;
  "M of N" is not supported.
- Alarm names are unique in the organization, up to 128 characters.

## Related

- [Create an alarm](create-an-alarm.md)
- [Alarm on a custom metric](alarm-on-a-custom-metric.md)
- [View an alarm's history](view-alarm-history.md)
