Share

Alerts and channels

An alert is what happens when a monitor or server changes state: an entry in your bell, a webhook event, an incident where it applies, and a message to every channel that should hear about it. This page explains the channels, what events exist and the controls that keep alerts useful instead of noisy.

How an alert is sent

  1. The engine writes the alert (visible under Alerts on the monitor and in the notification bell of every organisation member).
  2. Outbound webhooks subscribed to the event receive it (always, even during maintenance: they are your own code).
  3. Unless a maintenance window, the monitor's own maintenance flag or one of the rules under Alerts held back by Insights below silences it, a delivery job is queued for the owner's e-mail (if enabled) and for each channel mapped to the monitor; a monitor without mapped channels uses the organisation's default channels.
  4. A worker delivers the message. Failures retry with back-off (1, 5, 15 minutes); every attempt and the provider's answer is kept in the notification log (Admin → Notification log shows all of them, your channel page shows yours).

Channels

Create channels under Alert channels. Each one has a name, a type, its settings (stored encrypted), the events it wants, and the options below. Test sends a sample message straight away.

TypeWhat you enterNotes
E-mailan addressSent with the e-mail templates the administrator can edit (monitor down, recovered, degraded, SSL / domain expiry, server offline/online/alert)
Telegrambot token, chat IDCreate a bot with @BotFather and copy its token. Add the bot to your group or start a chat with it, then find the chat ID: open https://api.telegram.org/bot<token>/getUpdates after sending a message and read chat.id (group IDs are negative). Messages use HTML formatting with an "Open" link
Discordwebhook URLServer settings → Integrations → Webhooks → New webhook, copy the URL. Sent as an embed coloured red / yellow / green
Slackincoming webhook URLCreate a Slack app with Incoming Webhooks, add it to a channel, copy the URL. Sent as blocks with an "Open" button
Microsoft Teamsincoming webhook / workflow URLWorks with the Workflows "post to a channel when a webhook request is received" template (Adaptive Card) and with the legacy Office 365 connector format (tried second)
WebhookURL, optional signing secret, methodA JSON POST (event, title, message, data, sent_at, source) with X-AIM-Event; with a secret, X-AIM-Signature: sha256=<HMAC-SHA256 of the body>. For the richer, retried, per-event webhooks see API → Webhooks
SMSphone number in international formatOnly when the administrator configured an SMS provider (RT Communications, Twilio, Vonage, Textbelt, or any HTTP gateway) and your plan includes SMS. Numbers without + get the installation's country calling code (a leading 0 is dropped: 01711223344 becomes +8801711223344). Bengali text is sent as Unicode. Messages are shortened to 140 characters plus the link
WhatsAppphone numberOnly when the administrator configured a WhatsApp provider (Meta Cloud API, GREEN-API or an HTTP gateway)
Mobile pushprovider + keysPushover: user key and app token (down alerts are sent with high priority). ntfy: topic, optional server URL (default ntfy.sh) and optional access token; title, priority and tags are set. Gotify: server URL and app token
Browser & app pushaudience: me or orgWeb Push to the browsers and installed apps that switched push on under Devices (/app/devices). me (the default) reaches the channel owner's devices, org every active team member's devices. See below

The plan decides how many channels an organisation may have.

Browser & app push (Web Push)

This channel needs no outside service: the platform signs and encrypts each message itself (VAPID, RFC 8291) and hands it to the push service of the browser (Google, Mozilla, Apple, Microsoft).

  1. On each phone or computer, open Devices in the panel and switch push on (the browser asks for permission). An installed app (the PWA, or the Android / iOS app the site owner built) registers the same way. See Mobile app and push.
  2. Create a channel of type Browser & app push, set the audience to me or org, and press Test.

How messages look and behave:

  • the title and the first 400 characters of the message, with a link that opens the monitor or server;
  • events whose name contains down (monitor down, server service down, …) and escalations are sent with high urgency and stay on screen until tapped; one monitor's notifications replace each other instead of piling up;
  • the channel's answer says Sent to N of M devices; with no device registered the delivery fails with No device has push notifications switched on;
  • a browser that unsubscribed (the push service answers 404 or 410) is removed; a device that failed 20 times in a row is switched off and can be tested again from the Devices page.

The server needs PHP with openssl (with openssl_pkey_derive and AES-128-GCM) and curl; otherwise the channel reports what is missing.

Events

Channels receive every event unless you restrict the list. Event names are shown with the short form you can tick:

EventRaised when
monitor.downa monitor is confirmed down
monitor.upit recovers (the message includes the downtime)
monitor.degradedsome locations fail (only sent to channels when the monitor has Alert on degraded)
monitor.ssl_expiry / monitor.domain_expirythe certificate / domain enters its warning window (once a day, severity critical in the last 3 / 7 days)
monitor.dns_changed / monitor.content_changedthe DNS answer / page text differs from the previous check
monitor.escalationan incident was not acknowledged in time (escalation channels only)
server.offline / server.onlinean agent stopped reporting / reports again
server.alert / server.alert_cleareda threshold (CPU, memory, disk, inodes, swap, load, temperature) was crossed / is back to normal
server.service_down / server.service_upa watched service, a Windows IIS or SQL Server service, a Docker container or a disk changed to a bad state (inactive, failed, stopped, exited, dead, unhealthy, or a disk at 95% or more) / back to a good one
server.rebootthe agent's uptime shows a reboot
server.raid / server.smarta degraded RAID array / a SMART health failure
server.lve_faultCloudLinux: one hosting account hit its LVE limits (CPU, entry processes, physical or virtual memory, processes, IO, IOPS) at least CloudLinux faults per account (30 min) times in the last 30 minutes (default 20, 0 = off). Once an hour per account
server.backup_failedJetBackup: a backup run ended failed (critical) or partial (warning), or a backup job that is not disabled reports its last run as failed. Once per run; runs older than 24 hours when the agent first sees them are ignored
server.k8sKubernetes (agent with kubectl): one or more nodes are not Ready (cleared with server.alert_cleared when they recover), a pod is in CrashLoopBackOff, Error, ImagePullBackOff, ErrImagePull, OOMKilled, a container-config / create / run error or InvalidImageName, or its phase is Failed / Unknown, or a pod restarted at least Kubernetes pod restarts (30 min) times in 30 minutes (default 3, 0 = off). Once an hour per pod
monitor.anomalya monitor's response time stayed far above its normal value for its hour of day. Sent to channels only when the monitor has Send response-time anomalies to my alert channels ticked; always in webhooks. See Insights
server.anomalyCPU, memory or load of a server stayed far above normal (alert list and webhooks only)
monitor.flappinga monitor changed status 5 times within 30 minutes; one notice, then its alerts are held back until it is stable for 15 minutes (a stable again message follows)
monitor.correlatedseveral monitors that share an IP, server, host name or dependency went down within 10 minutes; one grouped notice replaces the alerts of the later ones
integration.error / integration.service_downa hosting-panel integration failed / reports a stopped service (bell and webhooks only)

The server events above go to the server's own channels, or the organisation's default channels when it has none; the owner's e-mail is included. Thresholds are set per server on the server form (see Servers).

Flapping and grouped notices follow the down / up filters. The flapping notice and the grouped N monitors … are down notice go to the channels that take monitor.down; the stable again and … are up again messages go to channels that take monitor.up (or monitor.down when the monitor is still down). Ticking only monitor.flapping or monitor.correlated in a channel's event list does not select them; webhooks receive monitor.flapping and monitor.correlated as their own events.

A recovery (monitor.up) is always delivered, even inside a cooldown or quiet hours.

Defaults

Mark channels as Default and every monitor and server without its own channel selection uses them. The owner of a monitor also gets an e-mail (Settings → Notifications → e-mail notifications) unless the plan has e-mail alerts switched off; a default e-mail channel pointing at the same inbox is skipped so you are not told twice.

Escalation

Mark one or more channels as Escalation channels and set Escalate after N minutes in the organisation settings. When a monitor incident is still open and nobody has acknowledged it after N minutes, the escalation channels receive an "ESCALATION: … still down after N min" message (once per incident). Acknowledging the incident stops it.

Cooldown

Cooldown seconds on a channel: after a message is sent, further monitor alerts to the same channel are skipped for that long (logged as skipped: cooldown). Recoveries still go through. Use it on SMS channels to cap the cost of a flapping monitor. The cooldown applies to monitor alerts (down, degraded, expiry, change and anomaly messages); server alerts and the grouped / flapping notices are not held by it.

Quiet hours

Quiet from / to on a channel (24-hour times in UTC, e.g. 23:00 to 07:00; a window that crosses midnight is fine). Inside the window the channel is skipped (logged as skipped: quiet hours), except for recoveries. Quiet hours apply to monitor alerts and to the grouped / flapping notices; server alerts are still sent inside the window. Keep one channel without quiet hours for the things that must wake you up.

Maintenance windows

Maintenance → New window: name, start and end (typed in your own time zone from Settings, stored in UTC), recurrence (once, daily, weekly, monthly), and the monitors and servers it covers (or all monitors). While a window is active:

  • checks keep running and results, incidents and webhooks are still recorded, but no channel is notified (the log says "suppressed by maintenance");
  • status pages show the window as active (and upcoming ones ahead of time) when Show on status page is on; you can limit that to certain pages.

To silence a single monitor, pause it on its page (checks stop until you resume it) or give it a maintenance window that covers only that monitor (checks go on, alerts wait).

Dedupe

Non-transition events are deduplicated: an SSL/domain expiry warning is sent at most once a day per monitor, content/DNS changes at most once an hour, server threshold alerts once until they clear, RAID/SMART once per 12 hours, LVE faults and Kubernetes pod problems once an hour per account / pod, a failed backup once per run, an anomaly at most once an hour per monitor or server metric.

Alerts held back by Insights

Some alerts are recorded but not sent, so one problem does not produce a stream of messages:

  • Dependencies: while an item's parent is down, its down alert (and the later recovery) is held back as suppressed: parent X down; a server's offline, threshold and service-down alerts likewise.
  • Grouping: a monitor that joins an incident group is covered by the grouped notice.
  • Flapping: the alerts of a flapping monitor wait until it is stable.

The alert, the incident and the webhook event are always kept. Insights lists the latest held-back monitor notifications with their reason; a held-back server alert says so in its own text. See Insights and intelligence.

Where to look when a message did not arrive

  1. Notification log (Admin → Notification log, or the channel's page): sent, failed with the provider's answer, or skipped with the reason (cooldown, quiet hours).
  2. The queue (Admin → Queue & workers): a failed delivery is retried; after the last attempt it is listed as dead and can be retried by hand.
  3. The cron job: nothing is delivered when cron/cron.php (or the workers) are not running. Admin → System health shows the last run.
Was this page helpful?

One click helps us improve these docs.

Share this pagePlain links - no trackers, nothing loaded until you click.