For a while last year, my phone got twenty-something push notifications a day: a CPU spike on one VPS, a single probe timeout, disk usage crossing 80%. For the first week I tapped every one. By week three I had perfected the art of dismissing them unread. Which is exactly how the one alert that actually mattered — main site certificate expiring in three days — got swiped away along with the noise.
The lesson wasn't "check your alerts more carefully." It was that an alert system's job is not to detect as much as possible, but to make every alert worth an interruption. Too many alerts is functionally the same as no alerts, except it also eats your attention. These are the rules I wrote down for the future me who will inevitably want to add "just one more" notification.
Rule 1: There are only two kinds of alerts
Every alert is either P1 or P2. No third tier. The middle ground is where alert fatigue breeds — anything "somewhat important" eventually gets ignored, so it might as well not exist.
- P1: act now. Site down, certificate expiring within 72 hours, backups failing twice in a row, disk under 10% free. These push to my phone and are allowed to wake me at 3 AM — because by definition they're things that get worse if I sleep on them.
- P2: good to know. Disk past 70%, sustained high load, a non-critical service restarted. These never push. They go into a single weekly digest email that I'm free to skim or skip entirely.
The litmus test before adding any alert: "when this fires, is there a specific action I'll take?" If the answer is "uh, look at it, I guess," it doesn't qualify as P1. "CPU is high" has no action. "Disk is nearly full, so clean up or resize" does. An alert without an action is just an anxiety generator.
Rule 2: Monitor the monitor first
The sneakiest failure mode is the monitoring system itself dying — you get zero alerts and a false sense of calm. My first probe ran on the same box as the main site. When that box died, the monitoring died with it, and I found out about the outage from a visitor's email.
Now everything watches something else: the probe service monitors all the real services, and a dead-simple external cron job on a cheap box at a different provider pings the probe's heartbeat endpoint. If the probe goes silent for ten minutes, the watchdog yells. Any single link in that chain can fail and someone still complains. The whole thing is embarrassingly plain:
# /etc/cron.d/watchdog — poke the probe every 5 minutes
*/5 * * * * root curl -fsS -m 10 https://probe.example.com/api/heartbeat \
|| curl -fsS -X POST https://bark.example.com/push \
-d "title=probehub is silent&level=critical"Same logic applies to the notification channel. Bark is my primary, but every P1 also goes out over email — when the push service is having a bad day, boring old SMTP usually isn't.
Rule 3: Respect your own sleep
P2 alerts are silenced from 11 PM to 8 AM and rolled into the next morning's digest. I felt guilty about this at first — what if something happens? Then I realized: something happening is the definition of P1, and P1 is never silenced. If I don't trust my own severity levels, the fix is better triage, not less sleep.
The companion rule: flapping alerts must earn their interruption. A probe has to fail three consecutive times to count as down; CPU has to stay over the line for ten minutes to count as high. Transient spikes don't get to page a human. Adding those windows cut my monthly alert volume from over a hundred to single digits, of which one or two a month actually require me to do something.
Rule 4: Every alert answers "and then what?"
The last rule is about the alert text itself. A person woken at 3 AM operates at half their normal IQ, so every P1 must contain three things: what's broken, where, and the first step to take. For example: "backup on s0 failed twice: check journalctl -u backup — usual cause is an expired rclone token." That last clause is a note to the version of me three months from now who has completely forgotten how this thing is wired.
I write every alert as if addressing a brand-new colleague with zero context who just got paged. In three months, that's literally who I'll be.
After the great quieting
Since the cleanup, alerts dropped from twenty a day to a few a week, and a push notification is once again worth reading immediately. The interesting shift is psychological: my first reaction to a ping used to be "ugh, again" — now it's "right, let me deal with that." That's probably the endgame of monitoring: silent almost all the time, and when it does speak up, you know it isn't wasting your breath.