One Sunday morning in May, Bark woke me up: the certificate for ziwen.io had zero hours left. Which is a polite way of saying it had already expired. A dozen services flipped from green padlock to red browser warning at the same time. The fun part: the first thing to scream wasn't a certificate monitor — it was the uptime probes. Every HTTPS check failed in the same minute, and my dashboard looked like a row of fireworks.
The root cause was embarrassingly simple. That machine renewed via certbot's HTTP-01, and during an earlier nginx cleanup I had deleted a location block that looked unused. It was, of course, the /.well-known/acme-challenge/ endpoint. Renewal kept failing, and the failure emails went to the server's local mailbox, which no human has ever opened. The most dangerous state for automation is "it used to work."
Three challenges, three ways to die
ACME offers three main ways to prove you own a domain, and each fails differently:
- HTTP-01: serve a token over port 80. Simple and reliable — until someone refactors the web server config. It needs port 80 reachable from the public internet and can't issue wildcards. My incident was exactly this failure mode.
- DNS-01: publish an
_acme-challengeTXT record. The only way to get a wildcard cert, independent of any web server, and works fine behind reverse proxies or CDNs. The cost: your ACME client holds a DNS API token, so scope that token as tightly as your provider allows. - TLS-ALPN-01: prove control during the TLS handshake on 443. No port 80 needed, nice for TLS-only setups, but no wildcards either, and it wants exclusive control of 443 — awkward when real sites already live there.
My rule of thumb: single hostname, one stable entry point, port 80 always exposed — HTTP-01 is fine. The moment wildcards, multiple machines, or a CDN enter the picture, go DNS-01 and don't look back. I still haven't found a case where TLS-ALPN-01 is the only answer.
The current setup: one wildcard, issued centrally
Six VPSes across three cloud providers means six renewal setups if every machine handles its own — six small time bombs. So I centralized: Certimate runs on one box, issues *.ziwen.io via DNS-01, then pushes the cert out to each VPS and the CDN through its built-in deployment pipelines. Issuance and distribution are separate stages with their own logs and failure notifications, instead of one cron job writing to a log nobody reads.
I also added CAA records to narrow which CA can issue for my domains. It doesn't stop everything, but it costs nothing:
ziwen.io. CAA 0 issue "letsencrypt.org"
ziwen.io. CAA 0 iodef "mailto:admin@ziwen.io"Three rules from the post-mortem
The fix itself took five minutes; the lessons took longer. Rule one: renewal failure notifications must travel the same channel as uptime alerts — a push I actually read, not a local mailbox. Rule two: monitor certificate expiry independently, from the outside, with a 14-day warning threshold. The renewal system and its monitor must not be the same thing, or it dies silently. Rule three: any nginx change isn't done until certbot renew --dry-run passes.
Certificates are invisible right up until they take your whole site's credibility down with them. The romance of infrastructure is tuning it until it's boring enough to forget — as long as you're certain something will kick the door down and wake you when it breaks.