Your Health Check Is Green and Your Service Is Down

Monitoring has two ways to be wrong, and we treat them as if they were symmetrical. They are not, and the asymmetry should drive how you build checks.

A false alarm costs you an interruption. Someone looks, finds nothing, mutters, goes back to what they were doing. Annoying, self-correcting, and — this is the important part — it tells you the monitoring is alive.

A false all-clear costs you the entire outage. Nobody looks, because nothing asked them to. The first signal is a customer, and by then you have lost both the time and the claim that you knew before they did.

We spend most of our tuning effort on the first failure mode. Almost all of the damage comes from the second.

This isn't unique to uptime monitoring, either. The exact same asymmetry runs through detection: a correlation rule that silently stops matching, a log forwarder that's been dropping events for a week, a webhook token that expired without anyone noticing. A false positive gets someone annoyed. A false negative gets someone breached, and nobody finds out until it's an incident report instead of an alert.

Three ways a check reports green on a broken service

1. The check is narrower than the thing it claims to cover

GET /health returns 200. The endpoint returns a literal {"status":"ok"} from a handler that touches nothing. It confirms that the process is running and the HTTP listener is bound. It says nothing about the database connection pool, the expired credential to the payment provider, or the disk that filled up twenty minutes ago.

This is not laziness; it is drift. The check was written when the service had two dependencies and was honest about both. Four years and eleven dependencies later, the check still passes because it still does exactly what it did in 2022.

The useful question is not "is the check passing" but "what would have to break for this check to fail?" If the answer is "the process would have to be dead", you have a liveness probe wearing a health check's name tag. Both are legitimate, but only one of them is allowed on the dashboard the on-call engineer looks at.

2. The check runs somewhere that cannot fail the way users do

Checks that run inside the cluster share fate with the thing they measure. They use internal DNS, skip the load balancer, bypass the CDN, sit inside the network policy, and often talk to the pod directly rather than through the service. Every one of those is a component that can break for users while leaving the check perfectly happy.

Certificate monitoring makes this concrete. Probe public HTTPS endpoints from inside an environment whose egress terminates and re-signs TLS, and every certificate that comes back is issued by the gateway, not by the site you think you are watching. Build a certificate monitor there, point it at production, and it reports healthy forever — because it is measuring the appliance one hop away rather than the service on the other side of the internet. The check is fine. The vantage point makes it meaningless. (This is the same failure mode, different shape, as checking only the leaf certificate and missing the intermediate — both are a monitor that is technically correct about the wrong thing.)

The rule that falls out: at least one check has to traverse the same path a user does, including the parts you do not own. Internal checks tell you which component broke. Only an external check tells you whether anyone is actually affected.

3. The check stopped running

This is the one that produces the longest outages, and it is structurally invisible. A cron job that no longer fires, a worker that died at 03:00, a pipeline whose credential silently expired — none of these emit anything. And the absence of an alert is indistinguishable from the absence of a problem.

Think about what your dashboard actually renders when a checker is dead. Most of them show the last known state, which was green, with a timestamp nobody reads. The system is not lying to you. You just asked it a question it has no way to answer.

The heartbeat inverts the logic

The fix for the third case is to stop asking "did something report a failure" and start asking "did something report at all".

A dead man's switch is an external service expecting a ping on a schedule. Your checker pings it after every successful run. If the ping stops, the external service alerts. The alert now fires on silence, which means the failure of your monitoring is itself a monitored event.

#!/bin/sh
set -e

run_all_checks            # exits non-zero if anything is wrong

# only reached when the checks ran AND passed
curl -fsS -m 10 --retry 3 "https://hc-ping.example/$CHECK_UUID" > /dev/null

Three things about that snippet are load-bearing.

set -e and the ordering. The ping is last. If the checks fail, or crash, or the box runs out of memory halfway through, the ping never happens, and you get an alert from the outside. Ping first and you have built a system that reports success before knowing whether there is any.

The pinged service must be external. A heartbeat monitored by the same infrastructure that runs the checker shares fate with it. Hosted dead man's switch services exist and are cheap; the point is the independence, not the vendor.

The expected period should be roughly twice the run interval. One missed run on a five-minute job is usually a blip. Two is a pattern. Alert on the pattern, or you will train yourself to ignore the alert — which returns you to a false all-clear by a slower route.

Designing checks that fail loudly

A few habits that follow from all of this.

Make "unknown" a distinct state from "healthy". If your checker cannot reach a target, that is not a pass and it is not necessarily a fail — it is stale data, and it should render differently. Most dashboards have two colours where they need three. A check that has not reported in an hour should look visibly wrong even if its last result was green.

Test the failure path, not just the success path. A check nobody has ever seen fail is a check nobody has verified. Break the dependency deliberately in a staging environment and confirm the alert actually arrives, at the actual destination, in the actual channel. Alert routing rots quietly: people leave, channels get archived, webhooks expire. The routing is part of the system and needs the same treatment as the code.

Write down what each check does not cover. One line next to the definition. "Does not verify database connectivity." It costs nothing to write and it is the sentence that saves you during the post-mortem, because it converts an unknown unknown into a known gap that somebody can choose to close or accept.

Prefer a check that occasionally cries wolf over one that never speaks. Given the asymmetry at the top of this post, a slightly noisy check is a trade you should be willing to make on purpose rather than one you drift into.

The uncomfortable summary

Green means one of two things: the system is healthy, or the system is not being measured. Most monitoring setups cannot distinguish between the two, and the dashboard renders both identically.

The work is not making the checks more sensitive. It is making sure that when the measurement itself fails, something notices — because that is the failure mode that costs you the whole outage, or the whole breach, and it is the only one that gets quieter the worse it gets.

Questions? Find me on u/Arch0ne or Ionut-Robert Sandu.