Knowing your system is broken before your customers tell you

What this actually solves

There is a specific, avoidable humiliation at the centre of this service — a customer telling you your system is down. Somebody emails, or posts publicly, and only then does anyone look. By the time the cause is understood, the outage has been running for hours and the story has already been told by the person who found it.

Underneath that is a harder problem. Most failures are not clean. The site is up but slow. Orders are being taken but confirmation emails are queued and not sending. One payment method fails while the rest work. A background job died on Tuesday and nobody noticed until the monthly report was empty. None of these trip a simple up-or-down check, and all of them cost money quietly for as long as they last.

Then there is the argument nobody can win without data. Somebody tells you the system is slow. Is it? Everywhere or on one page? For everyone or for the users furthest from the server? Since when? Without measurement, the reply is a guess, and the fix is a guess, and the confirmation that it worked is also a guess. Observability is what turns that whole conversation into numbers — metrics that count, logs that record what happened, and traces that show where a single request spent its time.

How the work runs

1. Define what counts as broken, in your terms. Not CPU percentages — the things a customer would notice. Checkout failing. Search returning nothing. The queue growing faster than it drains. Emails not going out. This list decides everything downstream, because it is what the alerts will eventually be about. One to three days.
2. Instrument the machine and the service. Request rate, error rate and latency for the application. Underneath it the host signals that cause outages and almost never get watched — free disk space, memory and swap pressure, and database connection pool saturation. Log growth deserves special attention, because logs quietly filling a disk until the database can no longer write is one of the ways an application dies with no error message at all. Three to five days.
3. Centralise logs, in a structured form, with a retention decision. Logs off the machines and into one searchable place, so the answer to what happened at 09.14 does not require connecting to four servers. Structured output rather than free text so it can be filtered rather than grepped by eye. Retention is set deliberately, because volume is what this costs. Three to five days.
4. Add tracing where requests cross boundaries. If a request touches several services or an external API, a trace shows which segment consumed the time. This is optional and we say so — on a single application with a database it adds cost without answering a question you had. Three to seven days, when it applies.
5. Alert on symptoms, route them, and attach a runbook to every one. Thresholds on the user-visible failures rather than on causes, sensible durations so a brief spike does not page anyone, an escalation path, and a link from each alert to the procedure for handling it. A synthetic check runs from outside your network so a DNS or certificate failure is caught from the customer's point of view, and certificate expiry is watched explicitly rather than remembered. Three to five days.
6. Tune against real traffic before calling it finished. Two weeks of live operation, after which every alert that fired is judged — was it worth waking someone for? The ones that were not are retuned or removed. Two weeks, running alongside.

What moves the price

How many services and hosts there are. One application on two servers is a contained setup. A dozen services, several databases and a queue system means more instrumentation, more dashboards and considerably more thought about what deserves an alert.

Hosted platform or self-hosted stack. Hosted tooling is faster to stand up and bills monthly. A self-hosted stack has no licence cost and becomes another production system you own, including its upgrades and its own failure modes.

Log volume and retention. This is the line that grows without anyone deciding to grow it. A chatty application retaining everything for a year is a fundamentally more expensive proposition than a disciplined one retaining a fortnight, and the choice is worth making consciously.

What usually goes wrong

Alerts fire on causes instead of symptoms. High CPU pages someone at three in the morning while every customer is being served perfectly well, because a batch job is running. Meanwhile checkout has been failing for one card type since midnight and nothing has fired at all. Alerting on the symptom the customer feels, with the causes available on a dashboard for whoever investigates, inverts that completely.

Everything becomes an alert and so nothing is. It is easy to enable a large default ruleset and feel thorough. By the third night of noise the notifications are muted, and the one alert that mattered arrives into a channel nobody reads. Fewer alerts is a feature that has to be actively defended, because adding one always feels responsible and removing one always feels risky.

The system is only watched from inside itself. An agent on the server reports that the server is healthy — and it is, from where it is standing. The expired TLS certificate, the DNS record that was changed, the firewall rule blocking an entire region are all invisible from inside the machine and completely obvious from outside it. The second failure mode here is worse and more ironic — a monitoring stack installed on the same host it monitors, with its own log and metric storage filling the same disk, so that when the disk fills the monitoring dies first and the alert never leaves the building.