Reliability held to a number, with an on-call rota behind it

What this actually solves

There is a stage a product reaches where downtime stops being annoying and becomes expensive. Every hour offline has a number attached — lost orders, contractual penalties, support load that costs more than the outage. At that point, reliability stops being something the development team handles when it comes up, and becomes a discipline of its own with its own budget.

The symptoms are recognisable. Incidents are handled by whoever happens to be awake and happens to have the right access. The same failure occurs three times because the fix was applied in a terminal and never written down. Nobody can say whether last quarter was more or less reliable than the one before it, because nothing was being measured to a defined standard. Every disagreement about whether to ship on a Thursday is settled by whoever is most senior in the room rather than by evidence.

Underneath all of that sits a conversation nobody has had. What level of reliability does this business actually need, and what is it prepared to pay for? Those are not the same question and both have answers. Each additional nine costs disproportionately more than the last — it means redundancy at every layer, and redundancy means paying twice for things that mostly sit idle. A business that genuinely needs that should buy it deliberately. A business that does not should stop feeling guilty about it and spend the money on something else. Making that choice explicit is the first thing this engagement produces.

How the engagement runs

1. Establish the baseline and agree the objectives. What are we measuring — the proportion of requests served successfully, the proportion served within a time the user would accept, or something specific to your product. What is the target, what is the resulting error budget, and where does the current architecture cap it regardless of process. This includes reading your suppliers' own commitments, because your ceiling is their floor. One to two weeks.
2. Write runbooks for the failures you actually get. Not a generic manual — the specific procedures for your top recurring incidents, each with the command to run and the check that confirms it worked. At three in the morning nobody reasons well, so the document is written to be followed rather than understood. This is the deliverable you will get the most obvious value from. One to two weeks.
3. Build the on-call structure. Who is called, in what order, after how long without acknowledgement, and what happens when they cannot be reached. Every alert routed and linked to its runbook, and every alert that does not deserve to wake someone removed from the paging path entirely. One week.
4. Put an incident process in place before you need one. Severity levels defined so everyone reaches for the same word. A communication pattern for customers. A written review after every significant incident that looks at why the system permitted the failure rather than at who typed the command — naming a person ends the investigation one step early and guarantees a repeat. One week.
5. Run the monthly cycle. Reliability reported against the objectives, error budget spend explained, incident actions chased to completion, patching windows kept, capacity reviewed ahead of known peaks, and backups restored into a scratch environment to prove they are real — because a backup nobody has restored is a belief, not a backup. Ongoing, each month.

What moves the price

The hours you need covered. Business hours in one time zone is a straightforward arrangement. Continuous cover is a different commercial proposition entirely, because it needs a rota deep enough that nobody is a single point of failure, and depth costs money whether or not anything happens.

How many systems and dependencies are in scope. One application with a database and a payment provider is a contained surface. A dozen services, several data stores and a set of third-party integrations each with their own outages means more objectives, more runbooks and more incidents that are nobody's fault and still yours to handle.

How much remediation is included. Watching a system, running the process and reporting is one level. Actively removing the causes — the single points of failure, the missing redundancy, the manual steps — is engineering work, and whether it sits inside the monthly fee or is quoted separately is agreed at the start rather than argued about later.

What usually goes wrong

A commitment is made that the underlying stack cannot honour. An uptime promise is given to a customer without adding up what the hosting provider, the database service, the payment gateway and the DNS provider each commit to. Those numbers multiply downward, not upward. When the inevitable supplier outage arrives, the penalty is owed on a promise that was arithmetically impossible on the day it was signed.

The rota is one person deep and called a rota. One engineer receives every alert. It works, so nobody notices the problem, until the week they are on a flight or genuinely ill and the pages arrive somewhere nobody is looking. Depth is the entire point of on-call - without a documented second and third contact and an escalation timeout, what exists is a single point of failure with a phone.

The review names a person and stops there. Human error is the beginning of an investigation, not the end of it. If the finding is that someone ran the wrong command against production, the questions that matter are why that command was available to them, why nothing asked for confirmation, and why nothing detected the damage until a person noticed. Reviews that stop at the name produce apologies. Reviews that continue past it produce changes, and changes are the only thing that stops the same incident recurring.