Data & Analytics
Pipelines that run without anyone pressing anything
Scheduled, incremental, repeatable data movement between your systems — with a heartbeat and alerts, so a silent failure cannot go unnoticed for weeks.
- From
- $1,500–$20,000
- Typical time
- 3–12 weeks
- Usually bought by
- Data teams
What this actually solves
Right now a person is doing this. Usually a junior analyst, usually on a Monday, exporting from three systems, pasting into a workbook, fixing the date formats that came across wrong, and sending a report out by lunchtime. It takes a few hours a week, which sounds tolerable until you total it across a year and notice you have bought several working weeks of skilled attention to do copy and paste.
The hours are not the real cost, though. The real cost is that manual work fails invisibly. A filter left applied from last week, a paste that lands one row out, a currency column that silently arrives as text — none of these announce themselves, and the report still looks like a report. Decisions get made on it. By the time anyone notices, the wrong number has been in circulation for a month and is in a board pack.
What this service buys is a system that does the work on a schedule, produces the same result every time it runs, and tells you when it cannot. That last part is the one people underestimate. A pipeline that works ninety-five percent of the time and says nothing about the other five percent is more dangerous than the analyst, because everybody has stopped checking.
What we do, and in what order
- Work out how each source can actually be reached. An API, a managed connector, direct database access or a file drop — and for each, the awkward specifics: rate limits, page sizes, authentication that expires, and whether records can be modified after creation. That last question changes the whole design. Two to five days.
- Design before writing code. Which field acts as the watermark for incremental loading and why, how re-running a window avoids duplication, what order the loads run in — often orders, then customers, then whatever depends on both — and how a backfill will run on a separate path. You see this design and can object to it before any code exists. Three to five days.
- Build extraction and transformation. Types coerced properly, dates normalised to one time zone, currencies made consistent, names standardised, duplicates resolved on a defined key. Under version control, because this is now software and will need changing. One to four weeks.
- Build the scheduling, the failure handling and the heartbeat. Every run writes when it started, when it finished and how many rows it moved. Failures alert. A volume check fires when today's row count falls far below the recent norm, which is what catches the source that returned an empty but technically successful response. Three to five days.
- Run the backfill. Separately, resumably, and reconciled against the source afterwards so you have counts rather than reassurance. Two days to two weeks, depending on history.
- Write the runbook and hand it over. What each job does, what each alert means, what to check first, and who to call. Then a supervised period where it runs unattended and we watch it. Two to three days.
What moves the price
How many sources, and how unpleasant their interfaces are. Some APIs are well documented, paginate sensibly and return clean data. Others rate-limit aggressively, change without notice, or require reconstructing state across several endpoints. The difference between the best and worst source of the same size is several times the effort.
Volume, frequency, and whether history is included. A daily job on a modest dataset is a different build from an hourly job at scale, where incremental logic and cost control stop being optional. Historical backfill is quoted separately because it genuinely is a separate piece of engineering.
Whether you are buying monitoring as well as movement. A pipeline that a technical person on your team will watch is cheaper than one that has to detect its own failures, escalate appropriately and be recoverable by someone who did not build it. That difference is usually the largest single factor in the range.
What usually goes wrong
Silent failure. The pipeline stops, and nothing says so. Reports keep generating on whatever data was last loaded, so they look completely normal — same layout, same shape, plausible figures — and the business runs on three-week-old numbers until someone coincidentally checks against the source. This single failure mode destroys more client relationships than any other in this work, and the prevention is trivially cheap on day one and impossible to retrofit trust after.
No protection against re-running. The job is triggered twice, or a retry fires after a partial success, and every row lands again. Revenue doubles for that day. Because the duplication is confined to one window it survives casual inspection, and it is found weeks later when someone asks why a Tuesday was extraordinary.
Records that change after they arrive. Orders get edited, refunded, cancelled or have line items added, but a pipeline that only ever fetches new rows never sees any of it. The warehouse keeps the original version forever and slowly diverges from the source system. Handling amendments — refetching a recent window, or keying on a modified timestamp rather than a created one — has to be designed in, because by the time the divergence is noticed nobody can tell which records were affected.
See where the site stands before you brief anyone.
If the site is already live, start from what it scores rather than from a guess. The audit reports what it checked, what it found, and — the part most tools leave out — what it could not check, and why.
Questions people ask before they commit.
- What happens when a pipeline breaks at three in the morning?
- Something breaks eventually — an API changes, a credential expires, a source goes down — so the design assumes it. Every run writes a heartbeat recording when it ran and how many rows it moved, failures raise an alert to a named channel or inbox, and a runbook says what to check and in what order. We also draw a deliberate line between what should wake a person and what can wait until morning, because an alerting system that cries wolf gets muted within a fortnight and then it is worse than nothing.
- What if the pipeline runs twice by accident?
- It should make no difference, and making that true is a design decision taken at the start rather than a fix applied later. A pipeline that only appends will duplicate everything on a rerun, which is the single most common defect in home-made data plumbing. We build so that re-running the same window produces the same result — either by replacing that window's data or by keying on a unique identifier from the source.
- Do we need a paid connector service or should this be custom code?
- Depends on which sources and how many. Where a managed connector exists for a mainstream system, it is usually cheaper than the hours it replaces and it keeps working when the source changes its API. Custom code earns its place for internal systems, unusual transformations, or where per-row connector pricing does not suit your volumes. Most real projects are a mix, and we will show you the comparison rather than defaulting to whichever we prefer building.
- Can you load our historical data too?
- Yes, as a separate exercise. Backfill runs through its own path because it behaves differently from a daily load — it hits rate limits the daily job never sees, it needs to be resumable when it fails partway, and it has to handle records that were amended after they were first created. Treating it as part of the routine pipeline is how a backfill ends up running for a week and duplicating half the data.
- What if the source system changes its fields?
- It will, usually without notice. New fields appearing are harmless if the load is written to tolerate them. Fields disappearing or changing type are the dangerous case, and the protection is a schema check on each run that alerts rather than silently dropping data. That check is cheap to build on day one and effectively impossible to add retrospectively once wrong data is already downstream.
- Is it not simpler to keep having someone do this manually?
- It is simpler until it is not. Manual work costs the same hours every week forever, breaks when that person is on leave, and its errors are invisible — nobody knows a column was pasted one row out until a decision has already been made on it. Automation costs once and then reports its own health. The strongest argument is not the hours saved, it is that failures become visible instead of silent.
Often taken together.
Start with the brief, not the invoice.
Send what you are trying to achieve. You get a written scope, a price and a date — or an honest “this is not the right service for that”.
Talk about ETL / data pipelines All Data & Analytics services