Data & Analytics
One place your questions can actually be asked
BigQuery or Snowflake with raw, staging and marts layers, a written grain for every table, and query cost controlled from the first week.
- From
- $2,500–$25,000
- Typical time
- 1–4 months
- Usually bought by
- Data-mature clients
What this actually solves
The trigger is almost always a question that no single system can answer. Which advertising campaigns brought the customers who spent the most over a year, and how many of them are still buying? The advertising platform knows what it spent and cannot see the second purchase. The shop platform knows the purchases and has no idea where the customer came from. The support system knows who complained. Each one holds a third of the answer and none of them will speak to the others.
The second trigger is time. When producing the monthly reporting consumes several days of skilled work, when a finance close is delayed because three exports have to be reconciled by hand, the cost of the current arrangement is already being paid — it is simply being paid in salary and delay instead of in infrastructure.
What a warehouse gives you is one place where all of it lands, in a shape somebody can query, in a language that has been stable for fifty years and will run on every platform you might move to. What it does not give you, and this matters, is truth by default. A warehouse assembled without a defined structure produces confident, wrong answers faster than a spreadsheet does. The design work is the service; the loading is the easy part.
What we do, and in what order
- Inventory the sources honestly. Every system, what it holds, how much data, how often it changes, and how it can be reached — an API, a native connector, a database, or a file drop. This is also where we find the source that everyone forgot about and that half the questions depend on. Two to four days.
- Choose the platform and model the running cost. Not a preference — a calculation based on your volumes, refresh frequency and who will be querying. You get the expected monthly cost and the three habits that would inflate it, before anything is committed to. One to two days.
- Design the layers and write the grain. Three tiers, always — raw exactly as it arrived and never edited, staging for cleaned and standardised shapes, and marts for the tables the business actually reads. Naming conventions agreed. Every table in the marts layer gets its grain written down before it is built, along with which facts and which dimensions it joins to. About a week.
- Build ingestion. Managed connectors where one exists and is worth its licence, custom extraction where it does not. Incremental loading by default, with a documented watermark field per source, and a separate path designed for history. One to three weeks.
- Build transformations, with tests attached. Staging and marts models under version control, each with assertions — uniqueness on keys, no unexpected nulls, accepted values on categorical fields, row counts within an expected band. A transformation without a test is an assumption waiting to be discovered by a director in a board pack. Two to four weeks.
- Access, cost guards, documentation and monitoring. Who can read what, query cost limits, a written description per table, and alerting so that a failed load emails a named human rather than being noticed three weeks later. About a week.
What moves the price
How many sources, and how badly behaved they are. Two well-documented systems with native connectors is a different project from six, two of which have paginated, rate-limited APIs and one of which is a legacy database nobody has credentials for.
Whether history has to come across. Bringing years of past data is frequently half the total effort — separate code path, separate failure modes, and reconciliation against the old system afterwards to prove nothing was lost or duplicated.
What already exists, and who maintains it after. A half-finished warehouse from a previous attempt often costs more to inherit than to rebuild, and that assessment happens before quoting. Building for an internal team who will own it means more documentation and training; building something that has to look after itself means more monitoring, alerting and defensive design.
What usually goes wrong
Tables are built before anyone defines the grain. This is the expensive one. Without a written statement of what one row represents, joins silently multiply rows, revenue reports come out several times too high, and because the shape of the report looks normal it can survive for months. Recovering trust after a board has acted on a tripled revenue figure costs far more than the day it would have taken to write the grain down.
Nobody watches what the queries cost. Warehouses bill for the bytes a query reads, not the rows it returns, so selecting every column from a wide table is expensive even when the result is small. One badly written scheduled query, running hourly, quietly becomes the largest line on the cloud bill, and it is usually found by the finance team rather than the data team.
The raw layer gets edited. Someone fixes a bad value directly in raw because it is faster than changing a transformation, and the warehouse loses the one property that makes it recoverable — that raw is exactly what the source sent. From then on a full rebuild produces different numbers from the live tables, and nobody can explain why. Raw is append-only; every correction belongs in staging where it is visible and repeatable.
See where the site stands before you brief anyone.
If the site is already live, start from what it scores rather than from a guess. The audit reports what it checked, what it found, and — the part most tools leave out — what it could not check, and why.
Questions people ask before they commit.
- Do we actually need a warehouse, or are we being sold one?
- A fair question and often the answer is no. If your questions can be answered inside one system, or by joining two sources in a reporting tool, that is the cheaper and better route and we will say so. A warehouse earns its cost when the questions genuinely sit between systems — which advertising brought the customers who are still here a year later, what a product line actually costs once returns and fulfilment are included — or when reporting is eating several days a month of somebody's time.
- BigQuery or Snowflake?
- For most small and mid-sized businesses, BigQuery. It bills on usage rather than a standing commitment, so a small dataset costs very little, and the analytics export lands in it natively, which removes an entire integration. Snowflake becomes the better answer at larger volumes, with heavier concurrency, or where an existing team already knows it. Postgres is a legitimate choice for a small business that will never outgrow it, and choosing it is not a failure.
- Is this going to be expensive to run every month?
- It should not be, and the running cost is a design decision rather than a fact of the platform. Warehouses charge for the data a query reads, so a single badly written scheduled query, repeated hourly, is how the bill grows quietly. We write the volume each scheduled query processes next to the query itself, avoid selecting every column from wide tables, and set budget alerts before handover.
- Can we bring our history in as well as new data?
- Yes, and it needs to be quoted as its own piece of work rather than assumed. Loading history is a different job from keeping a warehouse current — it runs through a separate path, it hits API limits that daily loads never touch, and it has to cope with records that were edited after they were created. On many projects it takes as long as everything else combined.
- Why did our revenue figure triple when two tables were joined?
- Because the grain of at least one table was not defined. Grain means what a single row represents — an order, a line within an order, or one product on one day. Join a table whose row is an order to one whose row is an order line and every order value is repeated once per line. It produces a number that looks plausible and is entirely wrong, and it can take weeks to find if nobody wrote the grain down.
- Who looks after it once it is built?
- That is agreed before the build, because it changes the design. Either your team takes it on, in which case documentation, naming conventions and version control matter more and we train accordingly, or the monitoring and alerting are built for someone external to act on. Either way, every table gets a written description of what it is, where it came from and when it updates, because an undocumented warehouse becomes unusable roughly a year after the person who built it moves on.
Often taken together.
Start with the brief, not the invoice.
Send what you are trying to achieve. You get a written scope, a price and a date — or an honest “this is not the right service for that”.
Talk about Data warehouse setup All Data & Analytics services