Will this keep working forever?

No, and anyone who says otherwise is selling you a disappointment. Websites are redesigned, class names change, pages move from server-rendered to JavaScript-rendered, and every scraper eventually breaks. The honest commitment is different — the pipeline will tell you when it has broken rather than failing silently, and it will be fixed within an agreed window. That is why ongoing maintenance is part of this service rather than an optional extra.

It depends entirely on what is being collected and from where, and we will not take work where the answer is uncomfortable. Public pages, yes. Content behind a login, no, because reaching it means breaking the terms you agreed to. Personal data — names, emails, phone numbers — falls under GDPR and equivalent laws, and requests to harvest contact details in bulk are declined. Republishing copyrighted content is a separate matter from recording a price. We check robots.txt and tell you what it says.

Could we get the same data from an API instead?

Frequently yes, and it is the first thing we check. A surprising number of sites expose a public or partner API that nobody asked about, and an API is better than scraping in every respect — faster, cheaper, permitted, structured, and it does not break when the design changes. If one exists we will use it and the project gets simpler.

What happens when the site blocks us?

The first defence is not looking like something that should be blocked. Requests are spaced, volumes are kept sane, and the pipeline runs on a schedule rather than in bursts, because hammering a server is both rude and the fastest route to a block. Where a site runs genuine anti-bot systems, we will say so before quoting — that is a materially different and more expensive category of work, and sometimes the right answer is that the data is not obtainable at a sensible price.

How will we know when a price actually changes?

The pipeline stores each run against the previous one and alerts on the difference rather than on every collection. You get a notification when a competitor moves a price, when a listing appears or disappears, or when something you are waiting for comes back into stock — not a daily email you stop reading in a fortnight.

Where does the collected data end up?

Wherever it is most useful to you. Google Sheets for small volumes, Airtable when your team needs to work with the rows, or a Postgres database via Supabase once the row count runs to hundreds of thousands. On top of that a Looker Studio view if you want the trend rather than the table. Storage choice is driven by volume and by who has to read it.

Automation & No-Code

Scraping and monitoring pipelines

Competitor prices, new listings and market data collected on a schedule — with alerts when something changes and when the collector itself breaks.

From: $400–$6,000
Typical time: 1–5 weeks
Usually bought by: Analysts, sellers

What this actually solves

The information you need to make a decision is already published. It is on twenty competitors' product pages, in a directory of newly registered businesses, on a listings site, or in a tender portal. It is visible to anyone. It is just not available in any form you can use, because it is spread across thousands of pages designed for a human reading one at a time.

That gap costs different people different amounts. A retailer whose price drifts above the market loses sales quietly, and finding out means someone opening twenty tabs every evening — which happens diligently for a fortnight and then stops. An agency that sells to newly opened businesses needs to know about them in the first week, not the first quarter. An analyst tracking a market needs the same fields captured the same way month after month, or the comparison is worthless. And some people need exactly one thing: to be told the moment a specific tender is posted or a specific item is back in stock.

What all four are actually buying is not the extraction. It is the schedule and the alert — the guarantee that this check happens every day whether or not anyone remembers, and that a change reaches them without them going to look. Extraction is the easy part. Keeping a collector alive against a web that changes underneath it is the part with the skill in it, and it is the part that determines whether this was worth buying in a year.

What we do, and in what order

Check for an API, and check the legal position. Both before anything is built. A public or partner API removes most of the fragility from the project. The legal review covers what is public versus what sits behind a login, what robots.txt indicates, whether any personal data is involved, and whether what is being collected is a fact or somebody's copyrighted content. One day.
Sample the pages by hand and write the selectors properly. Three or four pages read manually to find where the data actually is and whether the page is rendered on the server or assembled by JavaScript in the browser — with evidence for that conclusion, since it determines the whole approach. Each selector gets a reliability assessment and a fallback, because the ones that work only on the page you happened to open are the ones that fail in week two. One to three days.
Build the collector. Pagination beyond the first page, deliberate spacing between requests, and the cleaning layer — stripping currency symbols and separators, normalising dates, tidying names. Price parsing alone has to survive several formats and one special case: a listing that says price on request must resolve to an empty value, never to zero. Three days to two weeks.
Store it and detect change. Each run written against the previous, with differences flagged and alerts sent on the delta rather than on every collection. Storage in Sheets, Airtable or Postgres according to volume. Two to four days.
Add failure handling and a heartbeat. This is where a monitoring pipeline differs from a one-off scrape. Extraction that returns nothing must raise an alarm rather than write empty rows. Rate limits and temporary errors get retried with backoff. And a heartbeat check catches the scenario where the job did not run at all and therefore produced no errors to notice. One to two days.
Reporting and handover. A dashboard or scheduled report, plus a written note of which selectors are the fragile ones and what breakage will look like when it comes. One to two days.

Tools depending on the target: Apify for sites with mature ready-made scrapers, Browse AI for point-and-click collection, Firecrawl where pages need converting to clean text, and Playwright where the job needs writing properly. Make or n8n runs and supervises the schedule.

What moves the price

How many sources, and how hard each one is. One site is not one twentieth of twenty sites, but it is close enough that the count matters. Difficulty matters more: plain server-rendered HTML is cheap, JavaScript rendering and login walls are considerably more expensive, and a site running real anti-bot infrastructure is the most expensive category there is and sometimes the wrong purchase.

Volume and frequency. A hundred pages a day and a hundred thousand pages a day are different architectures — the second needs queueing, concurrency control, proxy considerations and a storage layer that will not fall over.

Whether keeping it alive is included. It should be. This is the one service in the category where ongoing maintenance is not an upsell but a condition of the thing working at all, because the failure mode is not if but when.

What usually goes wrong

Permanence is implied and then assumed. A proposal says daily automated price monitoring, you read that as settled forever, and two months later a site redesign kills the collector. The work was fine; the expectation was never set. One sentence at the start prevents it entirely — websites change their design, the system will tell you when that happens, and it will be fixed within an agreed time as part of the maintenance arrangement.

The wrong data gets collected because nobody drew the line. Contact details harvested in bulk, or pages pulled from behind a login because the credentials were available. These create liability that outlasts the project and, under GDPR and its equivalents, attaches to you as much as to whoever wrote the script. The boundary is not a formality — it is public versus not public, fact versus copyrighted content, and business data versus personal data.

A broken extractor keeps writing empty rows. This is the specific failure that makes scraping different from other automation. The job runs, the schedule fires, the log says success, and every row is blank because a selector no longer matches. The dashboard shows a competitor price of zero and somebody makes a pricing decision on it. Every field that matters needs a validity check, and an empty result on a page that should have data must be treated as a failure and alerted, not written to the sheet.

See where the site stands before you brief anyone.

If the site is already live, start from what it scores rather than from a guess. The audit reports what it checked, what it found, and — the part most tools leave out — what it could not check, and why.

See the website audit Talk about this work

Questions people ask before they commit.

Will this keep working forever?: No, and anyone who says otherwise is selling you a disappointment. Websites are redesigned, class names change, pages move from server-rendered to JavaScript-rendered, and every scraper eventually breaks. The honest commitment is different — the pipeline will tell you when it has broken rather than failing silently, and it will be fixed within an agreed window. That is why ongoing maintenance is part of this service rather than an optional extra.
Is this legal?: It depends entirely on what is being collected and from where, and we will not take work where the answer is uncomfortable. Public pages, yes. Content behind a login, no, because reaching it means breaking the terms you agreed to. Personal data — names, emails, phone numbers — falls under GDPR and equivalent laws, and requests to harvest contact details in bulk are declined. Republishing copyrighted content is a separate matter from recording a price. We check robots.txt and tell you what it says.
Could we get the same data from an API instead?: Frequently yes, and it is the first thing we check. A surprising number of sites expose a public or partner API that nobody asked about, and an API is better than scraping in every respect — faster, cheaper, permitted, structured, and it does not break when the design changes. If one exists we will use it and the project gets simpler.
What happens when the site blocks us?: The first defence is not looking like something that should be blocked. Requests are spaced, volumes are kept sane, and the pipeline runs on a schedule rather than in bursts, because hammering a server is both rude and the fastest route to a block. Where a site runs genuine anti-bot systems, we will say so before quoting — that is a materially different and more expensive category of work, and sometimes the right answer is that the data is not obtainable at a sensible price.
How will we know when a price actually changes?: The pipeline stores each run against the previous one and alerts on the difference rather than on every collection. You get a notification when a competitor moves a price, when a listing appears or disappears, or when something you are waiting for comes back into stock — not a daily email you stop reading in a fortnight.
Where does the collected data end up?: Wherever it is most useful to you. Google Sheets for small volumes, Airtable when your team needs to work with the rows, or a Postgres database via Supabase once the row count runs to hundreds of thousands. On top of that a Looker Studio view if you want the trend rather than the table. Storage choice is driven by volume and by who has to read it.

Often taken together.

Start with the brief, not the invoice.

Send what you are trying to achieve. You get a written scope, a price and a date — or an honest “this is not the right service for that”.

Talk about Scraping & monitoring pipelines All Automation & No-Code services