Monitoring competitor sites looks simple from the outside: fetch the page, compare the text, alert on change. The first prototype takes an afternoon. The problems start in week two.
Below are the problems that actually show up when this goes to production. All of them are solvable and none of them are free.
Pages are built in the browser, not on the server
Most modern campaign pages render on the client. Fetch the raw HTML and you usually get an empty shell and a JavaScript bundle. The bonus rate or the discount percentage is not in that shell, it arrives later with an API response.
The price of that is running a browser. Opening a real browser for every scan costs several times the memory and wall time of a plain HTTP request. In a system that wants to scan a hundred pages hourly, that difference lands directly on the infrastructure bill.
The practical approach is hybrid: try the cheap request first, fall back to a browser when the content is missing. Learning and storing which domain needs which path brings the cost down noticeably over time.
Bot defences are unstable, not an error case
Commercial sites throttle automated traffic. You will meet rate limits, browser challenges, geographic restrictions and session walls. The important part is that none of this is stable. A scan that worked yesterday can come back empty today.
So the monitoring system has to treat failure as a normal state. You need retries, backoff, retention of the last successful capture and a health view that surfaces a falling scan success rate.
The worst outcome is not a crash. The worst outcome is silently collecting empty content and manufacturing the conclusion that a campaign was pulled. A system that cannot tell empty from genuinely removed becomes the source of its own false alarms.
The DOM is noisy, and separating real change is the actual work
A campaign page comes back slightly different on every load. Session identifiers, timestamps, countdown timers, randomly ordered recommendations and ad identifiers all move. Compare raw text and you will report a change on nearly every scan.
What works is comparing extracted fields rather than raw text. Pull offer type, rate, cap and duration out of the page, then diff on those fields. A refreshed hero image stops producing a signal, and a rate change starts producing one.
We covered the product-side version of this distinction separately: Why isn't 'the page changed' enough?
The same URL does not serve everyone the same page
Competitor sites run A/B tests constantly and vary the shop window by geography and device. Fetch the same URL from two different servers and you can see two different campaigns. Compare those without knowing it and you will report a move that never happened.
The fix is to record the capture conditions: which region, which device profile, which language setting. If the conditions are not stored, two detections cannot be compared.
The extraction layer has to survive language and currency
Campaign copy is localised. The same offer is phrased one way in one market and completely differently in another. Number formats move too: thousands separators, decimal marks and currency symbols all vary by market.
An extraction layer built purely on pattern matching works in the first market and quietly produces wrong values in the second. A wrong value is more dangerous than an empty one, because the system still looks healthy.
So attach a confidence score to every extracted field and route low-confidence detections into a separate queue. Reviewing that queue catches the errors and improves the extraction rules over time.
Keeping evidence costs more than you expect
If you want a time-stamped capture behind every detection, storage grows fast. A hundred competitors, several scans a day and full-page captures build a serious archive within months.
Two decisions help here. First, archive only the scans where a change was detected rather than every scan. Second, store the capture alongside the extracted fields, so searching the archive does not mean opening images one by one.
There is also a retention decision. You do not need to keep everything forever. For most teams the last twelve months at full resolution and extracted fields only before that is enough. Skip this decision early and storage cost grows quietly.
Frequency is a product decision, not an engineering one
Scanning every page hourly is technically possible and usually unnecessary. Match frequency to how the page behaves: campaign windows move often, corporate pages rarely. Deriving frequency from observed change history is both cheaper and more accurate than a fixed interval.
For the product-side setup of the same process: How to track competitor campaigns?
This is a data product, not a script
Taken together, these points say one thing: competitor monitoring is a data product with maintenance, not a script. Source pages change without notice, extraction rules go stale, defences harden. Left without an owner, a monitoring system starts producing wrong data quietly within months.
So build two things from day one: a view of scan success rate, and a simple check on the distribution of extracted fields. The second one tells you a page structure has changed the moment the distribution jumps.
With Adversee
Adversee ships all of these layers already working: hybrid fetching, failure tolerance, field-level comparison and a time-stamped archive. You use the output instead of operating your own browser fleet.
Start tracking competitor moves automatically.