How to stop bots and scrapers from eating your bandwidth
A surprising slice of your traffic is robots, and not the good kind. How to spot the scrapers eating your bandwidth, tell them apart from Google, and shut them out at the edge.
Here is a thing that happens to almost every growing site. You open your analytics, see a healthy traffic number, and feel good. Then you open your raw server logs — the ones analytics never shows you, because scrapers do not run JavaScript and therefore never appear in Google Analytics at all — and find that a single IP range has requested your entire product catalogue four times this week, in alphabetical order, at three in the morning.
That is a scraper. It contributed nothing, cost you bandwidth, and if your prices are on those pages, it now knows them.
The instinct at this point is to block everything that is not a human. Do not do that: you would take Googlebot with it, and losing your search traffic is a far more expensive problem than the bandwidth you were trying to save. What you want is a bouncer, not a wall.
Sorting the bots you need from the ones you do not
Bots to keep
- Googlebot, Bingbot — they are why people find you
- Uptime and performance monitors you configured
- Link preview fetchers for Slack, WhatsApp, social platforms
- Payment and webhook callbacks from services you use
Bots to stop
- Price and content scrapers copying your catalogue
- Credential stuffers replaying stolen logins
- Vulnerability scanners probing for known exploits
- Spam bots hammering forms and comment endpoints
There is a third category that has grown a lot recently and does not fit neatly in either column: AI training crawlers. Whether you want those is a business decision rather than a security one, and it is worth making deliberately rather than by accident. Most of them respect robots.txt, and most publish the user-agent strings they use, so a few lines in robots.txt handles the well-behaved ones. The ones that ignore robots.txt are, by definition, in the second column.
Spotting the bad ones
The critical rule is that you identify bots by behaviour, not by what they claim to be. A user-agent string is a text field the client fills in; a scraper that says it is Chrome on macOS costs its author about four seconds of effort.
Behaviour is much harder to fake, because faking it means being slower and more expensive, which defeats the point of scraping.
Inhuman request rates and rhythm
A person does not load 60 product pages in 4 seconds. They also do not load them at exactly 250 millisecond intervals for six hours. Humans are erratic; scripts are metronomes.
Only the HTML, none of the furniture
A browser loading a page also fetches the CSS, the fonts, the images. A scraper wants the HTML and nothing else. A client that requests a thousand pages and zero stylesheets is not a browser.
Claimed identity that does not verify
Googlebot publishes its IP ranges, and a reverse DNS lookup on a real Googlebot IP resolves to googlebot.com and forward-resolves back. Anything claiming to be Googlebot from a consumer VPS provider is lying.
Mechanical URL patterns
Walking your sitemap top to bottom, or iterating product IDs sequentially, or requesting paths that only exist in a vulnerability scanner's wordlist.
Traffic from hosting ranges that never converts
Real customers browse from residential and mobile networks. A sudden flood from a data centre ASN with a zero conversion rate is a strong signal on its own.
Never block on user-agent alone, and be careful blocking whole countries. Both look decisive in a dashboard and both quietly cost you real customers. Country blocks in particular tend to be set during an incident and never reviewed again.
Why the edge is the right place to do this
This is the part that changes the economics. If you block a scraper at your origin, the request has already crossed the internet, opened a TLS connection, woken up your application, and consumed bandwidth on the way in. You paid for all of that, and then returned a 403.
Block it at the edge — out at the PoP nearest the bot — and the request never reaches your origin at all. Your application never wakes up, your origin bandwidth is untouched, and your logs stay readable. During a serious scraping run that is the difference between a nuisance and an outage.
The layers that work
No single mechanism catches everything, and that is fine, because they compose well. In roughly the order they should run:
| Layer | What it catches | False-positive risk |
|---|---|---|
| Network blocklist | Known-bad IPs, already identified across the network | Very low |
| Verified-bot check | Impostors claiming to be Googlebot | Very low |
| Rate limiting | Anything requesting faster than a human could | Low, if the limit is generous |
| Honeypot paths | Crawlers that follow links humans cannot see | Near zero |
| Proof-of-work challenge | Large-scale automation of any kind | Low |
| Country / ASN rules | Blunt, situational | High — use sparingly |
Honeypots are underrated
A honeypot is a URL that no human will ever visit — linked from your page but hidden from view, and disallowed in robots.txt. A real visitor never sees it. A well-behaved crawler reads robots.txt and skips it. Anything that requests it has both ignored robots.txt and followed an invisible link, which is about as close to proof of bad intent as you get on the open web. False positives are close to zero, which makes it one of the few signals safe to act on immediately and automatically.
Proof-of-work is the polite option
A proof-of-work challenge asks the client to solve a small computational puzzle before being served. A real browser does it in a blink and the visitor never notices anything happened. But a scraper making 100,000 requests now has to spend real CPU on each one, and the arithmetic that made the operation worthwhile stops working. It does not block anyone — it just makes industrial-scale scraping cost more than it returns, which is usually enough.
What to do this week
- Read your raw logs, not analytics. Scrapers do not execute JavaScript, so they are invisible in Google Analytics. Sort requests by IP and by user-agent and look at the top twenty of each.
- Put a generous rate limit on everything. Something well above what a human could do — the point is to catch the obvious, not to police your users.
- Tighten it on the endpoints that matter. Login, password reset, search and checkout deserve much stricter limits than your blog.
- Turn on verified-bot checking so impostor Googlebots stop getting a free pass.
- Add one honeypot path and watch what walks into it for a week before you decide what to do about it.
- Decide about AI crawlers on purpose. Whichever way you go, write it into robots.txt rather than leaving it to chance.
Frequently asked questions
How do I stop scrapers from copying my website?
Block them at the CDN edge based on behaviour rather than user-agent. The effective combination is rate limiting, verified-bot checks so impostor crawlers are rejected, honeypot paths that only automated clients will request, and a proof-of-work challenge that makes large-scale scraping too expensive to be worth it. Blocking at the edge means the request never reaches your origin or costs you bandwidth.
How can I tell a bad bot from Googlebot?
Verify it rather than trusting the user-agent string, which anyone can set. Google publishes its crawler IP ranges, and a genuine Googlebot IP resolves via reverse DNS to a googlebot.com hostname that forward-resolves back to the same IP. Anything claiming to be Googlebot from a hosting provider or VPS range is an impostor and can be blocked safely.
Will blocking bots hurt my SEO?
Not if you allowlist verified search crawlers. The risk comes from blunt rules — blocking an entire country or an overly aggressive rate limit that catches Googlebot during a crawl burst. Keep verified search engine bots exempt from rate limits and challenges, and check your Search Console crawl stats after any change to confirm crawling continued normally.
Does robots.txt stop scrapers?
No. robots.txt is a request, not an enforcement mechanism. Well-behaved crawlers such as Googlebot and most AI training crawlers honour it; scrapers and malicious bots ignore it entirely, and some use it as a map of the paths you would rather they did not see. It is worth maintaining for the bots that cooperate, but it is not a security control.
What is a proof-of-work challenge?
A proof-of-work challenge requires the client's browser to solve a small computational puzzle before the page is served. A real browser completes it in a fraction of a second and the visitor sees nothing unusual, but a bot making hundreds of thousands of requests must spend real CPU on every one. It does not identify bots — it makes automation at scale economically pointless.
Why do scrapers not show up in Google Analytics?
Google Analytics relies on JavaScript executing in the browser, and most scrapers request the raw HTML without running any scripts. That means scraping traffic is entirely invisible in analytics while still consuming real bandwidth and origin capacity. To see it, look at raw server or CDN logs, where every request appears regardless of whether JavaScript ran.
You will never get to zero bots, and chasing zero is how people end up blocking their own customers. The realistic goal is that the automated traffic you are paying for is traffic you chose to allow.
See how NordicCDN does this for your site:
Mads has worked in IT — mostly hosting — since he was 16. He took an early stake in a SaaS company and helped grow it through to its acquisition by Visma, has built and run data-center networks, and served as CTO of a Danish data center. He started NordicCDN to make fast, secure infrastructure simple to use.
