If your hosting bill climbed with no launch or campaign behind it, automated visitors are a likely cause. Crawlers and scrapers request pages from your site all day, and on a host that bills by request or by compute, every request that reaches your application can cost money.
Search engines, SEO tools, AI crawlers and scrapers all do this, and you can keep the ones that bring searchers while limiting the rest. Google's crawl budget guidance is aimed mainly at sites with at least 1 million unique pages or 10,000 rapidly changing pages, so a smaller site does not need to treat every crawl as a ranking emergency.
The first step costs nothing. Open your host's or CDN's request log for the last seven days and list the five biggest automated visitors.
What that list shows decides the rest of the order: keep the search engines, cache public pages, close URL combinations that have no search value, and rate-limit or block only what you have confirmed is costing you.
Key Takeaways
Seven days of host or CDN logs, grouped by verified crawler, user agent, IP or ASN, path, query string and cache status, show which automated visitors create work for your application.
Verify them with reverse DNS and published IP ranges, then protect them while you limit everything else. Blocking Googlebot is not a cost fix if Google traffic matters to you.
Compliant crawlers follow it, it is not a firewall, and Googlebot does not process crawl-delay. Rate limits and firewall rules stop requests before they reach your application.
Caching public pages, tightening filter and pagination URLs and adding edge rules can cut more cost than a move, because a new host does not remove crawler traffic.
Why do bots raise a hosting bill in the first place?
Every request that reaches your application can cost you something. The server builds the page, asks the database for listings, runs a function and sends the result back, and a host that bills by request, compute or bandwidth counts each one. A person visiting is one request. A crawler working through your listings can make thousands in an afternoon, and none of them will ever call or buy.
A directory makes this worse than a brochure site, because every combination of filter, sort and page number can become its own URL. Google's guidance on filter and sort URLs says crawlers can spend resources on filtered combinations that do not deserve indexing. Each of those requests can trigger a fresh response from your application unless something in front of it answers instead.
Automated traffic is also a normal part of the web. Cloudflare's 2025 Radar review reported that Googlebot generated 4.5% of HTML requests, with AI bots collectively slightly lower. Those are Cloudflare-customer measurements, not a count of the whole web, but they show why a busy site sees crawlers at all hours. The goal is fewer expensive requests, not zero bots.
Caching changes the cost, though not always completely. Serving a stored copy cuts the work your origin does, but billing depends on the platform's pricing model, so a cached request can still appear on a bill. If a line item surprises you, ask the host which requests it counts and at which layer, and ask for the answer in writing.
What should you look for in your request log first?
Open the request analytics in your host, CDN or firewall and set the range to the last seven days. If you can see the previous normal period too, put it beside the spike. You are looking for who asked, what they asked for, and whether the request reached your application or was answered from cache.
Group the log by these fields
The name a visitor gives itself, which anyone can fake.
Whether your CDN or host has confirmed the identity.
The address, or the network behind it.
Which pages, and which filter, sort or page-number combinations.
Whether the request succeeded, and whether it was a cache hit or a miss.
What each request cost to serve.
Sort by request count and write down the top five automated identities, their totals, the paths they asked for and how many reached the origin. Check anything claiming to be Googlebot or Bingbot separately, because a user agent string is not proof of identity. Google explains how to verify Googlebot with reverse and forward DNS lookups, and our post on fake Googlebot traffic shows what spoofed requests tend to look like.
On one of the sites we run, a directory of about 138,000 listings drew about 4.7 million billed requests in under two weeks, one day near 2.6 million, with nothing launched. Most of it came from SEO-tool and scraper bots, in waves. We blocked them in robots.txt, kept every search engine and added a rate limit, and the host's support pointed out that robots.txt is a request not every bot honors.
Which crawlers should you keep, and which can you limit?
Four groups show up in a large site's logs, and each calls for a different decision. Search engines bring searchers, so you keep them. SEO tools, AI crawlers and scrapers each need a look at identity and cost before you limit anything.
| Group | Examples | What the owner says about robots.txt | First control |
|---|---|---|---|
| Search engines | Googlebot, Bingbot | Both follow robots.txt; Googlebot does not process crawl-delay, Bingbot does | Verify identity, keep, fix wasteful URLs |
| SEO tools | AhrefsBot, SemrushBot, MJ12bot, DotBot, DataForSEO | Ahrefs says its bots obey robots.txt and crawl-delay; Semrush documents its robots.txt handling | Confirm identity, then robots.txt plus a rate limit |
| AI crawlers | GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot | OpenAI documents separate crawlers for search, training and user-requested fetching; Common Crawl says CCBot obeys robots.txt and rate-limits | Decide by purpose and cost |
| Scrapers | Any user agent, often spoofed | No commitment to follow it | Rate limits and firewall rules by behavior, IP or ASN |
Each of these companies explains its own crawler: Ahrefs, Semrush, Common Crawl and OpenAI. For Moz, Majestic and DataForSEO, read each crawler's own current policy page before you count on it to honor robots.txt or a crawl rate; Majestic's FAQ is one place to start.
AI crawlers are not one kind of visitor. Because OpenAI documents separate crawlers for search, training and user-requested fetching, blocking one name does not necessarily block the others. An AI user agent is not automatically an attack, and a search crawler is not automatically worth its cost, so decide by purpose and by the requests each one sends you. Our post on Google's AI training control covers the robots.txt side of that choice.
Scrapers can claim any user agent, so a name in the log proves nothing on its own. Judge them by behavior: how fast they request, which networks they come from and which URLs they hit. That is where rate limits and firewall rules do the work, because a scraper that ignores robots.txt simply skips the rules your file sets.
What can robots.txt do, and what does it leave open?
robots.txt tells compliant crawlers which URLs they may request. RFC 9309, the standard behind it, describes a protocol for crawler instructions, not a security boundary. A crawler that has not committed to following it can ignore it, and anyone who knows a URL can still request it.
It also does not remove a page from Google's index. Google's robots.txt introduction says to use it for URLs Google should not crawl at all, and the crawl budget guide says not to use it to temporarily reallocate crawl budget. Canonical tags and noindex guide what gets indexed, but they do not reliably stop requests from arriving, as Google's canonicalization guide explains.
Crawl rate has its own rules. Googlebot does not process the crawl-delay directive, as Google's explainer on crawl budget says, so adding it will not slow Google down. Bingbot honors crawl-delay, and Bing's Crawl Control tool can reduce crawling during busy hours.
Compliance also varies from crawler to crawler. Researchers at the University of Chicago and UC San Diego found that robots.txt compliance is uneven, especially among non-search crawlers. So, use robots.txt for what it does well, telling well-behaved crawlers to skip filter, sort, search and session URLs, and put enforcement somewhere else. Our guide to sitemaps and robots.txt covers the file itself.
In what order should you cut the cost?
Work from the cheapest and safest step to the most disruptive. Each one is easier to judge once the one before it has changed your log numbers.
- Measure first: Turn on request and origin logging with retention if your logs lack detail, then repeat the seven-day check.
- Cache public pages: Cache anonymous HTML, listing pages, images, CSS and JavaScript at the edge, and keep query-string variants out of the cache key unless they change the response.
- Tighten URL combinations: Decide which filter, sort and pagination URLs deserve indexing, remove unneeded internal links to the rest and disallow them for compliant crawlers.
- Add limits before the application: Put rate limits and firewall or bot rules in front of your app, start in monitor mode and return a 429 where it fits.
- Limit by identity and cost: Slow or block SEO-tool, AI and scraper traffic only after you have confirmed who it is and what it costs.
Each platform names these controls differently. Cloudflare documents rate limiting rules, bot management and caching, where bot management assigns bot scores and needs an explicit rule to block or challenge. Amazon CloudFront explains how caching reduces requests sent to the origin, and Vercel's firewall rules can log, deny, challenge or rate-limit traffic before it reaches the application.
If stable directory and catalog pages are built on every request, pre-building them as static HTML or incrementally regenerated pages removes the application work for each visit, human or not. As a rough guide, caching and edge rules are a small to medium job, tangled filter and pagination URLs a medium one, and a database that struggles under catalog pages a medium to large one.
Before any rule goes live, test it against Googlebot and Bingbot. After the change, check Search Console crawl data and Bing verification, then compare your billing and origin numbers with the baseline week. If indexing or leads drop, roll the rule back and look again.
Do you need a new host to fix this?
Often a configuration change solves more than a migration. If the expense comes from pages built on every request, open URL parameters or missing edge rules, the same problems follow you to a new host, because a new host does not remove crawler traffic on its own.
Before paying for a move, find which layer is expensive: the CDN, the origin, the database or the functions. Bring four numbers to that decision: total requests, origin requests, cache hits and costly requests by path. If the bill's own line items are what confuse you, ask your host what each charge counts and which site or project it belongs to.
Outside help earns its fee when the site has a large number of URL patterns, database-backed listings, several layers of caching, or a real risk of blocking Googlebot or Bingbot by mistake. You can read the logs yourself. The harder part is changing several layers without mistaking cache hits for origin work or trusting a spoofed user agent. Our post on slow server response and AI crawlers covers the speed side of the same load.
What can you stop worrying about?
Not every crawl is a ranking emergency. Google's crawl budget guide is aimed mainly at sites with at least 1 million unique pages, or at least 10,000 pages that change rapidly, and Google says sites outside those conditions generally need only an up-to-date sitemap and routine indexing checks.
A crawler with an AI name is not automatically an attack, and noindex or canonical tags will not cut your request count. Aim for fewer expensive requests reaching your application while the crawlers you need still get stable pages.
Can you cut bot costs and keep Google?
Pick an answer to begin.
1. Does Googlebot process the crawl-delay directive in robots.txt?
2. A log entry says "Googlebot" in the user agent. What should you do before allowing or blocking it?
3. What does robots.txt do?
Frequently Asked Questions About Bots Increasing Hosting Costs
Does robots.txt stop every bot?
No. It guides compliant crawlers, and it is not a firewall. Rate limits and firewall rules are what stop requests from reaching your application.
Will blocking Googlebot save money?
It may reduce Google's requests, but it also removes Google's ability to crawl the blocked URLs. If search traffic matters to you, keep verified Googlebot and Bingbot and limit other traffic instead.
Does Google honor crawl-delay?
No. Google says its crawler does not process the crawl-delay directive. Bingbot does honor it.
Should I block all AI crawlers?
Decide by purpose and cost. Search, user-requested retrieval and training crawlers serve different functions, and blocking one name may not block the others.
How do I verify Googlebot?
Check the request against Google's published verification methods, which include reverse DNS and forward DNS lookups. A user agent string alone is not proof.
Can caching reduce crawler cost?
It can reduce origin work when repeated requests share cacheable responses, but billing depends on the platform's pricing model. Check which requests your host counts.
What This Means for You
Bots raise a bill when their requests reach your application and trigger work there. Count them in a seven-day log, keep verified Googlebot and Bingbot, then cache public pages, tighten URL combinations that have no search value and put rate limits and firewall rules in front of the application. Treat robots.txt as a request to well-behaved crawlers, and limit SEO-tool, AI and scraper traffic only after you know who it is and what it costs.
Done in that order, the free steps come first and may be enough: a smaller load on your origin, fewer billing surprises and a clear record of what changed. You also keep the search traffic that brought the attention in the first place.
If your site has many URL patterns, database-backed listings or several caching layers, or you want a second set of eyes before a rule touches Googlebot or Bingbot, Web Leveling can read your logs with you and tell you which layer is costing you. Our web hosting work keeps the pages that visitors and bots request as finished files on accounts in your name, which keeps the work behind each request small. We work with small and medium businesses across the country and overseas. Send us your top five bots and your request totals, and we will tell you where to start.
Terms
Crawler and hosting words in this post
Tap a term to see what it means.
Origin. The server or application behind your CDN that builds a page when no stored copy is available.
Cache hit. A request answered from a stored copy at the edge, without reaching your origin.
User agent. The name a browser or bot gives itself in a request, which can be faked.
ASN. A number that identifies the network an IP address belongs to.
WAF. A web application firewall, a set of rules that allows, blocks or challenges requests before they reach your application.
Rate limit. A cap on how many requests one visitor or group can make in a set time.
Crawl budget. The amount of crawling a search engine will spend on a site, which Google's guidance treats as a concern mainly for very large or fast-changing sites.
