Log-file analysis: where your crawl budget really goes

Search Console summarises how Google crawls your site; your server logs record every request. For large or fast-changing sites, log-file analysis is the most direct way to see where Googlebot’s attention goes: which templates and parameter URLs absorb it, which redirects and errors waste it, and which important pages it rarely visits.

A log-file analysis, step by step

1. CollectOrigin and CDN logs, several weeks of them
2. VerifyKeep only requests from genuine crawlers
3. ClassifyGroup URLs by template, parameter and status
4. CompareAgainst sitemaps, crawls and Crawl stats
5. Actrobots.txt, internal links, lastmod, speed

What a log line contains

A typical line in the widely used “combined” format looks like this:

66.249.66.1 - - [12/May/2026:06:25:14 +0000] "GET /shoes/?colour=red&sort=price HTTP/1.1" 200 48213 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) … (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

It records who made the request (the IP address), when, what was requested (the method and URL), the status code, the size of the response, the referrer and the user agent. Many servers can also log how long each response took, which is worth switching on. That’s enough to answer most crawl questions, provided you keep enough of it.

Getting the logs, and handling them properly

• Origin or CDN. If a CDN serves cached pages, those requests never reach your origin server, so origin logs alone undercount crawling. Most CDNs can export logs, though sometimes only on certain plans.
• Retention. Hosts often keep logs for a short time and then delete them. Set up an export to your own storage before you need the data, and collect several weeks so you see patterns rather than one-off spikes.
• Personal data. Visitor IP addresses are online identifiers, so under UK GDPR you should treat raw logs as personal data. Keep only the fields you need, restrict access, set a retention period, and filter down to verified crawlers or strip visitor IPs before the data goes to anyone for analysis. If an agency does that analysis, it’s processing on your behalf and needs a contract that covers it.

Verifying Googlebot

Anyone can put “Googlebot” in a user agent. To verify a request, run a reverse DNS lookup on its IP address, check that the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the same IP:

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

At scale, match IP addresses against the ranges Google publishes instead. Requests that claim to be Googlebot but fail the check are something else: put them in their own bucket, not in your crawl analysis.

What to analyse

• Hits by section or template. Group URLs by pattern (products, categories, articles, filters, internal search) and compare each group’s share of crawling with its value to you.
• Status codes. The mix of 200, 301, 404, 410 and 5xx responses, over time and by template.
• Parameter URLs. Sorting, filtering, tracking and session parameters multiply URLs. With the URL Parameters tool gone from Search Console, robots.txt and internal linking are your main levers.
• Redirects. Googlebot requesting the same redirected URLs again and again usually means internal links or sitemaps still point at them.
• 404s and soft 404s. 404s show up directly. Soft 404s return 200, so look for templates with unusually small responses and check the Page indexing report.
• Sitemap gaps. URLs that are crawled but missing from your sitemaps (often junk or orphan pages), and sitemap URLs that are never crawled.
• Important pages crawled rarely. Compare when key pages were last crawled with how often they change.

Googlebot requests by URL type over one month on a large shop

Filter and sort parameters38%

Product pages21%

Category pages14%

Redirects11%

404s and soft 404s7%

Articles and guides5%

Other4%

AI crawlers in your logs

You’ll also see AI crawlers, such as GPTBot (OpenAI), ClaudeBot (Anthropic) and PerplexityBot (Perplexity). Report them separately from search engines, check what they request and how often, and decide bot by bot what robots.txt allows. In our log reviews, many of them fetch HTML without the JavaScript files a browser would request, which is one more reason to keep key content in the initial HTML. Don’t look for Google-Extended: it’s a robots.txt token, not a crawler, and Google’s requests arrive under its usual user agents.

Comparing with Crawl stats

Search Console’s Crawl stats report, available for root-level properties, gives Google’s own view: total requests, download size and average response time, broken down by response, file type, purpose (discovery or refresh) and Googlebot type, plus host status for robots.txt, DNS and server connectivity. Use it as a cross-check. If your logs show far fewer Googlebot requests than the report, you’re probably missing a layer, such as the CDN. If the trends agree, the logs give you the URL-level detail the report leaves out.

Acting on what you find

From finding to fix

If the logs show	Then	Effort
Googlebot stuck in filter, sort or calendar URLs	Block the infinite spaces in robots.txt and stop linking to them	Quick win
Repeated hits on redirected URLs	Point internal links and sitemaps at the final URLs	Quick win
404s caused by your own links	Fix the links; keep 404 or 410 for pages that are gone	Quick win
Slow responses or bursts of 5xx errors	Fix server response time, caching and capacity	Urgent
Soft 404s on thin templates	Return a real 404, or give the page real content	Project
Key pages crawled rarely	Strengthen internal links and keep lastmod accurate	Project
Sitemap URLs never crawled	Check they’re linked, canonical and worth indexing	Review

A few of these need care:

• robots.txt for infinite spaces. Disallow the patterns that generate endless combinations, such as sort orders, calendars and internal search results. Blocked URLs aren’t crawled, but robots.txt doesn’t remove URLs that are already indexed, and Google can’t see a noindex on a page it isn’t allowed to crawl. Our guide to faceted navigation covers which filter pages deserve to be indexed.
• Internal links that point at redirects. Update them to the final URL. Google says long redirect chains have a negative effect on crawling, and every extra hop is another request.
• Accurate lastmod. Google uses it when it is consistently and verifiably accurate. A sitemap that stamps every URL with today’s date makes the value useless.
• Server response time. When a site responds quickly, Google can crawl more; slow responses and server errors make it crawl less.

Does crawl budget matter for your site?

Google says crawl budget is mainly a concern for very large sites or sites whose content changes very often, and that most sites don’t need to worry about it. Its rough guide is a million or more unique pages changing weekly, or 10,000 or more changing daily. Large online shops with faceted navigation often fall into that group, which is why logs are worth the effort in e-commerce SEO. On smaller sites, logs are still useful for diagnosing a migration, a rendering problem or unexpected bot traffic, but crawl budget is rarely the bottleneck. Log-file analysis is part of our technical SEO service for exactly these cases.

---

Wondering whether crawling is what’s holding your site back? Start with a free SEO audit: we’ll check crawlability and indexing and tell you what to fix first.