Skip to main content
Submit a request

Managing Web Crawler Traffic

Updated

1. What Are Web Crawlers?

Web crawlers (also known as bots or spiders) are automated programs that visit websites to gather data. They may be:

  • Search engine crawlers (e.g., Googlebot, Bingbot) → Index content for search results.

  • Ad crawlers (e.g., AmazonAdBot, AppNexusBot) → Scan content and ads.txt for ad targeting and eligibility.

  • Monitoring/security bots → Check uptime or scan for vulnerabilities.

Crawlers identify themselves by their user agent string and typically follow the rules set in robots.txt.

2. Why Do Crawlers Matter?

  • Search traffic → Blocking search crawlers can hurt SEO visibility.

  • Advertising revenue → Blocking ad crawlers may prevent them from:

    • Accessing ads.txt, which is required to validate authorized sellers.

    • Scanning content for contextual targeting, which can affect CPMs.

  • Server performance → Some crawlers can hit sites very frequently, creating strain on hosting resources.

3. Options for Managing Crawlers

Allowing Crawlers (Recommended Default)

  • Ensures maximum visibility for both search engines and ad partners.

  • No revenue impact.

Throttling / Frequency Capping in robots.txt

  • Effect: Reduces server strain by slowing down crawl frequency, without fully blocking.

  • Directive: Crawl-delay (note: not all crawlers honor this).

  • Benefit: Maintains contextual scanning while reducing server load.

Example:

User-agent: ExampleBot
Crawl-delay: 10

Blocking Crawlers at the Firewall

  • Effect: Completely prevents the crawler from accessing the site, including ads.txt.

  • Risk: Can directly reduce ad revenue if ad crawlers cannot confirm ads.txt or scan pages.

  • Use case: Only for malicious/abusive crawlers (scrapers, spam bots).

Blocking Crawlers via robots.txt

  • Effect: Crawler cannot access certain pages or directories, but can still fetch ads.txt.

  • Risk: Prevents contextual scanning, which may lower CPMs for ad partners.

  • Use case: Limit access to sensitive or non-monetized areas of the site.

Example:

User-agent: ExampleBot
Disallow: /

4. Best Practices

  1. Never block ad crawlers via firewall – this prevents ads.txt access and can impact revenue.

  2. Use robots.txt for crawl control – safe way to manage frequency or scope of crawling.

  3. Monitor logs – identify which bots are hitting the site most often, using IPs and user agents.

  4. Whitelist legitimate crawlers – search engines, ad crawlers, monitoring services.

Related articles