Skip to main content
Submit a request

Ensuring Ad Crawler Access

Updated

A Guide to ads.txt Accessibility and Protecting Your Ad Revenue

This guide explains the critical role of the ads.txt. Understanding how this file can be blocked accidentally and the steps needed to ensure it remains accessible to all advertising partners is crucial for protecting your revenue.

Why Crawlers Access Matters

The ads.txt (Authorized Digital Sellers) file is a text file that lives on your server and serves as a public, machine-readable declaration of who is authorized to sell your inventory.

Humans do not visit your site's ads.txt file. Instead, the file's entire audience consists of bots sent by ad exchanges, SSPs, and DSPs such as Google, Amazon, and The Trade Desk. Before they bid on your inventory, their crawlers quickly check this file.

Ensuring Ads.txt Access

One of two things typically blocks bots from accessing your file: overzealous security firewalls or a misconfigured robots.txt file.

Source #1: CDN and Hosting Firewalls

A security layer called a Web Application Firewall (WAF) often protects your website. It can be part of a Content Delivery Network (CDN), such as Cloudflare, a service from your hosting platform (e.g., AWS), or a plugin on your site (e.g., for WordPress).

  • Create a specific "allow" rule in your firewall settings to grant open access to your ads.txt file. Since the list of crawlers is constantly changing, the most effective and future-proof strategy is to allow all traffic to access the /ads.txt URL path regardless of source IP.
  • Instructions for your Technical Contact: "Please create an exception in our WAF (e.g., Cloudflare, AWS WAF) that identifies requests for the /ads.txt path and ensures they are allowed through. The goal is to ensure this one file path is not affected by security rules that block bot traffic."

Source #2: The robots.txt File

The robots.txt file contains instructions for crawlers. Legitimate bots check this file to see which parts of your site they are permitted to access.

  • Ensure no rules block access to /ads.txt. If any rules do, add a specific rule that explicitly grants access, overriding other Disallow rules.

  • Example of an Allow rule that would ensure bots can access /ads.txt:

    User-agent: *
Allow: /ads.txt

Related articles