SupplyReader
SupplyReader is a crawler that reads the public files publishers and ad platforms put up so that others in advertising can see who is authorised to sell their inventory.
What it reads
/ads.txtand/app-ads.txton publisher and app developer websites (IAB Tech Lab specification)/sellers.jsonon advertising platforms- For advertising companies, their home page and a few of their own pages about partners, quality or certifications
- For apps, their privacy policy page and public app store listings
It does not log in, submit forms, run scripts or collect personal data.
How it behaves
- It identifies itself on every request as
SupplyReader/1.0 (+https://supplyreader.com). - Most files are read about once a month. A file someone is actively monitoring is read every few days.
- It sends no more than two requests at a time to any server, at least a second apart.
- If your server answers 429 or 503, it stops asking for the rest of that run and honours
Retry-After. - If your site does not resolve or keeps failing, it waits longer each time, up to 90 days.
Opting out
To stop SupplyReader reading your site, add this to your robots.txt:
User-agent: SupplyReader
Disallow: /
Or email us with your domain and we will remove it.
Contact
Questions, problems or a request rate you would like changed: crawler@supplyreader.com. We reply within two working days.
Abuse reports: abuse@supplyreader.com.