Which Southeast Asian Websites Block AI Crawlers? A robots.txt Study of Top Domains (2026)
We read robots.txt on 3,000 top Southeast Asian domains. 12.4% block GPTBot, 2.2% block OAI-SearchBot, and Cloudflare wrote 61% of the AI blocks.
- Sample
- 3,000 top domains, 500 per country
- Method
- robots.txt and llms.txt read on every domain, parsed to RFC 9309