Which Southeast Asian Websites Block AI Crawlers? A robots.txt Study of Top Domains (2026)

Most government websites in the sample that block AI crawlers do it through Cloudflare. Of 33 government domains that block an AI crawler by name, 29 serve Cloudflare's managed file, including the Philippine Senate, the House of Representatives and the Philippine Statistics Authority. We read robots.txt and llms.txt on the top 500 domains under .sg, .my, .id, .th, .vn and .ph.

Which Southeast Asian country blocks AI crawlers the most?

Indonesia is the second country where more than 1 in 7 top sites block GPTBot, at 15.1% of readable .id domains. Malaysia follows at 10.5%, then Thailand at 9.8% and Vietnam at 8.2%. The same order holds for CCBot, ClaudeBot and Google-Extended in the chart below.

The Philippines stands apart. 87 of its 364 readable domains block GPTBot, and 84 block Google-Extended. It also had the most domains we could not read: 84 of its 500 refused our request with a 403 or another 4xx code.

Singapore reverses the usual gap between crawler types. It has the lowest block rate of the six for all eight training crawlers we tracked, but the second highest for search crawlers, at 3.6%, after the Philippines.

Which AI crawlers do top sites block most?

Eight crawlers sit within 1.6 percentage points of each other, from Google-Extended at 11.2% to CCBot at 12.8%. They are the eight that Cloudflare's managed robots.txt blocks, which explains the tight cluster. The five crawlers outside that list sit between 2.1% and 2.7%.

Sites that write their own rules pick a different order. Among the 117 domains that block an AI crawler by name without Cloudflare's file, the most blocked tokens are:

• CCBot: 69 domains. Common Crawl's CCBot page gives Disallow: / as the way to opt out.
• GPTBot: 62 domains. OpenAI says GPTBot collects content that "may be used in training our generative AI foundation models."
• Bytespider: 61 domains. ByteDance publishes no crawler documentation we could reach, so the token comes from the crawler's own user agent string.

Google-Extended and Applebot-Extended trail at 34 hand-written blocks each. Neither is a crawler that fetches pages. Google says Google-Extended is a control token for Gemini training and grounding, and Apple says Applebot-Extended "does not crawl webpages."

How much of the blocking comes from Cloudflare?

180 of the 185 Cloudflare-managed files that block AI crawlers name the same eight tokens: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. The file starts with a legal notice and then a block like this one, read from psa.gov.ph:

# BEGIN Cloudflare Managed content

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

User-agent: Amazonbot
Disallow: /

Cloudflare's managed robots.txt documentation says the service prepends this block to a site's existing file, or creates the file when the site has none. Its Content-Signal line sets search to yes and AI training to no.

Remove Cloudflare and the Philippines and Singapore look alike. Hand-written rules block an AI crawler by name on 7.1% of readable Philippine domains and 7.2% of Singapore ones. The managed file adds 15.7% in the Philippines and 3.1% in Singapore.

The managed file leaves OAI-SearchBot, Claude-SearchBot and PerplexityBot untouched, so it does not stop a site from appearing in those AI search products. I would still check for it first in a GEO audit, because Cloudflare adds the block at the edge and it does not appear in the site's own CMS or code.

Do top sites block AI search crawlers as well?

66 readable domains, 2.9%, block at least one of the three search crawlers: OAI-SearchBot, Claude-SearchBot or PerplexityBot. 339, or 14.8%, block at least one training crawler.

Most search crawler blocks are not aimed at AI. Of the 50 domains that block OAI-SearchBot, 40 do it through a User-agent: * group that disallows everything. Wildcard groups also account for 40 of the 59 PerplexityBot blocks.

Named blocks can carry exceptions. The New Straits Times (nst.com.my) disallows OAI-SearchBot, ChatGPT-User, PerplexityBot and both Claude agents, but opens one section to them:

User-agent: *
Disallow:
Content-Signal: search=yes,ai-input=no,ai-train=no

User-agent: OAI-SearchBot
Allow: /plus
Disallow: /

foodpanda.sg does the reverse, letting OpenAI's search and user agents in while keeping out its training crawler:

User-agent: OAI-SearchBot
Crawl-delay: 5
Allow: /

User-agent: GPTBot
Crawl-delay: 5
Disallow: /

The split matters because the agents do different jobs. OpenAI's crawler documentation says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers."

OpenAI also says robots.txt rules "may not apply" to ChatGPT-User, since a person starts those visits. Perplexity's crawler page says of its Perplexity-User fetcher: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

Do government and university sites block AI crawlers?

4 of 332 readable government domains write their own AI crawler rules, 1.2%. Another 29 block through Cloudflare's managed file, 13 of them in the Philippines. We sorted domains by their registry suffix: gov, go, mil and mi count as government; edu, ac and sch count as education.

Universities show the same pattern: 37 of the 43 education domains that block an AI crawler by name use the managed file. On commercial and other domains, the managed file accounts for 119 named blocks and hand-written rules for 107.

Government sites are also the most likely to close their doors to all bots. 10 government domains, 3.0%, serve User-agent: * with Disallow: /, among them Singpass, the Bangko Sentral ng Pilipinas and Thailand's e-procurement site.

What do AI crawler rules on large regional sites look like?

The Indonesian newspaper Kompas (kompas.id) opens its robots.txt with a notice that prohibits using its content for "the development of any software, machine learning, artificial intelligence (AI), and/or large language models (LLMs)." It then lists 19 user agents in a single group ending in one rule:

User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: claude-user
User-agent: ClaudeBot
User-agent: OpenAI
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: CCBot
User-agent: Baiduspider
User-agent: PerplexityBot
User-agent: cohere-ai
User-agent: DuckAssistBot
User-agent: quillbot.com
User-agent: YouBot
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: AliyunSecBot
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
Disallow: /

The Straits Times domain straitstimes.com.sg redirects its robots.txt to straitstimes.com. That file blocks Google-Extended except on URLs carrying a query flag:

User-agent: Google-Extended
Disallow: /
Allow: /*?*ai-allowed=1

amazon.sg disallows 11 of the 13 agents we tracked, including OAI-SearchBot, Claude-SearchBot and ChatGPT-User. It has no group for Amazonbot, Amazon's own crawler, or for Applebot-Extended, so both fall back to its general rules.

Which sites publish an llms.txt file?

Singapore leads at 11.6% of its 500 domains, and Indonesia trails at 5.6%. The highest-ranked adopters include Lazada, which serves an llms.txt on its Singapore, Malaysia, Thailand, Philippines and Vietnam domains, and the Vietnamese news sites Dân trí, VietNamNet, Tuổi Trẻ and Thanh Niên.

The llms.txt proposal, published by Jeremy Howard in September 2024, asks for a markdown file whose first element is an H1 with the site's name. 233 of the 276 files we found open that way.

• WordPress plugins wrote 55: All in One SEO 23, Yoast SEO 21 and Rank Math 11, each opening with a "Generated by" line.
• 407 domains return a web page with status 200 at /llms.txt, shopee.sg among them.
• 16 more serve an empty file or a copy of their robots.txt rules.

Checking the status code alone would have credited those 407 domains with an llms.txt they do not have. Government domains lag here too, at 2.3%.

Which old AI crawler tokens do sites still block?

Anthropic's crawler support page lists three agents: ClaudeBot, Claude-User and Claude-SearchBot. Sites in the sample name older tokens more often than two of them.

A stale token can leave a gap. 10 files name anthropic-ai but not ClaudeBot, among them espn.com.sg, pep.ph and the three JobStreet domains. Under RFC 9309, ClaudeBot then follows each site's User-agent: * group instead of the block the site meant for Anthropic.

Kompas lists "OpenAI" as a user agent, a token missing from OpenAI's crawler page, which documents GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User.

Methodology

Sample. We downloaded Tranco list 46VQX, generated on September 12, 2026 from 30 days of CrUX, Farsight, Majestic, Cloudflare Radar and Umbrella data. We took the 500 highest-ranked domains ending in .sg, .my, .id, .th, .vn and .ph, second-level forms such as .com.sg and .co.id included.

All six country domains have more than 500 entries in the list, so n is 500 for each.

"robots.txt file" includes 153 domains that answered with an HTML page and status 200; those parse as having no rules. "Readable" is the denominator for all block rates. We excluded domains that refused us, served a bot wall or did not answer, since we could not read their policy.

Fetch. On September 13, 2026, between 14:23 and 14:56 UTC, we requested https://<domain>/robots.txt and /llms.txt, and nothing else. We tried the www host when the apex gave no response, a 5xx or more than five redirects.

Requests followed up to five redirects, timed out after 15 seconds and identified as SeotalResearch/1.0 (robots.txt study; +https://www.seotal.com/). Each domain got at least 1.1 seconds between requests, with 8 domains in progress at a time for the first 528 and 16 after that.

Parsing. Our parser follows RFC 9309: user agent groups matched on the product token without regard to case, matching groups combined, and the * group used only when no group names the bot.

A bot counts as blocked when a disallow rule covers all paths and no allow rule opens part of the site. A disallow with allow exceptions counts as partial, which is where 16 Google-Extended rules fall. We count a file as Cloudflare-managed when it contains the line # BEGIN Cloudflare Managed content.

Checks. We re-fetched 15 robots.txt files by hand, 10 at random and 5 from blocking domains, and compared each verdict with the parser's: all 15 matched. While testing, we moved 14 bot wall pages served with status 200 out of the readable group, and kept 7 files that carried valid rules under an HTML content type.

Limitations.

• The ccTLD is a proxy. Sites on .com are not in the sample even when their audience is in the region, and 58 readable domains redirected robots.txt to a host outside their country domain.
• robots.txt is a request. It does not enforce anything, and user-initiated agents may not follow it.
• Firewall blocks are invisible here. A site can block AI crawlers in Cloudflare's WAF or any other firewall without a line in robots.txt.
• Our user agent was refused by 333 domains. Those servers may give other visitors a readable file.
• It is one snapshot. Files can change the day after we read them.
• RFC 9309 treats an unreachable file as full disallow. We excluded those 366 domains rather than count them as blocking.

The per-domain results are in ai-crawler-blocking-southeast-asia.csv, one row per domain with a verdict for each bot, and the totals are in the summary JSON.

Get your robots.txt checked by an SEO and GEO specialist

One Cloudflare setting decides whether GPTBot, ClaudeBot and Google-Extended may crawl your pages, while AI search crawlers follow whatever rules you wrote yourself. Seotal matches you with vetted freelance SEO specialists who work on ChatGPT Search, Perplexity, Google AI Overviews and Gemini, using structured data, source citations and llms.txt files.

You get 3 to 5 vetted profiles within 72 hours of your brief. Hourly work costs $15 to $30 with no minimum, and a part-time retainer costs $1,000 to $2,000 a month for 60 to 80 hours, with a 3-month minimum and a 30-day replacement guarantee (see pricing).

For the demand side, see AI search use in Southeast Asia and Southeast Asia SEO statistics. Hire an AI SEO specialist, hire an SEO expert or start with the guides on how AI search finds local businesses, AI Overviews for local businesses, schema and Business Profiles for AI search and SEO and AEO for front-end developers.

AI crawler blocking: frequently asked questions

Does blocking Google-Extended remove a site from Google Search?

No. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." It governs use of content for Gemini training and grounding.

How long does ChatGPT take to respect a robots.txt change?

About a day for search. OpenAI says it can take around 24 hours from a robots.txt update for its search systems to adjust.