WP Engine, one of the world's largest managed WordPress hosting platforms, is silently blocking AI search crawlers, ClaudeBot (Anthropic), GPTBot (OpenAI), and Amazonbot. At the infrastructure level, with no customer opt-out possible, according to an investigation published May 6, 2026 by Will Scott on Search Engine Land. The documented result over 30 days: 0% Claude AI citation presence on affected sites.
TL;DR: If your site runs on WP Engine, AI search engines (Claude, and partially ChatGPT and Perplexity via Amazonbot) likely can't crawl it properly. 29% of ClaudeBot and GPTBot requests receive HTTP 429. Here's how to check and your four options if you're affected.
The Numbers That Matter
Will Scott, founder of Search Influence, analyzed 7 days of server logs (April 4–10, 2026) on his own WP Engine-hosted site. The data is unambiguous:
For context: the same site recorded 37.8% citation presence in Google AI Mode, 22.2% in Copilot, 9.6% in ChatGPT, but zero in Claude. The correlation is direct and measurable.
Cloudflare's Q1 2026 crawl-to-referral analysis puts this in perspective: ClaudeBot makes 20,583 crawl requests per referral it sends back. GPTBot: 1,255 to 1. Every 429 rejection is therefore a page that will never exist in their answers.
What WP Engine Is Actually Doing
The blocking rule operates at WP Engine's infrastructure layer, below the WordPress application. It cannot be seen or disabled from the customer dashboard. WP Engine confirmed this officially to Search Engine Land: "Platform-wide rate limiting on certain high-impact bots…can't be selectively disabled per bot."
Attempted workaround via the Web Rules Engine (IP allowlist): confirmed ineffective. "Allowing AI bot IPs via Web Rules Engine does not override WP Engine's platform-wide rate limiting rules, which operate at the infrastructure level."
Bots that are not blocked: ChatGPT-User (0% rate-limited), PerplexityBot (0% rate-limited), anthropic-ai (older Anthropic user-agent), CCBot (Common Crawl). Bytespider (TikTok), however, receives a full HTTP 520 in 61% of cases.
How Other Managed WordPress Hosts Compare
| Host | AI Bot Policy |
|---|---|
| WP Engine | ❌ Platform-level block, no opt-out |
| Kinsta | ✅ No platform block; opt-in Bot Protection |
| Pressable | ✅ No default block; managed via robots.txt |
| Pantheon | ✅ Explicitly does not block identified bots |
| Flywheel (WP Engine subsidiary) | ⚠️ Undocumented despite parent policy |
This creates a clear dividing line in the managed WordPress market. While Google's cryptographic bot authentication system aims to increase trust in legitimate AI crawlers, WP Engine moves in the opposite direction: block by default, with no transparency to site owners.
Check if your site is accessible to AI crawlers, within 24h
How to Check if You're Affected
Three-step diagnostic (requires the curl command-line tool):
- Simulate ClaudeBot user-agent: Run repeated requests against your domain with the ClaudeBot UA string, then compare with a browser request. A discrepancy in HTTP codes (200 vs 429) confirms UA-based blocking.
- Identify your host: Inspect response headers (
x-powered-byorserverfields) to confirm who manages your infrastructure. - For WP Engine customers: Check that « Utilities > Redirect Bots » is OFF and Web Rules contain no AI user-agent blocks. If 429s persist → escalate to product engineering support.
Your Four Options on WP Engine
- Escalate to product engineering, WP Engine acknowledges an « exceptional use case » pathway for sites with documented GEO strategies.
- Try the Web Rules Engine allowlist, confirmed by WP Engine as ineffective against platform-level rules. Worth attempting, but expect no result.
- Migrate to a host without platform-level blocking, Kinsta, Pressable, and Pantheon are all documented as AI-crawler-transparent.
- Accept the block as deliberate policy, only valid if AI search citations are not part of your growth strategy.
Why This Changes the GEO Conversation
We've documented extensively that your site's accessibility to AI agents is a foundational prerequisite for appearing in AI-generated answers. This WP Engine situation adds a new layer: your host can invisibly block you from AI search, regardless of your content quality, robots.txt configuration, or GEO strategy. See our complete method for appearing in AI Overviews, hosting transparency is now part of that checklist.
Cicéro's Take
Zero Claude AI citations over 30 days. That's not a content problem or a structural problem. It's an infrastructure problem most site owners don't even know they have. Before optimizing your content for AI, verify that AI can actually read your site. It's the simplest check in 2026, and the most overlooked.
Sources
- → Search Engine Land (Will Scott), « Your managed WordPress might be blocking AI bots and you can't see it » (May 6, 2026)
- → Cloudflare Q1 2026 Crawl-to-Referral Analysis. Bot crawl efficiency data cited in SEL investigation
- → WP Engine, Official statement confirmed by Will Scott (May 6, 2026)
Growth and SEO content strategist, I founded Cicéro to help businesses build lasting organic visibility. On Google and in AI-generated answers alike. Every piece of content we produce is designed to convert, not just to exist.
LinkedIn