Unblock the crawlers
- Trigger
- You have never checked your robots.txt against the AI user agents.
- Move
- Allow GPTBot, ClaudeBot, PerplexityBot, Google-Extended and OAI-SearchBot explicitly.
- Why it works
- A blocked crawler describes you from whatever it finds elsewhere. Usually a competitor's comparison page.