# FormulaCompare.com - Robots.txt # Updated: April 2026 User-agent: * Allow: / Allow: /blog/ Allow: /tools/ Allow: /assets/ # Sitemap location Sitemap: https://formulacompare.com/sitemap.xml # LLMs.txt for AI assistants # See: https://formulacompare.com/llms.txt # Extended: https://formulacompare.com/llms-full.txt # ============================================ # Search Engine Crawlers # ============================================ User-agent: Googlebot Allow: / Crawl-delay: 0 User-agent: Bingbot Allow: / Crawl-delay: 0 User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: Baiduspider Allow: / User-agent: YandexBot Allow: / # ============================================ # AI Search & Retrieval Crawlers # ============================================ # OpenAI - ChatGPT, GPT, SearchGPT User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Anthropic - Claude User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: anthropic-ai Allow: / # Google - Gemini AI User-agent: Google-Extended Allow: / # Apple - Apple Intelligence, Siri User-agent: Applebot-Extended Allow: / # Perplexity AI User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Mistral (Le Chat) User-agent: MistralAI-User Allow: / # Meta - Meta AI User-agent: Meta-ExternalAgent Allow: / # Amazon - Alexa, Amazon AI User-agent: Amazonbot Allow: / # Microsoft - Copilot User-agent: Bingbot Allow: / # You.com AI Search User-agent: YouBot Allow: / # DuckDuckGo AI Answers User-agent: DuckAssistBot Allow: / # ============================================ # AI Training Crawlers # ============================================ # Common Crawl (used by many AI models) User-agent: CCBot Allow: / # Cohere AI User-agent: cohere-ai Allow: / # Allen Institute for AI User-agent: AI2Bot Allow: / # ByteDance - Doubao AI User-agent: Bytespider Allow: / # Diffbot Knowledge Graph User-agent: Diffbot Allow: / # Huawei Petal Search AI User-agent: PetalBot Allow: / # ============================================ # Social Media Crawlers # ============================================ User-agent: facebookexternalhit Allow: / User-agent: Twitterbot Allow: / User-agent: Pinterest Allow: / User-agent: LinkedInBot Allow: / User-agent: WhatsApp Allow: / User-agent: TelegramBot Allow: /