# ======================================== # Opagio — robots.txt # https://opag.io # Updated: 2026-04-20 # # POLICY: Explicitly permissive for indexing and live AI retrieval. # We want to be found, indexed, cited in AI answers, and included in # RAG/knowledge-base systems. We do NOT grant rights to use Opagio # content as training data for model weights. # ======================================== # ── Default: ALLOW EVERYTHING ────────────── User-agent: * Allow: / Allow: /llms.txt Allow: /llms-full.txt Allow: /ai.txt # Private/authenticated routes only Disallow: /api/ Disallow: /dashboard Disallow: /account Disallow: /auth/ Disallow: /valuator/reports/ Disallow: /questionnaire/*/report Disallow: /questionnaire/*/results Disallow: /admin Disallow: /billing Disallow: /org/ Disallow: /portfolio/ Disallow: /webhooks/ Disallow: /platform/ Disallow: /prototypes/ # Old WordPress cruft — block crawling to save crawl budget Disallow: /element_category/ Disallow: /fusion_tb_category/ Disallow: /author/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /wp-admin/ # ── SEARCH ENGINE CRAWLERS ───────────────── # Explicit allow for clarity (inherits from * above) User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: bingbot Allow: / User-agent: Applebot Allow: / User-agent: DuckDuckBot Allow: / User-agent: DuckAssistBot Allow: / User-agent: Slurp Allow: / User-agent: Yandex Allow: / User-agent: Baiduspider Allow: / User-agent: BraveBot Allow: / User-agent: Sogou Allow: / User-agent: Naver Allow: / # ── AI / LLM CRAWLERS — WELCOME FOR INDEXING & CITATION ─── # We explicitly permit indexing, retrieval, and citation of # our public content in live AI answers and RAG systems. # We do NOT grant rights to use this content as training data # for model weights (see Content-Signal: ai-train=no above). # OpenAI User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / # Anthropic (Claude) User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: claude-web Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: anthropic-ai Allow: / # Google AI User-agent: Google-Extended Allow: / User-agent: Googlebot-AI Allow: / User-agent: GoogleOther Allow: / User-agent: GoogleOther-Image Allow: / User-agent: GoogleOther-Video Allow: / User-agent: google-vertex-ai Allow: / User-agent: GoogleAgent-Mariner Allow: / User-agent: Google-CloudVertexBot Allow: / # Apple AI User-agent: Applebot-Extended Allow: / # Microsoft / Copilot User-agent: CopilotBot Allow: / # Perplexity User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # You.com User-agent: YouBot Allow: / # Cohere User-agent: Cohere-Crawler Allow: / User-agent: cohere-ai Allow: / # Meta / Facebook AI User-agent: Meta-ExternalAgent Allow: / User-agent: meta-externalagent Allow: / User-agent: Meta-WebIndexer Allow: / User-agent: meta-webindexer Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: FacebookBot Allow: / # Amazon User-agent: Amazonbot Allow: / # xAI (Grok) User-agent: Grokbot Allow: / User-agent: xAI-Bot Allow: / # Mistral AI User-agent: MistralBot Allow: / User-agent: MistralAI-User Allow: / # DeepSeek User-agent: DeepSeekBot Allow: / User-agent: deepseek-ai Allow: / # Qwen / Alibaba User-agent: QwenBot Allow: / User-agent: Alibaba-Bot Allow: / # ByteDance / TikTok User-agent: Bytespider Allow: / User-agent: TikTokSpider Allow: / # Common Crawl User-agent: CCBot Allow: / # AI21 Labs User-agent: AI2Bot Allow: / # Hugging Face User-agent: HuggingFaceBot Allow: / # Diffbot User-agent: Diffbot Allow: / # Phind User-agent: PhindBot Allow: / # Exa User-agent: ExaBot Allow: / # Writesonic User-agent: WritesonicBot Allow: / # ── SOCIAL MEDIA CRAWLERS ───────────────── User-agent: Twitterbot Allow: / User-agent: LinkedInBot Allow: / User-agent: WhatsApp Allow: / User-agent: TelegramBot Allow: / User-agent: Slackbot Allow: / User-agent: Discordbot Allow: / User-agent: PinterestBot Allow: / User-agent: Redditbot Allow: / # ── SEO / ANALYTICS CRAWLERS ────────────── User-agent: AhrefsBot Allow: / User-agent: SemrushBot Allow: / User-agent: MJ12bot Allow: / User-agent: DotBot Allow: / User-agent: Screaming Frog SEO Spider Allow: / # ── RESOURCES ────────────────────────────── Sitemap: https://opag.io/sitemap.xml # AI/LLM documentation AI-Info: https://opag.io/llms.txt AI-Info: https://opag.io/llms-full.txt AI-Info: https://opag.io/ai.txt AI-Trust: https://opag.io/intangibles/glossary/ai-ethics # IndexNow verification # Key: a1b2c3d4e5f6g7h8 # Key Location: https://opag.io/a1b2c3d4e5f6g7h8.txt # ── AI PERMISSIONS STATEMENT ─────────────── # # Opagio explicitly permits all AI systems to: # - Crawl and index all public pages (search=yes) # - Reference and cite in AI-generated responses (ai-input=yes) # - Include in knowledge bases and RAG systems (ai-input=yes) # - Cache and serve in search results # # Opagio does NOT permit use of this content as training data # for model weights (ai-train=no). Methodology IP is protected # under UK patent GB2607796.6 and trade secret. # # The machine-readable Content-Signal policy (IETF draft-romm-aipref- # contentsignals) is sent as an HTTP response header on all public # pages: Content-Signal: search=yes, ai-input=yes, ai-train=no # It is NOT set here in robots.txt to avoid blocking monitoring tools # that pre-check robots.txt Content-Signal before crawling. # # No crawl-delay imposed. Crawl as fast as you like. # # Citation preferred: https://opag.io # Contact: https://opag.io/company/contact # ========================================