# robots.txt for eternity2.dev # # This is an open, non-commercial, educational site about the Eternity II # puzzle. Everything here is meant to be read, indexed, learned from, and # cited, by humans, by search engines, and by AI crawlers and agents alike. # # Policy: everyone is welcome. We explicitly name the major AI crawlers below # (both the training crawlers and the live search/retrieval/agent fetchers) so # the intent is unambiguous, then fall through to a blanket allow for every # other user-agent. There is nothing to gate: there is no server, no user # data, and no paywall, the whole site is static. # --------------------------------------------------------------------------- # OpenAI (https://platform.openai.com/docs/bots) # --------------------------------------------------------------------------- User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # --------------------------------------------------------------------------- # Anthropic (https://docs.anthropic.com/en/docs/claude-code / crawler docs) # --------------------------------------------------------------------------- User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: anthropic-ai Allow: / # --------------------------------------------------------------------------- # Google (Gemini / Vertex / NotebookLM training + AI products) # Google-Extended is a robots.txt control token, not a live crawler. # --------------------------------------------------------------------------- User-agent: Google-Extended Allow: / User-agent: GoogleOther Allow: / User-agent: Google-CloudVertexBot Allow: / # --------------------------------------------------------------------------- # Perplexity # --------------------------------------------------------------------------- User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # --------------------------------------------------------------------------- # Microsoft / Bing (Copilot) # --------------------------------------------------------------------------- User-agent: bingbot Allow: / # --------------------------------------------------------------------------- # Apple Intelligence (Applebot-Extended is a control token) # --------------------------------------------------------------------------- User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / # --------------------------------------------------------------------------- # Meta AI # --------------------------------------------------------------------------- User-agent: meta-externalagent Allow: / User-agent: FacebookBot Allow: / # --------------------------------------------------------------------------- # Amazon # --------------------------------------------------------------------------- User-agent: Amazonbot Allow: / # --------------------------------------------------------------------------- # Other AI / search engines we welcome by name # --------------------------------------------------------------------------- User-agent: DuckAssistBot Allow: / User-agent: MistralAI-User Allow: / User-agent: cohere-ai Allow: / User-agent: CCBot Allow: / User-agent: Bytespider Allow: / # --------------------------------------------------------------------------- # Everyone else, humans, search engines, agents, anything not named above. # --------------------------------------------------------------------------- User-agent: * Allow: / # Sitemap + AI-readable site maps # # sitemap.xml covers every page in every language (with hreflang alternates). # The llms.txt files are the agent-oriented map of the same content, one per # language; llms-full.txt is that language's whole research corpus in a single # fetch. Sitemap: is the only directive crawlers parse here, so the llms files # are listed as comments for the humans and agents that read this file directly. # # https://eternity2.dev/llms.txt (English) + /llms-full.txt # https://eternity2.dev/fr/llms.txt (French) + /fr/llms-full.txt # https://eternity2.dev/es/llms.txt (Spanish) + /es/llms-full.txt # # Every research page also has a raw-markdown twin at its own URL + ".md". Sitemap: https://eternity2.dev/sitemap.xml