User-agent: * Allow: / Disallow: /r/ # Agent instructions: https://www.crawlcheck.io/agents.md # Content map: https://www.crawlcheck.io/llms.txt # Entity graph: https://www.crawlcheck.io/entitymap.json # Data policy and opt-out: https://www.crawlcheck.io/policy Sitemap: https://www.crawlcheck.io/sitemap.xml Llms: https://www.crawlcheck.io/llms.txt EntityMap: https://www.crawlcheck.io/entitymap.json