# kilowatts.uk robots.txt # Named bots first (Allow), then a SINGLE catch-all * (Allow). # Never add a second "User-agent: *" group. # OpenAI's ad checker reads this file first and stops if it is not # explicitly allowed. Keep these as their own groups. User-agent: OAI-AdsBot Allow: / User-agent: OAI-SearchBot Allow: / # --- Semrush (Site Audit + rank tracker) --- User-agent: SiteAuditBot User-agent: SEMrushBot User-agent: SemrushBot Allow: / # --- Major search engines --- User-agent: Googlebot User-agent: Googlebot-Image User-agent: Storebot-Google User-agent: Google-InspectionTool User-agent: AdsBot-Google User-agent: Google-Feedfetcher User-agent: Feedfetcher-Google User-agent: Bingbot User-agent: BingPreview User-agent: DuckDuckBot User-agent: Applebot User-agent: BraveBot User-agent: Yandex User-agent: Baiduspider User-agent: Slurp Allow: / # --- Screaming Frog SEO Spider --- User-agent: Screaming Frog SEO Spider Allow: / # --- SEO tools --- User-agent: AhrefsBot User-agent: AhrefsSiteAudit User-agent: KilowattsLinkSpider Allow: / # --- Social / link previews --- User-agent: meta-externalagent User-agent: facebookexternalhit User-agent: Facebot User-agent: Twitterbot User-agent: LinkedInBot User-agent: Discordbot User-agent: WhatsApp User-agent: Slackbot User-agent: TelegramBot User-agent: Pinterestbot Allow: / # --- AI search / answer engines (FULL site — must not fall under *) --- # OpenAI user fetches. OAI-AdsBot and OAI-SearchBot are allowed above. User-agent: ChatGPT-User # Anthropic Claude (search + user fetch — Semrush flags these if missing) User-agent: Claude-SearchBot User-agent: Claude-User User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Claude-Web # Perplexity User-agent: PerplexityBot User-agent: Perplexity-User # Google AI / Gemini grounding User-agent: Google-Extended # Others used in AI answers User-agent: YouBot User-agent: Amazonbot Allow: / # --- AI training crawlers only (llms files, not full site) --- User-agent: GPTBot User-agent: CCBot User-agent: Bytespider User-agent: GrokBot User-agent: xAICrawler User-agent: Diffbot Allow: /llms.txt Allow: /llms-full.txt Allow: /llms.json Disallow: / # --- Everyone else: crawl public pages --- User-agent: * Allow: / # --- Sitemap --- Sitemap: https://kilowatts.uk/sitemap.xml # LLMs: https://kilowatts.uk/llms.txt # LLMs-Full: https://kilowatts.uk/llms-full.txt # LLMs-JSON: https://kilowatts.uk/llms.json