# Gridiron & Wine — robots.txt # Automated collection of this site's content or data (scraping, bulk download, # dataset building, AI/ML training) requires our prior written consent. # See https://gridironandwine.com/terms.html#automated-access # # 2026-07-21: admin / internal / member pages are de-indexed via # X-Robots-Tag: noindex in _headers, deliberately NOT with Disallow here. # Disallow stops the fetch, so Googlebot never sees the noindex and the URL # can still be indexed url-only from any external link. Only paths that must # never be fetched at all are disallowed below. # # /js/ and /css/ are intentionally NOT disallowed — Google needs JS and # CSS to render pages for indexing (recommended since 2014). # # 2026-10-02 (antiscrape1002): grouped by crawler type. A crawler that matches # a named group IGNORES the "*" group, so every group repeats its own rules. # Disallow lines come before "Allow: /" so first-match parsers (e.g. Python's # urllib.robotparser) and longest-match parsers (Google, Bing) agree. The AI group # comes first because substring-matching parsers would otherwise give e.g. # Applebot-Extended the Applebot (search) group. # ── 1. AI training / AI answer crawlers and dataset scrapers: no access ───── # (OpenAI and Perplexity moved to group 2 on 2026-10-02 — Shane: allowed. geo1028 (2026-10-28): the other # answer-time fetchers — Claude-User, Claude-SearchBot, DuckAssistBot, MistralAI-User — moved to group 2 as well. # Model-TRAINING crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot ...) stay here.) User-agent: CCBot User-agent: Claude-Web User-agent: anthropic-ai User-agent: Google-CloudVertexBot User-agent: Bytespider User-agent: TikTokSpider User-agent: Amazonbot User-agent: FacebookBot User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: Diffbot User-agent: omgili User-agent: omgilibot User-agent: Webzio-Extended User-agent: cohere-ai User-agent: cohere-training-data-crawler User-agent: AI2Bot User-agent: Ai2Bot-Dolma User-agent: Timpibot User-agent: ImagesiftBot User-agent: YouBot User-agent: PanguBot User-agent: iaskspider User-agent: ICC-Crawler User-agent: VelenPublicWebCrawler User-agent: FirecrawlAgent User-agent: img2dataset User-agent: PetalBot User-agent: DataForSeoBot User-agent: magpie-crawler User-agent: Scrapy Disallow: / # ── 2. AI search / answer fetchers + 4 training crawlers (Shane geo1028d) (OpenAI + Perplexity 2026-10-02; Claude, DuckDuckGo, Mistral 2026-10-28) ───────── # Public pages are open so ChatGPT search, Perplexity, Claude, DuckDuckGo AI answers and Le Chat can read and cite them. # Training crawlers stay BLOCKED (group 1). Grok and Copilot need no line: Grok's fetcher has no published token # (falls to "*", group 4) and Copilot / ChatGPT search also use Bing's index (Bingbot, group 3). # Raw data files and endpoints stay closed, exactly as for "*" below. Disallow # lines come before "Allow: /" (same first-match / longest-match reasoning). User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-User User-agent: Claude-SearchBot User-agent: DuckAssistBot User-agent: MistralAI-User User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended Disallow: /netlify/ Disallow: /.netlify/ Disallow: /data/ Disallow: /feeds/ Disallow: /odds_board/ Disallow: /chart_data.json Disallow: /power_projections.json Disallow: /worldcup_model_feed.json Disallow: /x_feed.json Disallow: /*.json$ Disallow: /*.json? Allow: / # ── 3. General-purpose search engines ─────────────────────────────────────── # Full access to pages. The public JSON that pages load to RENDER their content # (/feeds/, /odds_board/, /power_projections.json, ...) stays fetchable for these # crawlers ONLY so Power Rankings / Analytics / Odds Board render for indexing; # those files carry X-Robots-Tag: noindex so they never appear in results. User-agent: Googlebot User-agent: Googlebot-Image User-agent: Googlebot-News User-agent: Bingbot User-agent: msnbot User-agent: Slurp User-agent: DuckDuckBot User-agent: Applebot User-agent: Yandex User-agent: Baiduspider Disallow: /netlify/ Disallow: /.netlify/ Disallow: /data/ Allow: / # ── 4. Everyone else (incl. link-preview fetchers) ────────────────────────── # Pages and images stay open so shared links keep their previews # (facebookexternalhit, Twitterbot, LinkedInBot, Slackbot, Slackbot-LinkExpanding, # Discordbot, WhatsApp, TelegramBot all fall through to this group). Data files # and endpoints are closed. User-agent: * Disallow: /netlify/ Disallow: /.netlify/ Disallow: /data/ Disallow: /feeds/ Disallow: /odds_board/ Disallow: /chart_data.json Disallow: /power_projections.json Disallow: /worldcup_model_feed.json Disallow: /x_feed.json Disallow: /*.json$ Disallow: /*.json? Allow: / Sitemap: https://gridironandwine.com/sitemap.xml