# A curated map of this site for answer engines and LLM agents lives at
# https://checklist.gg/llms.txt (llmstxt.org convention). It is a hand-written
# index of the ~45 pages worth reading, not a dump of the 1,100-URL sitemap.
# It is advisory only - no crawler is required to fetch it, and nothing here
# depends on it.
#
# IMPORTANT, before editing any group below. Under the longest-match rules in
# RFC 9309 a crawler obeys exactly ONE group: the one whose user-agent token is
# the most specific match for its name, and it ignores every other group
# including "*". So a named group does not layer on top of "*" - it REPLACES
# it. That is why the two /api lines are repeated verbatim in every named group
# rather than being written once under "*". Delete them from a group and that
# crawler starts walking /api/revalidate.
#
# Any crawler NOT named below falls into the "*" group and is therefore
# allowed. That is the deliberate default: this site wants to be read. The
# named groups exist to make the AI-crawler position explicit and, more
# importantly, editable one bot at a time - not to restrict anything that "*"
# would otherwise permit. Every group in this file currently resolves to the
# same effective policy; what differs is that each one can now be flipped on
# its own without touching the others.

User-agent: *
Allow: /

# The remaining /api handlers are machinery, not pages: /api/revalidate and
# /api/revalidatePlanner are secret-guarded webhooks with nothing to index.
Disallow: /api/

# ...except /api/og, which renders the Open Graph card image. Every page points
# og:image, twitter:image and the schema.org ImageObject contentUrl at it, and
# Twitterbot and Googlebot-Image honour robots.txt when fetching those, so
# blocking it would strip the preview image off every share and every rich
# result. The longer Allow beats the shorter Disallow under the longest-match
# rule in RFC 9309, so this line wins for /api/og only.
Allow: /api/og


# ---------------------------------------------------------------------------
# Answer-engine crawlers that SEND TRAFFIC. Allowed deliberately.
#
# These either build the index an assistant cites from, or fetch a page live
# because a user asked about it. Blocking any of them removes checklist.gg
# from the answer, which is the opposite of what this site wants. They are
# NOT the same thing as the training crawlers further down - vendors
# deliberately split the two so a site can accept citations while declining
# training, and the names are easy to confuse.
# ---------------------------------------------------------------------------

# OAI-SearchBot indexes pages so they can be surfaced and linked in ChatGPT
# search results. It is not a training crawler - that is GPTBot, below.
User-agent: OAI-SearchBot
Allow: /
Disallow: /api/
Allow: /api/og

# ChatGPT-User fetches a page live when a ChatGPT user or an agent acting for
# one follows a link. User-initiated, not bulk crawling, not training.
User-agent: ChatGPT-User
Allow: /
Disallow: /api/
Allow: /api/og

# Claude-SearchBot indexes pages so Claude can cite them in search results.
User-agent: Claude-SearchBot
Allow: /
Disallow: /api/
Allow: /api/og

# Claude-User fetches a page live on behalf of a Claude user who asked about
# it. User-initiated, not training.
User-agent: Claude-User
Allow: /
Disallow: /api/
Allow: /api/og

# PerplexityBot builds the index Perplexity cites and links from.
User-agent: PerplexityBot
Allow: /
Disallow: /api/
Allow: /api/og

# Perplexity-User fetches a page live when a Perplexity user follows a link.
User-agent: Perplexity-User
Allow: /
Disallow: /api/
Allow: /api/og

# Bingbot is Bing's search crawler and also the basis of Microsoft Copilot's
# grounding. Blocking it costs both search and assistant visibility.
User-agent: Bingbot
Allow: /
Disallow: /api/
Allow: /api/og

# meta-externalfetcher fetches an individual page to support a product action
# a user took - following a shared link, asking Meta AI about a specific URL.
# It is the user-initiated counterpart to meta-externalagent below, in the same
# way ChatGPT-User is to GPTBot, and it is the one of the pair that can put
# checklist.gg in front of a person.
User-agent: meta-externalfetcher
Allow: /
Disallow: /api/
Allow: /api/og


# ---------------------------------------------------------------------------
# Permission tokens, not crawlers.
#
# Neither of these fetches anything. They are opt-out signals that govern what
# an already-crawled copy of this site may be used for. Listing them with
# Allow: / is the documented way to state consent explicitly instead of
# leaving it to the default.
# ---------------------------------------------------------------------------

# Google-Extended governs whether content Googlebot already fetched may be
# used to ground and train Gemini and AI Overviews. It has NO effect on
# Google Search crawling, indexing or ranking - that is Googlebot, which
# stays on the "*" group above. Allowed because AI Overviews is answer-engine
# traffic, which is the point of this file. Note this token mixes grounding
# and training together; there is no way to accept one and decline the other.
# The /api lines are inert for a token that never fetches, but they are kept
# so this group cannot silently widen access if that ever changes.
User-agent: Google-Extended
Allow: /
Disallow: /api/
Allow: /api/og

# Applebot-Extended governs whether content Applebot already fetched may be
# used to TRAIN Apple foundation models. It does not control whether the site
# appears in Siri or Spotlight results - that is Applebot, which stays on the
# "*" group. So this one is training-only: allowing it buys no traffic.
# Left allowed to match the site's prior effective policy; see NOTE below.
# /api lines kept for the same reason as Google-Extended above.
User-agent: Applebot-Extended
Allow: /
Disallow: /api/
Allow: /api/og


# ---------------------------------------------------------------------------
# Training crawlers. Allowed only because that was already the effective
# policy under "*" - naming them here changes nothing, it just makes the
# existing position visible.
#
# NOTE FOR A HUMAN: every group in this section, plus Applebot-Extended
# above, is training-only. Allowing them does not win a single citation or
# click. They are listed separately precisely so that opting out is a
# one-word edit per group (Allow: / -> Disallow: /) without touching any of
# the traffic-driving crawlers above. That is a business decision, not an SEO
# one, so it has been left as-is rather than changed unilaterally.
# ---------------------------------------------------------------------------

# GPTBot collects training data for OpenAI models. Blocking it does NOT
# remove checklist.gg from ChatGPT search - that is OAI-SearchBot, above.
User-agent: GPTBot
Allow: /
Disallow: /api/
Allow: /api/og

# ClaudeBot collects training data for Anthropic models. Blocking it does NOT
# remove checklist.gg from Claude's search results - that is Claude-SearchBot.
User-agent: ClaudeBot
Allow: /
Disallow: /api/
Allow: /api/og

# CCBot is Common Crawl. It sends no traffic and cites nobody; its archive is
# a standard input to third-party training corpora, so allowing it is an
# indirect, irrevocable grant to everyone downstream. Strongest candidate in
# this file for a Disallow if training is unwanted.
User-agent: CCBot
Allow: /
Disallow: /api/
Allow: /api/og

# Amazonbot feeds Amazon search and Alexa answers, and is also documented as
# training input. Mixed purpose, so it sits here rather than above.
User-agent: Amazonbot
Allow: /
Disallow: /api/
Allow: /api/og

# meta-externalagent crawls for Meta AI, covering both product grounding and
# model training with no way to separate them. Mixed purpose.
User-agent: meta-externalagent
Allow: /
Disallow: /api/
Allow: /api/og


Sitemap: https://checklist.gg/sitemap.xml