Can AI crawlers read your content?

Start with the page

Readability, access, and visibility are different checks

A page can be reachable while its main information is missing from the response. A readable page can still be excluded from search. An eligible page may never be selected for a particular answer. Each question needs its own evidence.

Google's current AI search guidance starts with useful original content and ordinary search requirements. A page must be indexed, eligible to show a search snippet, and included in Google's generative AI features. Meeting those conditions does not guarantee that it will appear.

Useful information

Give readers an answer they can use

Explain who you help, what you deliver, the conditions that matter, and what supports your claims. A worked example, a clear method, or a documented result can answer a question that a generic summary leaves open.

Keep page titles and structured data consistent with the visible content. Google says its AI features need no special schema, Markdown, or llms.txt file. There is no prescribed page length or need for a separate page for every wording of a question. Read Google's guidance and common misconceptions.

Access choices

Search discovery and training have different controls

Review robots.txt, page indexing directives, and hosting or firewall rules together. A robots allow rule does not rule out an access challenge elsewhere. For real crawler access, inspect available logs and the provider's published verification guidance; a request with a copied user-agent name is only a simulation.

Google Search generative AI

In Search Console, check Settings > Search generative AI. Inclusion is the default, but a property can inherit another property's setting. This control covers participation in the listed AI features; it does not control model training. Check the control and inheritance rules.

OpenAI search: OAI-SearchBot

This crawler supports discovery in ChatGPT search. OpenAI documents its robots.txt setting and published IP ranges separately from the training crawler.

OpenAI training: GPTBot

GPTBot collects content that may be used for model training. Its control is independent of OAI-SearchBot, so permitting search does not require permitting this training crawl.

User-requested visits: ChatGPT-User

This agent handles certain user-initiated visits. It is not the automatic search crawler, and OpenAI says robots.txt rules may not apply to these requests.

The OpenAI distinctions above follow the official crawler documentation. An observed visit, search eligibility, and permission to use content for training are different facts.

Measurement

Count appearances, visits, and inquiries separately

Google's Generative AI performance report currently reports impressions for AI Overviews and AI Mode. It offers page, date, country, and device views, not a separate AI query, click, CTR, or conversion breakdown. Its data overlaps the Web performance report, so do not add the totals together.

If the report is absent, check the property's inclusion setting and available impression data. Google also lists access availability as a possible cause. Search Labs experiments are excluded from this report.

Use your website analytics and inquiry records to understand subsequent visits and business results. A citation observed in one answer is a dated observation. It does not establish how often customers see it, how many visit, or whether the visit leads to work.