Build Logs & Industry Playbooks

Can AI Assistants Read Your Website? A Practical Check

By Mudabbir Hasan, Co-founder · Published · Last updated

To be recommended by AI assistants, your site first has to be readable by them. Check that robots.txt does not block their crawlers, that your key content is in the HTML the server sends and not only built by JavaScript, and that pages return correct status codes. An llms.txt file is cheap to add, but none of the vendor documentation we read mentions it.

What decides whether an AI assistant can use your site?

Before an assistant can cite or recommend you, something has to fetch your pages and read them. That depends on three things you control: whether the crawler is allowed in, whether the content is in the HTML it receives, and whether the page answers with the right status code. None of this guarantees a recommendation. It only removes the common reasons for being invisible.

Three gates

  1. 1

    Allowed in?

    robots.txt does not block the crawler.

  2. 2

    Content in the HTML?

    The headline and key text arrive without running scripts.

  3. 3

    Correct status code?

    A missing page returns a real 404.

Which crawlers should robots.txt allow?

OpenAI documents three agents, and its settings work independently. OAI-SearchBot surfaces sites in ChatGPT's search features, and OpenAI recommends allowing it. GPTBot crawls content that may be used to train its models, and disallowing it signals that your content should not be used for training. ChatGPT-User is used when a person's action in ChatGPT triggers a page visit, and because users initiate those requests, OpenAI says robots.txt rules may not apply. OpenAI notes it can take about 24 hours for a robots.txt change to take effect in search.

Our robots.txt names the main AI crawlers, for example GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended, and gives them the same rules as everyone else: allowed everywhere except the API, build and thank-you paths. We want to be found, so we do not block training crawlers. If you feel differently, you can allow search crawlers and block training ones. Check each vendor's own documentation for what its crawler does, because we have only read OpenAI's in full.

OpenAI's three agents

  • OAI-SearchBot

    Surfaces sites in ChatGPT search. OpenAI recommends allowing it.

  • GPTBot

    May feed model training. Disallowing says no.

  • ChatGPT-User

    Visits a page when a person asks. robots.txt may not apply.

Source: OpenAI.

Is your content in the HTML the server sends?

Google says server-side or pre-rendering is still a great idea, partly because not all bots can run JavaScript. Google itself renders JavaScript in a queue that can take longer than a few seconds, and it uses the rendered HTML to index a page. A crawler that does not run scripts sees only what the server sent. If your headline and main text appear only after scripts run, those bots may see an empty shell.

Frameworks differ here. In Next.js, which our site uses, layouts and pages are Server Components by default, and the rendered result plus any Client Components is used to prerender HTML. Client Components are for interactivity and browser-only features. A page built this way arrives with its content already in the HTML.

How do you test it?

  • Fetch the page without a browser: run curl against your page URL and search the output for your headline and a key sentence. If they are missing, scripts are building your content.
  • Use Google's URL Inspection tool or the Rich Results Test to see the rendered HTML Google indexes. Google's guidance is that content not visible in the rendered HTML cannot be indexed.
  • Check your links are real anchor elements with an href, not fragments or script-only navigation.
  • Check status codes. A missing page should return a real 404, not a 200 with an error message.
  • Open your robots.txt and look for blanket disallow rules left over from a staging site.

Test it without a browser

# Does the headline arrive without a browser?$ curl -s https://example.com/page | grep -i "<h1"<h1>Your headline appears here</h1> # No output? Scripts are building your content.

What about llms.txt?

llms.txt is a plain-text file at the root of your site that lists your key pages for language models. We publish one, generated from our own page data so it stays current. None of the vendor documentation we read for this article mentions it, so we cannot say any assistant uses it. Treat it as a low-cost extra and not as a ranking lever.

What actually helps you get recommended?

We have no evidence that any single file or setting causes an assistant to recommend a site, and we would be wary of anyone who claims one does. What the checks above do is make sure you are not invisible by accident. After that, it comes down to having specific, sourced pages that answer real questions, and being mentioned on other sites.

Website design and development services

What should you do this week?

  • Run the curl test on your homepage, a service page and an article.
  • Read your robots.txt and decide, crawler by crawler, what you want to allow.
  • Inspect two important pages in Google's URL Inspection tool.
  • Add an llms.txt if you want one, and make it generate from your content so it cannot go stale.

Sources

Related guides

Talk to an engineer about your project

Start with a free project call. We learn what you are building and recommend the right next step, with no pitch.

Book a call