Can AI crawlers read our website?

“Who can check if AI crawlers can access our website?”

A person sees your website in a browser. An AI crawler often sees something else: the raw HTML, without the scripts that fill in prices, menus and answers. If a fact is not in that raw text, many AI systems cannot use it. They guess, or they take the fact from another site.

This is not a question of design or speed. In one of Avisible’s audits, a tourism operator’s site scored 94% on performance. The same site showed AI almost nothing about prices, trip lengths or FAQ answers.

Below are the five technical issues Avisible sees most often in its audits, plus one thing to check first: whether AI crawlers are allowed in at all.

Avisible recommends checking in this order:

1. Let AI crawlers in. Check robots.txt and your firewall or CDN settings, and allow the crawlers on purpose.
2. Put key facts in the HTML, not in scripts. Turn off JavaScript and open your key pages. What disappears, AI may not see.
3. Write important facts as text. Facts that live only in images or PDFs are invisible to AI.
4. Put the brand name in the body text. Write the name next to the key facts, so a sentence AI lifts still names you.
5. Give every page one H1, a unique title and the right language tag. Titles and headings tell a crawler what a page is about.
6. Add structured data and make your profiles agree. Schema markup, business profiles and directory listings should say the same thing under the same name.

Start with your robots.txt file and your firewall or CDN settings. AI companies use their own crawlers, for example GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic) and PerplexityBot (Perplexity). Google AI Overviews use the normal Google crawler.

If these crawlers are blocked, the AI system relies on other sites to describe you. Blocks are sometimes set by a security setting or a hosting default, not by a decision. Ask your developer to confirm what is allowed, on purpose.

When Avisible crawled the tourism operator’s site without scripts, it read 812 pages. Prices, trip durations, FAQ answers and links to activities were all loaded by script. In the raw text they did not exist.

The result showed up in the AI answers. Asked about prices, AI answered “from €0” or gave no price. On the same site, the cancellation terms were written in plain text. AI repeated those terms correctly every time.

The test is simple: turn off JavaScript and open your key pages. If the price, the product facts or the answer to a common question disappear, AI may not see them either. The fix is usually server-side rendering or putting the key facts in the HTML.

A consumer brand in one of Avisible’s audits had key product information in images with no alt text. To AI, that information was invisible. An international congress organiser had key content locked in PDFs.

AI can sometimes read a PDF, but you cannot rely on it. If a fact matters for a buying decision, write it on the page as text. Keep the image or PDF as an extra, not as the only place where the fact lives.

This one surprises most teams. In 16 answers, AI read the tourism operator’s own page and still did not name the operator. The name was in the logo and the page title, but not in the body text.

AI often lifts a sentence or a paragraph and uses it without the context around it. If the sentence says “we offer husky safaris from Rovaniemi”, the “we” is lost. Write the brand name next to the key facts: “\[Brand\] offers husky safaris from Rovaniemi.” It feels repetitive to a human reader. It is clear to a machine.

Page titles and headings tell a crawler what a page is about. On the tourism operator’s site, 347 pages shared a title with another page. 69 German and French pages had English titles. 459 of the 812 pages had under 300 words.

The congress organiser had no real H1 heading on 56 of its 71 pages. Its llms.txt file returned a 404 error.

Titles also matter outside your own site. An industrial manufacturer’s product was listed by a reseller with a title that left out the model name. In 15 Gemini answers, the product appeared without its name.

Check that every page has one clear H1, a unique title, and a language tag and title in the page’s own language.

Structured data (schema markup) tells machines in a fixed format what a page describes: an organisation, a product, a price, an event. The tourism operator had zero structured data. Adding Organization and Product schema, with sameAs links to your official profiles, helps AI connect the facts to the right company.

Entity data also lives outside your site. The same operator’s Google Business Profile called it a “bicycle rental service”. AI repeated that word for word. The brand was also written in six different ways across the web. A marine services firm had two websites that disagreed on the details of four offices, and AI repeated both versions.

Make your business profiles, directory listings and your own sites say the same thing, with the same name.

Fixing the technology removes obstacles. It does not create visibility on its own. One technology company in Avisible’s data had every page indexable, llms.txt live, all AI crawlers allowed and about 900 words per page on average. It was still almost never named in open questions, where the buyer does not name a brand.

Two other sites had 100% schema coverage and still had large gaps in open questions. Technical work makes sure AI can read you. Content and outside sources decide whether AI chooses you.

Avisible checks how AI crawlers see your website as part of the Visibility Audit. The audit includes a crawl of your site without scripts, so you see what AI can read and what it cannot. Avisible connects the technical findings to real AI answers: which missing fact caused which wrong or empty answer.

Avisible’s AI visibility platform tracks your buyer questions over time, so you can see whether answers change after the fixes go live.

Avisible works with your own developer or web agency. Avisible finds the issues and puts them in order of impact. Your developer makes the fixes.

Avisible offers both: a one-off measurement, either the Visibility Snapshot or the full Visibility Audit, and continuous tracking on the platform. Avisible is an AI visibility tool and service in one, so you can start with a single check and keep tracking when you are ready.

The site findings come from Avisible’s audits, where Avisible crawls the client’s site without running scripts and compares the result with the AI answers it collects (ChatGPT, Gemini, Google AI Overviews and others, in several languages). Figures are from individual client audits and are not averages across all clients.

• Why doesn't AI search find our company?
• How to improve your AI visibility: start with what AI already reads
• What's the difference between SEO and GEO?

Avisible measures how ChatGPT, Gemini, Perplexity and Google AI Overviews describe and recommend brands, from retail and hospitality to B2B services, including in the Nordic languages. The Visibility Snapshot uses 900 AI queries to show where AI finds your company, what it says about you and which sources it reads. Delivered in three working days, €499.