Is Your Website Blocking AI Search Crawlers? A Practical Site Audit

In short: Publishing useful pages does not ensure that an AI search system can access them. A blocked crawler, broken navigation or poor page structure can limit discovery. An audit can identify obstacles, but fixing a score does not guarantee an AI citation.

Disclosure: This article contains Semrush affiliate links. Dijipedya may receive a commission if you sign up through them, at no extra cost to you. We have not run a live audit of a customer site for this article.

First, separate access from inclusion

There are two different questions: can a search crawler access a page, and will an AI system use that page in an answer? The first is a technical check. The second depends on the user’s question, the available sources and the system’s behavior. Removing a block can make a page eligible for discovery; it is not a promise of inclusion or traffic.

Semrush’s Site Audit AI Search Health panel checks technical readiness and whether major AI-related bots are restricted. According to Semrush, the score draws on crawlability, structured data, internal linking and bot access. A separate “Blocked from AI Search” widget identifies restricted bots and affected pages.

A five-step technical check

1. Run a crawl. Create a Semrush Site Audit campaign for the domain. Semrush documents an OpenAI-Search user agent setting that lets you inspect how its crawler sees the site.
2. Review the AI Search Health widget. Treat the score as a summary that points to issues. Open the underlying findings before drawing a conclusion.
3. Inspect bot restrictions. Check the “Blocked from AI Search” widget and the site’s robots.txt rules. A deliberate restriction may reflect a publisher’s policy; record why it exists before changing it.
4. Check affected pages. In the Issues tab, filter for AI Search problems. Prioritize important public pages with crawl or linking problems, and confirm their responses and internal paths.
5. Re-run the audit and compare. Verify that the issue has actually cleared. Then use AI Visibility Toolkit reports, if subscribed, to watch mentions and citations separately over time.

What the tool can and cannot establish

The audit can flag blocked bots and common technical barriers. It cannot prove that ChatGPT, Google AI Mode or another assistant will cite a page after a fix. Semrush’s AI Search Health documentation says its crawler tests access and structure; those are prerequisites and signals, not a distribution guarantee. A useful editorial page still needs clear information, accurate claims and a reason to be referenced.

A second common mistake is treating all AI crawlers as interchangeable. Semrush lists several user agents from different services. Their roles and site policies may differ. Audit each relevant bot and understand your publishing policy before changing robots.txt globally.

A practical priority order

Start with accidental blocks on pages you actually want discovered. Next fix broken internal links and inaccessible navigation. Then review machine-readable structure and whether the page clearly answers the target question. Finally measure the brand’s presence in answers and citations; do not use an improved technical score as a substitute for outcome data.

Which pages should you check first?

Start with the pages that explain what the business offers: product pages, service descriptions, documentation and the strongest explanatory articles. A blocked archive page is a different priority from a blocked product page. Confirm that a crawler can follow internal links to the page and receive usable content, then inspect its canonical and structured information where relevant.

Do not change a site-wide robots.txt rule only because a dashboard calls it a warning. A publisher may intentionally restrict a bot. Document the existing rule, identify the affected user agent, and decide whether the desired outcome is search discovery, model training restriction, or both. These are policy choices as well as technical settings.

Interpret the health score without overpromising

AI Search Health provides a useful starting signal, while its issues list is more actionable than the headline score. Semrush says it incorporates bot access, crawlability, structured data and internal linking. A rise in the score can confirm that selected technical checks improved. It does not measure whether an assistant selected your page as a source in a particular answer.

Use a second measurement for that question. Semrush’s Visibility Overview separates brand mentions, citations and cited pages; Prompt Tracking can examine a fixed set of questions. Reading both layers together helps you distinguish “our site can be accessed” from “our brand is appearing in relevant answers.”

How to verify a fix

Record the original issue and affected URL, make a focused change, re-run the crawl and check the same URL again. Review the response from the relevant crawler user agent if the issue concerned bot access. Only then compare mentions or citations over subsequent reporting periods. AI answers vary, so a one-day movement is not a reliable measure of success.

Access and pricing

Semrush describes AI Search Health within Site Audit and also references it in the AI Visibility Toolkit workflow. Plans and limits vary. The standalone AI Visibility Toolkit is currently listed at $99 per month with no free trial, according to Semrush’s knowledge base. Check the relevant product and checkout page before choosing a plan.

Frequently asked questions

Will allowing a crawler guarantee an AI citation?

No. Access removes one possible obstacle; it does not determine which source an assistant uses in an answer.

Should I unblock every AI bot?

No universal rule fits every publisher. Identify the specific bot and the purpose of your restriction, then decide according to your own publishing policy.

What should I review after a Site Audit?

Open the affected pages and issues, verify crawl access and linking, then compare later mentions and citations as separate outcome measures.

Is AI Search Health the same as AI Visibility Score?

No. AI Search Health summarizes technical readiness in Site Audit; AI visibility reports measure brand presence in sampled answers.

Sources (accessed September 29, 2026): Semrush, AI Search Health in Site Audit; Semrush, Blocked from AI Search; Semrush, AI Visibility Toolkit.