AI Can Read Your Website. Here's How We Know Ours Can.

Strip away the acronyms, and whether a search engine or AI model can use your website comes down to three questions.

• Can the crawler get in? Every website uses a file called robots.txt, along with signals like meta robots tags and X-Robots-Tag headers, to tell crawlers what they’re allowed to access. If your site blocks GPTBot or ClaudeBot, those AI crawlers won’t see your content. In most cases, this is what someone really means when they claim a website “can’t be read by AI.”
• Can the crawler see the content? Access doesn’t guarantee visibility. Many modern websites rely on JavaScript to load their content after the initial page opens. A person visiting the site may see a polished, complete page, while a crawler only finds an empty shell. It’s a common issue—and one that can quietly undermine an otherwise well-built website.
• Can the crawler understand it? Once the content is visible, its structure determines how easily a search engine or AI model can interpret it. Clear headings, descriptive page titles and meta descriptions, canonical tags, clean URLs, and structured data all help provide context. This area is evolving quickly, and the industry is still learning what makes content most useful to AI models.

The first two questions have clear, technical answers. If crawlers can access your pages and see their content, saying that “AI can’t read your website” isn’t accurate. The third question—how well AI can understand and use that content—is where the ongoing work of answer engine optimization really begins.



We ran a Curaytor website that hasn’t received any additional support from our Managed Growth team through CrawlerCheck, a free public tool that tests whether a website is accessible to AI crawlers.

We chose a tool anyone can use because we wanted our claim to be easy to verify, not something you simply had to take our word for. The result was clear: the site is accessible to AI crawlers.

You can review the full test results here. We’re happy to share them, and you can use the same tool to test any website for yourself.

Search engines, AI bots, social bots, SEO tools—the full list received the same result: 100% allowed. CrawlerCheck tested 34 AI crawlers, and the Curaytor site allowed access to all 34, with zero blocked. That includes GPTBot, ClaudeBot, PerplexityBot, and every other AI crawler included in the test.

The report also found a valid robots.txt file and a live XML sitemap. Together, these files tell crawlers what they’re allowed to access and help them find the content across the site. This didn’t require any custom work from our Managed Growth team or an additional package. It’s built into the Curaytor platform.

Curaytor sites also include an editable llms.txt section in the backend, giving website owners another way to provide AI models with clear information about their site. If llms.txt is new to you, we explain what it is and how it supports real estate SEO here. You can also see how easy it is to edit llms.txt within the Curaytor platform.

When we inspected what a crawler actually retrieves from a Curaytor page, here’s what came back:

• The page’s full text, rendered and readable, not an empty JavaScript shell
• A complete title tag and a meta description written for that specific page
• Open Graph and Twitter Card data, which control how the page appears when shared
• A canonical URL that identifies the authoritative version of the page
• A clean semantic structure with real headings, descriptive link text, and logical navigation

Everything an AI model would need to understand the page and answer a question about that market is available in plain text. You don’t have to take our word for it. The test results show exactly what a crawler can access.

What matters most is that we didn’t make any changes to the site before running this test. Every Curaytor site is built with these fundamentals already in place, including crawler access, rendered content, metadata, canonical tags, and clean URLs. An agent shouldn’t have to understand the technical details or know which features to request. This should all come standard.

AEO will continue to change, and we’ll keep improving the platform as it does. But we also believe every client deserves a strong technical foundation from the day their site launches, whether they’ve spent years learning about AEO or haven’t given it a single thought.

Here’s where we won’t overstate the case. Answer engine optimization is still new, and anyone claiming to have completely “solved” it is getting ahead of the evidence. AI models regularly change how they find, interpret, and cite information. The tools used to measure AI visibility are still developing, and some of the best practices being shared today look different from the advice we heard six months ago.

The fundamentals are much clearer. An AI system can’t use content it can’t access, read, or understand. Once those technical requirements are met, useful, specific, and regularly updated content has the best chance of being surfaced. There’s no trick to that. It’s the same principle that has always guided good search strategy, now applied to the way people find information through AI.

That’s what our content engine is designed to support. Curaytor clients consistently publish neighborhood guides, market updates, and answers to the questions local buyers and sellers are actually asking. That content lives on a technical foundation AI crawlers can access and understand.

We’ve seen the value of pairing the right foundation with an ongoing marketing strategy. The Linnell Group, for example, has sustained a 5x return on ad spend across nine years with a Curaytor-run program. Our platform also draws on proprietary campaign data each week, helping us improve the work based on what’s producing results for real estate teams right now.