Dev log
How AI search finds your site: crawlers, robots.txt and llms.txt
AI answers start with a crawler. I explain which bots fetch your pages, which ones train models, and how I let AI search in without giving away content.
5 min read
Dev log Dev
SEO & AI search
How to be found in search engines and AI answers: structured data, crawlers and content.
Dev log
How AI search finds your site: crawlers, robots.txt and llms.txt
AI answers start with a crawler. I explain which bots fetch your pages, which ones train models, and how I let AI search in without giving away content.
5 min read