Dev log
How AI search finds your site: crawlers, robots.txt and llms.txt
AI answers start with a crawler. I explain which bots fetch your pages, which ones train models, and how I let AI search in without giving away content.
5 min read
Tag
AI search
Dev log
How AI search finds your site: crawlers, robots.txt and llms.txt
AI answers start with a crawler. I explain which bots fetch your pages, which ones train models, and how I let AI search in without giving away content.
5 min read