---
title: What is AI crawler? · Aeonav
description: A bot that fetches web pages for AI training or live retrieval, such as GPTBot or PerplexityBot.
canonical: https://www.aeonav.fivecoder.com/glossary/ai-crawler
updated: "2026-09-15"
---

[AI search glossary](http://www.aeonav.fivecoder.com/glossary)

# AI crawler

A bot that fetches web pages for AI training or live retrieval, such as GPTBot or PerplexityBot.

AI crawlers are how engines read your site, whether to train on it or to retrieve it at answer time. If your important pages are blocked or slow, you make yourself harder to cite.

Some tools track these visits as a signal of AI interest in your content. At minimum, make sure your robots rules do not accidentally shut the door on engines you want to reach.

## Go deeper

- [Robots.txt and AI crawlers](http://www.aeonav.fivecoder.com/blog/robots-txt-ai-crawlers)
- [How many websites block AI crawlers?](http://www.aeonav.fivecoder.com/blog/how-many-websites-block-ai-crawlers)

## Related terms

[llms.txt](http://www.aeonav.fivecoder.com/glossary/llms-txt)[Retrieval-augmented generation](http://www.aeonav.fivecoder.com/glossary/retrieval-augmented-generation)

## See it in your own data

Aeonav is an open-source AI visibility platform. Self-host it for free and track how AI answer engines mention and cite your brand.

[Get started](http://www.aeonav.fivecoder.com/docs/getting-started)[Compare tools](http://www.aeonav.fivecoder.com/ai-visibility-tools)