Track AI Bots Visiting Your Site
AI Crawler Logs shows you when AI platform bots visit your website. Understanding when and how AI crawlers access your content helps you optimize for AI visibility.
Important: How AI Crawler Detection Works
AI crawlers like GPTBot, ClaudeBot, and PerplexityBot do not execute JavaScript - they only fetch raw HTML. This means JavaScript-based tracking cannot detect actual crawler visits.
To track AI crawler visits, you need server-side tracking, which is available on Growth and Business plans.
Tracking Methods
| Method | What It Tracks | Plans |
|---|---|---|
| JavaScript Script | AI referral traffic (humans clicking from AI answers) | All plans |
| Server Log Import | Actual AI crawler visits from your Nginx/Apache logs | Growth, Business |
| Cloudflare Worker | Real-time crawler detection at the edge | Growth, Business |
What Are AI Crawlers?
AI crawlers are bots that visit websites to gather information for AI platforms:
- GPTBot - OpenAI's crawler for ChatGPT
- PerplexityBot - Perplexity's web crawler
- ClaudeBot - Anthropic's crawler for Claude
- Google-Extended - Google's AI training crawler
- Bingbot - Microsoft's crawler (feeds Copilot)
Setting Up Server Log Import (Growth/Business)
Import your Nginx or Apache access logs to detect AI crawler visits:
- Go to AI Crawler Logs > Setup
- Select the Server Log Import tab
- Optionally configure Nginx to create a separate AI crawler log
- Upload your log file (supports .log, .txt, .gz up to 50MB)
- Surva.ai parses the logs and extracts AI crawler visits
Nginx Configuration (Optional)
Add this to your nginx.conf to create a dedicated AI crawler log:
map $http_user_agent $is_ai_crawler {
default 0;
~*GPTBot 1;
~*ClaudeBot 1;
~*PerplexityBot 1;
~*Google-Extended 1;
~*anthropic-ai 1;
}
access_log /var/log/nginx/ai_crawlers.log combined if=$is_ai_crawler;
Setting Up Cloudflare Worker (Growth/Business)
Deploy a Cloudflare Worker for real-time crawler detection:
- Go to AI Crawler Logs > Setup
- Select the Cloudflare Worker tab
- Copy your unique API key
- Create a Worker in Cloudflare and paste the provided code
- Route the Worker to your domain
- Crawler visits are tracked automatically in real-time
Dashboard Overview
The Crawler Logs dashboard shows:
- Total Bot Visits - All AI crawler visits detected
- Visits by Bot - Breakdown by crawler type
- Top Pages Crawled - Which pages bots visit most
- Crawl Frequency - How often each bot visits
- Crawl Timeline - Visit patterns over time
Understanding Crawler Behavior
Look for patterns in crawler visits:
- Frequency - More visits often means more indexing
- Pages visited - Which content interests AI bots
- Time patterns - When crawlers are most active
- New page discovery - How quickly bots find new content
Robots.txt Considerations
Your robots.txt file controls which bots can crawl your site. To allow AI crawlers:
User-agent: GPTBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: /
Note: Blocking AI crawlers may reduce your AI visibility.
Privacy and Data
Server-side tracking only records:
- Bot user agent (which crawler)
- Page URL visited
- Timestamp of visit
- IP address (for verification)
No personal user data is collected - only AI bot visits are tracked.