See which AI crawlers read your site

Why it matters

Your server log records every request to your site, including every visit from AI and search crawlers. Reading it shows whether AI tools reach your pages at all, where they hit errors, and how many visitors only pretend to be famous bots.

The free Log Analyzer reads the file for you in seconds and deletes it afterwards.

Before you start

• Your access log, as a .log or .gz file up to 50 MB
• One to seven days of log is usually enough

Step by step

1. 1

   Download your access log

   In cPanel: Metrics, then Raw Access. On a server: /var/log/nginx/access.log.

   cPanel›Metrics›Raw Access
2. 2

   Upload it to the Log Analyzer

   Up to 50 MB. A .gz file works too. It is deleted after reading.

   Tools›Log Analyzer
3. 3

   Check "Blocked by your site"

   If a search crawler is blocked, AI search cannot read those pages. Often a firewall setting.

4. 4

   Check "Fake bots"

   Requests that used a bot's name from the wrong address. Safe to block.

5. 5

   Check "Asked about by people"

   Pages AI tools read because someone asked about them: your most valuable pages for AI.

If something goes wrong

I cannot find the log

In cPanel look under Metrics, then Raw Access. Other hosts call it Access Logs. On your own server, try /var/log/nginx/access.log or /var/log/apache2/access.log.

The report finds no crawlers

Check that the file is an access log, not an error log, and that it is in the common "combined" format.

A search crawler shows as blocked

Look for a firewall, CDN or security plugin rule that rejects bots, and allow the crawler there. Robots.txt alone does not cause a 403.

What to do next

Fix any blocks, then install the tracking snippet so new visits are recorded every day without uploading logs.