Blog SEO basics
robots.txt and sitemap.xml in plain English
October 6, 2026
Two of the most consequential files on your site have no design, no copy, and rarely get opened after launch. robots.txt is the doorman: it tells crawlers where they may go. sitemap.xml is the directory: it lists every page you want found. When either is wrong, everything downstream of it is wrong too.
robots.txt: the doorman
A healthy small-business robots.txt is short:
User-agent: * Allow: / User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / Sitemap: https://yourdomain.com/sitemap.xml
The dangerous versions come from copy-paste: a staging file with Disallow: / that shipped to production, a theme tutorial's rules blocking CSS and JS so pages render broken to crawlers, or a security service injecting AI-crawler disallows you never chose. Remember that robots.txt is public; read yours at yourdomain.com/robots.txt today.
sitemap.xml: the directory
The sitemap is an XML list of your URLs, ideally with last-modified dates. Crawlers use it to find pages your navigation does not surface and to notice updates sooner. WordPress SEO plugins and most static generators produce one automatically; your only jobs are to confirm it exists, contains your real pages rather than junk archives, and is referenced from robots.txt.
The five-minute audit
- Open /robots.txt: does it allow what you want crawled, including AI bots, and nothing you meant to hide is listed in it?
- Open /sitemap.xml: does it load, and are the URLs in it the pages you actually care about?
- Does robots.txt point to the sitemap?
Our scan probes both files on every run, checks HTTPS enforcement alongside them, and tests AI crawler access against what robots.txt claims, because the file saying "allowed" while the firewall says 403 is a real and common failure.
Quick answers
What should a normal robots.txt contain?
For most business sites: allow everything, point to your sitemap, and explicitly allow the AI crawlers you want. Blocking admin paths is fine; blocking CSS, images, or whole sections usually backfires.
Does my site need a sitemap.xml?
Yes. It is a machine-readable list of your pages that helps crawlers find everything, notice updates faster, and see your site's real size. Most platforms generate one automatically.