What Is Robots.txt?
Last updated 2026-09-12
robots.txt is a plain-text file at the root of a domain that tells well-behaved web crawlers which parts of a site they may or may not crawl.
Understand what robots.txt controls
It uses "User-agent" lines to target specific crawlers, and "Disallow"/"Allow" lines to block or permit paths from being crawled.
Know what it doesn't control
robots.txt is a crawling instruction, not an indexing one — a disallowed page can still appear in search results (usually without a description) if other pages link to it. Use a noindex meta tag to actually prevent indexing.
Point crawlers to your sitemap
Add a "Sitemap:" line with your sitemap's full URL so crawlers can find your page list directly from robots.txt.
Example
This site's own robots.txt is live at /robots.txt — open it directly to see a real file: User-agent rules plus a Sitemap line pointing at /sitemap.xml.
Important Considerations
- robots.txt is publicly visible to anyone, so never use it to hide sensitive or private URLs — it only asks crawlers not to visit, it doesn't restrict access.
- Blocking a page in robots.txt also blocks search engines from seeing a noindex tag on that page, since they can't crawl it to read the tag.
- Malicious crawlers can simply ignore robots.txt — it relies on voluntary compliance.
Frequently Asked Questions
- Can robots.txt remove a page from Google entirely?
- Not reliably — disallowing a page stops crawling, but the URL can still be indexed from external links. Use a noindex meta tag (which requires the page to be crawlable) to actually prevent indexing.
- Is robots.txt required?
- No — if it's missing, crawlers assume everything is allowed. Most sites still include one to manage crawl budget and point to their sitemap.