CaelLab Team
2025-12-07

How should robots.txt be written?

robots.txt is a plain text file in a website’s root directory that controls search engine crawler access. Below are its core syntax, common cases and cautions in detail:

1. Basic syntax and structure

  1. File location and naming​
    • It must be in the site’s root directory (e.g. https://www.example.com/robots.txt), and the filename must be strictly lowercase.
    • If the file does not exist, crawlers may access the whole site by default.
  2. Core directives​
    • User-agent: specifies the target crawler (e.g. Googlebot, Baiduspider or * for all crawlers).
    • Disallow: paths that must not be accessed (e.g. /admin/ blocks crawling the admin directory).
    • Allow: paths that may be accessed (used to override Disallow rules).
    • Sitemap: optional; declares the sitemap path (must be a full URL).
  3. Wildcards and matching rules​
    • *: matches any characters (e.g. /private*/ blocks all directories starting with private).
    • $: matches the end of a URL (e.g. Disallow: /*.jpg$ blocks all .jpg images).
    • Paths are case-sensitive, and a trailing slash changes the scope (e.g. /images/ blocks the whole directory, while /images blocks only the file of that name).

2. Common case examples

  1. Block all crawlers​ User-agent: * Disallow: /Effect: no content on the site is indexed.
  2. Allow a specific crawler​ User-agent: BadBot Disallow: / User-agent: * Allow: /Effect: only BadBot may access the whole site.
  3. Restrict directories and file types​ User-agent: * Disallow: /cgi-bin/ Disallow: /images/*.gif Allow: /public/Effect: the cgi-bin directory and all GIF images are blocked, the public directory is allowed.
  4. Dynamic pages and parameter filtering​ User-agent: * Disallow: /*?*Effect: URLs with query parameters (e.g. index.php?id=1) are not crawled.

3. Advanced usage and cautions

  1. Per-crawler rules​ User-agent: Googlebot Disallow: /temp/ Allow: /public/ User-agent: Baiduspider Disallow: /admin/Each crawler can have its own rules.
  2. Supplementary Meta tags​ Control the crawling behaviour of a single page further in HTML with <meta name="robots" content="noindex, nofollow">.
  3. Common mistakes and tuning​
    • File encoding: ANSI or UTF-8 is recommended (to avoid garbled characters on the first line).
    • Path conventions: avoid stray spaces at the end of a path or confusion over slashes.
    • Testing tools: check the syntax with the Robots.txt tester in Google Search Console.

4. Complete example template

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /*.php$
Allow: /public/
Sitemap: https://www.example.com/sitemap.xml

5. Cautions

  • Not mandatory: robots.txt is an advisory protocol, and some crawlers may ignore its rules.
  • Privacy: sensitive information still needs server-side access control rather than reliance on robots.txt.
  • Updates take time: after a change, wait for search engines to refresh their cache (usually hours to days).

With a sensible robots.txt you can manage how far search engines crawl, and improve site performance and privacy. Combining it with a sitemap (Sitemap) and Meta tags gives finer control.

Back to updates