SEO Robots.txt

Robots.txt is a plain text file that sits at the root of your website and tells search engine crawlers which pages or sections they are allowed to visit and which to skip. It is one of the first files a crawler checks when it arrives at your site. Setting it up correctly protects your crawl budget and keeps unwanted pages out of search results.

What a Robots.txt File Looks Like

The file is publicly accessible at yourwebsite.com/robots.txt. Here is a simple example:

Diagram: Basic Robots.txt File

User-agent: *
Disallow: /wp-admin/
Disallow: /thank-you-page/
Disallow: /staging/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.yourwebsite.com/sitemap.xml

Line-by-line explanation:
User-agent: *          --> These rules apply to ALL crawlers
Disallow: /wp-admin/   --> Block crawlers from WordPress admin area
Disallow: /thank-you/  --> Block this page from being indexed
Allow: /admin-ajax.php --> This specific file CAN be crawled
Sitemap: ...           --> Tell crawlers where your sitemap is

Understanding Robots.txt Syntax

The file uses a simple set of commands:

• User-agent: Specifies which bot the rule applies to. Use * for all bots, or a specific bot name like Googlebot.
• Disallow: The path the bot should NOT visit.
• Allow: Overrides a Disallow rule for a specific file or folder within a blocked area.
• Sitemap: Points crawlers to your XML sitemap location.

What to Block with Robots.txt

Not all pages on your website need to be crawled. Blocking low-value pages saves your crawl budget for important pages.

Pages Worth Blocking

• Admin and login pages (/wp-admin/, /wp-login.php)
• Thank-you and confirmation pages that have no SEO value
• Internal search result pages (/?s=)
• Staging or test environments
• Duplicate content sections (some pagination pages)
• User account pages (/account/, /cart/)

Pages You Must NOT Block

NEVER block these in robots.txt:
  - Your CSS and JavaScript files (blocking these breaks rendering)
  - Important content pages you want ranked
  - Your XML sitemap (always keep it accessible)
  - Your homepage
  - Core product, service, or blog pages

Robots.txt vs Noindex Tag: Key Difference

This is one of the most common points of confusion in SEO:

Diagram: Robots.txt vs Noindex

ROBOTS.TXT DISALLOW:
  - Tells the bot NOT to crawl the page.
  - Bot never visits the page.
  - BUT: the page can still appear in search results 
    if other sites link to it (Google knows it exists 
    from external links, even without visiting).

NOINDEX META TAG (on the page itself):
  - Bot DOES crawl the page.
  - Bot reads the noindex tag and drops it from the index.
  - The page definitely will not appear in search results.

USE robots.txt to: Block crawling of private or 
                   irrelevant pages.
USE noindex to:    Definitely remove a page from 
                   search results.

Common Robots.txt Mistakes

Mistake 1: Blocking Your Entire Website

A robots.txt with Disallow: / blocks all crawlers from your entire website. This completely removes your site from Google. This mistake happens accidentally when copying templates. Always check your robots.txt after any major website migration.

Mistake 2: Blocking CSS and JavaScript

If your robots.txt blocks the CSS and JS files, Google cannot render your pages properly. It sees your pages as broken or empty, which harms rankings significantly. Always allow CSS and JS directories.

Mistake 3: Thinking Robots.txt Keeps Pages Secret

Robots.txt is publicly visible. Anyone can visit yoursite.com/robots.txt and see exactly what you are hiding. Do not use it to "hide" sensitive pages. Use proper authentication (passwords) for truly private content.

How to Find and Edit Your Robots.txt

In WordPress, your robots.txt is managed through your SEO plugin. In Yoast SEO: SEO → Tools → File Editor. In Rank Math: General Settings → Edit robots.txt. Always test your robots.txt changes using Google Search Console's robots.txt Tester before saving.

Key Takeaway

Robots.txt controls which pages search engine bots can and cannot crawl. Block low-value pages like admin areas and thank-you pages to save crawl budget for your important content. Never block CSS, JavaScript, or pages you want ranked. Understand that blocking a page in robots.txt does not remove it from search results — use the noindex tag for that purpose.