WordPress robots.txt Setup: How the Virtual File Works and What to Avoid

This article is a localized edition of the Japanese original. Read the Japanese original

robots.txt is a file that tells search engine crawlers which URLs they may crawl. WordPress outputs its contents automatically even when there is no robots.txt file in the site root (a virtual robots.txt). This article explains what WordPress outputs, how to change it, and what to watch out for when you write it.

What robots.txt can and can’t do

robots.txt controls crawling. It is not a way to keep pages out of search results (indexing). Google's documentation also explains that a page blocked by robots.txt can still appear in search results as a bare URL without a description if other sites link to it.

For pages you don't want in search results, use noindex instead of robots.txt. A crawler only sees noindex after loading the page, so if you also block that page in robots.txt, the noindex is never read.

Check your current robots.txt

# Check the status code and headers
curl -sI https://example.com/robots.txt

# Check the contents
curl -s https://example.com/robots.txt

robots.txt must sit directly under the host name (https://example.com/robots.txt). If WordPress is installed in a subdirectory, such as https://example.com/blog/, WordPress's virtual robots.txt doesn't answer at the root, so put a real file in the root.

Watch the status code too. Google treats a robots.txt that returns 404 or another 4xx as no restrictions, and temporarily stops crawling when it returns 5xx. If a PHP error makes the virtual robots.txt return 500, crawling of the whole site is affected.

The virtual robots.txt WordPress outputs

If there is no real file in the site root, WordPress's do_robots() function outputs something like this.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

• Crawling of /wp-admin/ is disallowed, and only admin-ajax.php, which front-end features also use, is allowed
• The Sitemap: line is added when the core XML sitemap, introduced in WordPress 5.5, is enabled

If you check "Discourage search engines from indexing this site" under "Search engine visibility" in "Settings" → "Reading" (設定 → 表示設定 in the Japanese UI), WordPress 5.3 and later no longer output Disallow: / in robots.txt. Instead, a noindex meta tag is output on each page. The core sitemap is disabled in that case, so the Sitemap: line is not output either.

# 1 = allow search engines, 0 = ask search engines not to index
wp option get blog_public

Make sure it hasn't been set to 0 by accident on your production site.

Two ways to change it

If you use both, only the real file is used. If you edit robots.txt in an SEO plugin's settings, also decide on a single place to manage it.

Add lines with the robots_txt filter

Put this in a child theme's functions.php or in a custom plugin. This example inserts lines that stop crawling of on-site search result pages right after Allow: /wp-admin/admin-ajax.php.

<?php
add_filter( 'robots_txt', function ( $output, $public ) {
	if ( ! $public ) {
		return $output;
	}

	$allow = 'Allow: ' . wp_parse_url( admin_url( 'admin-ajax.php' ), PHP_URL_PATH ) . "\n";
	$extra = "Disallow: /*?s=\n" . "Disallow: /search/\n";

	return str_replace( $allow, $allow . $extra, $output );
}, 10, 2 );

Core adds its Sitemap: line first, at priority 0. Appending text to the end would place it after the Sitemap: line, so this example inserts the lines right after the Allow line. When $public is false (the site is set not to be indexed), it does nothing.

Use a real file

If you use a real file, base it on WordPress's default contents.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /*?s=
Disallow: /search/

Sitemap: https://example.com/wp-sitemap.xml

In Sitemap:, write the URL of the sitemap you actually use. For the WordPress core sitemap, it is /wp-sitemap.xml. If an SEO plugin outputs the sitemap, the URL differs (for example, Yoast SEO uses /sitemap_index.xml), so open it in a browser and write the URL that works.

Things to watch when writing it

Don’t block CSS and JavaScript directories

You may see examples that Disallow /wp-includes/, /wp-content/plugins/, and /wp-content/themes/, but these contain the CSS and JavaScript used to render your pages. Google advises not to block such files if the page is harder to understand without them. Don't block /wp-content/uploads/ either, since it holds your images.

Handling search result pages

On-site search results (/?s=keyword or /search/keyword/) are usually not useful as search engine results even if crawled, so crawling is sometimes stopped as in the example above. Google's robots.txt supports * (any string) and $ (end of URL), so writing Disallow: /*?s= also matches URLs like /page/2/?s=….

However, if you remove search result pages from search results with noindex, blocking them in robots.txt means the noindex is no longer read. To remove search result pages that are already indexed, first wait for noindex to take them out and block them in robots.txt only afterward, or use just one of the two.

Block specific crawlers

For other crawlers, write a separate User-agent group. As of October 2026, the following names appear in each company's official documentation.

# SEO tool crawlers
User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

# OpenAI's crawler
User-agent: GPTBot
Disallow: /

# Token that controls use in Google's generative AI (does not affect Google Search crawling)
User-agent: Google-Extended
Disallow: /

Whether to obey robots.txt is up to the crawler, so it has no effect on crawlers that ignore it. To stop the access itself, block it at the web server or firewall.

Directives Google doesn’t support

Google interprets four fields: user-agent, allow, disallow, and sitemap. crawl-delay is ignored. The file size limit is 500 KiB, and anything beyond it is not read.

Check it in Search Console

The old "robots.txt Tester" was retired after Google announced in November 2023 that the new "robots.txt report" would replace it. As of October 2026, use these instead.

• robots.txt report: Open it from "Settings" in Search Console. It shows when Google last fetched robots.txt, whether the fetch succeeded, and any errors and warnings found while parsing. You can also request a recrawl after fixing it
• URL Inspection: Inspect an individual URL to see whether robots.txt blocks it

The robots.txt report is available for Domain properties and for URL-prefix properties without a path (for example, https://example.com/). Google usually caches robots.txt for up to 24 hours, so changes may not take effect right away.

Summary

• robots.txt controls crawling; it doesn't prevent indexing. Use noindex to remove pages from search results, and don't block those pages in robots.txt
• Without a real file, WordPress outputs a virtual robots.txt. When the core sitemap is enabled, it includes a Sitemap: line
• Make changes with either the robots_txt filter or a real file, not both
• Don't block /wp-includes/, /wp-content/plugins/, or /wp-content/themes/
• In Sitemap:, write the URL of the sitemap you actually use
• Check with the robots.txt report and URL Inspection in Search Console

Related articles:

• What to do when your WordPress sitemap doesn't show
• Find and fix noindex mistakes
• Set WordPress search result pages to noindex
• Basic Google Search Console setup for WordPress

Questions about WordPress? The publisher of this site is based in Japan and replies in Japanese. Contact (Japanese)