Custom Robots.txt in Blogger Without Blocking Your Posts

Every few weeks a reader emails me the same panicked message: their Blogger traffic dropped by half overnight, and nothing changed except "a small SEO tweak." Nine times out of ten that tweak is a custom robots.txt in Blogger, copied from a tutorial, that quietly told Google to stop crawling their posts. It takes about thirty seconds to break and, usually, a few weeks to recover.

The setting itself is useful when you understand it. Blogger gives every blog a sensible default file, and most blogs never need to touch it. This guide shows you what the default does, the very few changes worth making, and how to test a custom file before Google acts on it.

What Robots.txt Actually Does on a Blogger Blog

Robots.txt is a plain text file at the root of your blog, for example yourblog.blogspot.com/robots.txt. Search engine crawlers read it before anything else and use it to decide which addresses they may request.

Two things it does not do, and both trip people up:

It does not remove pages from Google. A blocked page can still appear in results if other pages link to it, usually with no description underneath.
It does not hide anything. Anyone can open your robots.txt in a browser and read it.

If your goal is to keep a page out of search results, you want a noindex instruction, not a robots.txt block. We cover that in detail in our guide to Blogger label pages and noindex.

Why Bloggers Break It So Often

The custom robots.txt box sits right next to settings that really do help, so it feels like part of the same checklist. Blogger shows no warning when you save a file that blocks your whole site, and Search Console reports the damage days later, after Google has already recrawled. By then the cause is hard to connect to the drop.

In the cases I have helped with, the pattern is nearly always the same. Someone pastes a twenty line file from an old tutorial, the file includes a rule written for WordPress or for an older Blogger address format, and posts start showing "Blocked by robots.txt" in the Pages report. The blog looks perfectly normal to visitors, which is why it can go unnoticed for weeks.

The Default Blogger Robots.txt, Line by Line

Open your own blog's robots.txt now. Unless you changed it, you will see something close to this:

Line	What it means
User-agent: Mediapartners-Google	The rules below apply to the AdSense crawler
Disallow: (empty)	The AdSense crawler may read everything, so it can match ads to content
User-agent: *	The rules below apply to every other crawler
Disallow: /search	Do not crawl search results and label listing pages
Allow: /	Everything else is open
Sitemap: https://yourblog.blogspot.com/sitemap.xml	Where to find the list of your posts

That is a good file. It blocks the endless /search and /search/label/ listings, which are thin duplicate pages, and leaves your posts, static pages and images open. If you are on a custom domain, the sitemap line uses your domain automatically.

When a Custom Robots.txt in Blogger Is Worth It

I enable the custom option in only three situations:

Adding a second sitemap. Blogger also generates sitemap-pages.xml for static pages like About and Contact. Adding a line for it is harmless and slightly helps discovery.
Blocking a specific private path. For example, a set of old test posts you cannot delete yet. Even then, noindex is usually the better tool.
Restoring a broken file. If someone before you pasted something risky, turning custom on and pasting the default back is the fix.

To edit it, go to Settings, scroll to Crawlers and indexing, switch on Enable custom robots.txt, then click Custom robots.txt and paste your file. Save. The change goes live immediately.

The Tutorials I Would Ignore

Many popular Blogger SEO posts share a "best robots.txt" that does more harm than good. Three lines show up repeatedly.

Disallow: /20*

Blogger post addresses begin with the year, such as /2026/10/my-post.html. This line blocks every one of them. Tutorials pair it with Allow: /*.html to reopen the posts, which works for Google because the longer, more specific rule wins. Other crawlers may not resolve the conflict the same way. You gain nothing and add risk.

Disallow: /p/

This blocks all your static pages, including About, Contact and Privacy. Those pages help readers and search engines understand who runs the blog. Blocking them is a step backwards.

Disallow: /*?m=1

The mobile parameter worries people because it looks like duplicate content. Blogger already adds a canonical tag pointing to the main address, so Google consolidates the two. Blocking the parameter can prevent Google from seeing that canonical at all.

My honest view: if a tutorial's robots.txt is longer than eight lines, it is solving a problem you probably do not have.

A Safe Custom File You Can Use

If you want a custom version, keep it to the default plus one extra sitemap line. Paste these lines, each on its own line, replacing the address with yours:

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://yourblog.blogspot.com/sitemap.xml
Sitemap: https://yourblog.blogspot.com/sitemap-pages.xml

Leave a blank line between the AdSense block and the general block. Spelling matters: "Dissallow" or a missing colon is ignored silently, and you will not get an error from Blogger.

Testing Before Google Reacts

Do these checks within five minutes of saving:

Open yourblog.blogspot.com/robots.txt in a private browser window and confirm it shows your new text.
In Google Search Console, open Settings, then the robots.txt report. It shows the version Google last fetched and flags lines it could not parse. You can request a recrawl from there.
Use URL Inspection on one recent post and one static page. Both should say crawling is allowed.

If anything says blocked, switch the custom option off. Blogger instantly returns to its default file. Then check the rest of your setup with our list of basic SEO settings most new bloggers skip.

FAQ

Is custom robots.txt the same as custom robots header tags?

No. Header tags in the same settings panel control noindex and similar instructions per page type. Robots.txt controls crawling. Edit one at a time so you know which change caused what.

How long does Google take to notice a new robots.txt?

Google usually refetches robots.txt within about a day. You can speed that up from the robots.txt report in Search Console after you fix a mistake.

Will blocking label pages hurt my rankings?

The default already blocks them through /search, and that is fine for most blogs. Your posts carry the ranking value, not the label listings.

The Practical Takeaway

For most Blogger blogs, the best custom robots.txt is no custom robots.txt. If you do enable it, keep the default lines, add the pages sitemap, test in Search Console the same day and switch it off at the first sign of a blocked post. More guides like this live in our SEO for bloggers section.