Category: SEO and Google Search

SEO: robots.txt blocks the page despite being present in sitemap.xml

Read this guide in Polish

The site map contains an address, but the robot cannot enter it due to Disallow rule. Sitemap is a clue to discover URLs, and robots.txt can independently prohibit their scanning.

How to determine the source of the error

Open production /robots.txt and check the rules for a specific path.

In Search Console, perform a URL inspection and compare the lock indication with the actual robots file.

What to do step by step

1. Remove the Unintended Disallow rule for public sites, retaining consciously protected areas.

2 . After changing the file, check if the URL returns 200, has a valid canonical and does not contain noindex.

How to check the score

The robot can download a public website, and the report after the crash does not indicate a robot lock.

What to do

Do not use robots.txt to hide private data from unauthorized users.