Robots.txt returns 503 and Google limits crawl
The site has the correct pages, but the server sometimes does not share a robot.txt file.
How to identify the source of the problem
Check the status and response time /robots.txt during load and proxy rules.
Check the HTML server, HTTP status, canonical and robots for the final production address.
Repair steps
Provide a stable, fast response to robots.txt independent of the base and heavy functions.
How to Test Solution
The robot can constantly download the configuration, and important pages remain available.
Security and restrictions
The indexation and position in the search engine are not guaranteed. Do not create secondary copies of the page or artificial test results.