Technical
Notice

robots.txt file is missing

This error appears when your website doesn't have a robots.txt file and the address where it should be returns a page not found error (404). Here's what it means and how to create it in a few minutes.

What this error means

The robots.txt is the text file at yourdomain.com/robots.txt that crawlers check before going through your website. It's used to tell them which sections they shouldn't crawl and where your sitemap is, the list of pages on your website. When requesting that address, Ruk Audit received a 404, the response a server gives when a page doesn't exist. For Google, that means there are no restrictions, so it crawls your entire website. Google's own documentation considers it acceptable not to have a robots.txt. If instead of a 404 the address returns a server error, things change, because Google stops crawling your website for the first 12 hours. That error has its own page. This error usually shows up when the website was custom-built and nobody created the file, or when something on the server prevents the robots.txt generated by the content management system from being seen.

Why creating the robots.txt file matters

If you weren't planning to restrict anything, this warning doesn't call for urgent action.

Impact on SEO rankings

Without rules, Google spends time on pages that bring you nothing in the results, such as the cart, internal searches or the admin area. On a small website it's barely noticeable. In a store with thousands of filters and combinations, that time is taken away from the pages you do want to rank.

Crawling and indexing obstacles

Without robots.txt there's also no line indicating where your sitemap is. Search engines you haven't submitted the sitemap to have to find your pages by following links, and pages with few links take longer to be crawled.

Negative signals for site quality

On a website built with WordPress, Shopify or Wix, a missing file is a sign that something on the server is blocking that address, such as a redirect rule or a security plugin, and that same thing may be affecting other pages.

How to fix it step by step

If this error showed up in your audit, here are the steps to leave it behind.

Step 1

Check the address in your browser

Type yourdomain.com/robots.txt into your browser. If you see a not found page, the file doesn't exist and you can move on to the next step. If it takes you to the homepage or the login, a redirect is hiding the file. Ask your hosting provider or whoever maintains the website to make the robots.txt show at that address without redirecting. If you see the file's text but the audit still returns a 404, a security plugin, a firewall or the CDN is probably blocking it for crawlers. Check the plugin from WordPress first, and if that's not it, ask your hosting provider to look at the firewall or the CDN.

Step 2

Create the file depending on where your website is

WordPress creates an automatic robots.txt even if there's no file on the server, so if you get this error, a redirect or a block is most likely hiding it, as in the previous step. To customize it, the easiest way is your SEO plugin, such as Yoast or Rank Math. In Yoast it's under Yoast SEO > Tools > File editor, and in Rank Math under General Settings > Edit robots.txt, with advanced mode turned on. Shopify and Wix create one by default on every website, so if you get this error on one of them, let whoever manages your website or the platform's support know. To change it, in Shopify it's edited from the theme code and in Wix from the robots.txt editor in its SEO section. On a custom-built website, create a file named exactly robots.txt with a plain text editor, such as Notepad. Upload it to the root of the domain, your website's main folder, from your hosting's file manager. Each subdomain needs its own, because the one for yourdomain.com doesn't work for blog.yourdomain.com.

Step 3

Write minimal content

To start with, a few lines are enough. The first one indicates that the rules apply to all crawlers, each Disallow blocks a section and the last one declares the sitemap with its full address. One example is these four lines, each on its own line and with a blank line before the sitemap one, "User-agent: *", "Disallow: /cart/", "Disallow: /search/" and "Sitemap: https://www.yourdomain.com/sitemap.xml". Replace the paths with those of the sections you don't want crawled and the URL with your sitemap's. If you write "Disallow: /" on its own, you block the entire website.

Step 4

Check the file and audit again

Open yourdomain.com/robots.txt again and confirm that the text shows exactly as you wrote it. Then check the robots.txt report in Google Search Console, which is under Settings, to see whether Google has already fetched it. Finally, run Ruk Audit again to confirm that the error has disappeared.

Does your site have this problem?

Audit your website for free and discover if this and other SEO errors are affecting your ranking.

Get a free audit. Over 3022 reports delivered.
Hey, wait a moment
To analyze a website, you first need to create an account in the application. Click the button below to go to the registration screen.
Create account