robots.txt file is missing
This error appears when your website doesn't have a robots.txt file and the address where it should be returns a page not found error (404). Here's what it means and how to create it in a few minutes.
What this error means
Why creating the robots.txt file matters
Impact on SEO rankings
Without rules, Google spends time on pages that bring you nothing in the results, such as the cart, internal searches or the admin area. On a small website it's barely noticeable. In a store with thousands of filters and combinations, that time is taken away from the pages you do want to rank.
Crawling and indexing obstacles
Without robots.txt there's also no line indicating where your sitemap is. Search engines you haven't submitted the sitemap to have to find your pages by following links, and pages with few links take longer to be crawled.
Negative signals for site quality
On a website built with WordPress, Shopify or Wix, a missing file is a sign that something on the server is blocking that address, such as a redirect rule or a security plugin, and that same thing may be affecting other pages.
How to fix it step by step
If this error showed up in your audit, here are the steps to leave it behind.
Check the address in your browser
Type yourdomain.com/robots.txt into your browser. If you see a not found page, the file doesn't exist and you can move on to the next step. If it takes you to the homepage or the login, a redirect is hiding the file. Ask your hosting provider or whoever maintains the website to make the robots.txt show at that address without redirecting. If you see the file's text but the audit still returns a 404, a security plugin, a firewall or the CDN is probably blocking it for crawlers. Check the plugin from WordPress first, and if that's not it, ask your hosting provider to look at the firewall or the CDN.
Create the file depending on where your website is
WordPress creates an automatic robots.txt even if there's no file on the server, so if you get this error, a redirect or a block is most likely hiding it, as in the previous step. To customize it, the easiest way is your SEO plugin, such as Yoast or Rank Math. In Yoast it's under Yoast SEO > Tools > File editor, and in Rank Math under General Settings > Edit robots.txt, with advanced mode turned on. Shopify and Wix create one by default on every website, so if you get this error on one of them, let whoever manages your website or the platform's support know. To change it, in Shopify it's edited from the theme code and in Wix from the robots.txt editor in its SEO section. On a custom-built website, create a file named exactly robots.txt with a plain text editor, such as Notepad. Upload it to the root of the domain, your website's main folder, from your hosting's file manager. Each subdomain needs its own, because the one for yourdomain.com doesn't work for blog.yourdomain.com.
Write minimal content
To start with, a few lines are enough. The first one indicates that the rules apply to all crawlers, each Disallow blocks a section and the last one declares the sitemap with its full address. One example is these four lines, each on its own line and with a blank line before the sitemap one, "User-agent: *", "Disallow: /cart/", "Disallow: /search/" and "Sitemap: https://www.yourdomain.com/sitemap.xml". Replace the paths with those of the sections you don't want crawled and the URL with your sitemap's. If you write "Disallow: /" on its own, you block the entire website.
Check the file and audit again
Open yourdomain.com/robots.txt again and confirm that the text shows exactly as you wrote it. Then check the robots.txt report in Google Search Console, which is under Settings, to see whether Google has already fetched it. Finally, run Ruk Audit again to confirm that the error has disappeared.
Does your site have this problem?
Audit your website for free and discover if this and other SEO errors are affecting your ranking.