If you want to improve the SEO of your WordPress website, then robots.txt is an indispensable factor to discuss. So, what is robots.txt, why is it so important, and how you can optimize your robots.txt file for WordPress SEO to make your blog more search engine friendly or search engine friendly. SEO friendly? If you don’t know about the answer to this question, then in this article I will provide an article on all the answers in simple and easy language.
What is Robots.txt?
Robots.txt is a simple text file that controls how search engines behave with your website. It directs search engines about where they are allowed to index and where they cannot index. When a search engine robot crawls your website or blog, the search engine will first follow the orders from robots.txt.
By default, search engines are designed to index as much information as they can and robots.txt can limit all of that. You may be wondering why you should limit search engines to get more information from your site, otherwise the more chances your website will have a good ranking place in search results, here’s the answer.
In total, there are three reasons to use this robots.txt file:
- First: Opening all of your data to a search engine makes your site vulnerable and easy to hack.
- Second: Allowing search engines to access your site can unrestrictedly consume a large amount of bandwidth and as a result, make your website slow down.
- Third: The more pages on your website, the more pages search engines will crawl. This will take a lot of time for the search engines to crawl your site. Limiting crawl to only the most important parts will determine how quickly the search engines pick up on changes to your site – and thus how quickly you will rank
How to Optimize WordPress SEO with Robots.txt?
After reading all the information above, you may have a clear understanding of the definition and function of robots.txt and why you should use it for your WordPress website. Excessive constraints on search engine behavior may have a negative impact on SEO while too much freedom for them can put your website in jeopardy.
Therefore, you need to determine a solution that can give consideration to using the robots.txt file for website security and SEO. Following is an example of creating your robots.txt file to limit search engine behavior and improve WordPress SEO.
First, login to your cPanel web hosting. The robots.txt file is in the root folder of your WordPress site. However, if you don’t find the robots.txt file, feel free to create a new file and name it as robots.txt, or you can read the article below for how to create a new file in cPanel.
Then, you need to right click on your robots.txt file and click edit or edit code. Then, select utf-8 as the character encoding and click edit.
What is the ideal robots.txt file?
The format for the robots.txt file is very simple. The first line usually gives the user agent name. User agent or (User-Agent) is the name of the search bot with which you want to communicate. For example, Googlebot or Bingbot. You can use an asterisk * to instruct all bots.
The next line follows with the Allow or Disallow instructions for search engines, so they know which parts you want indexed, and which parts they don’t want indexed.
Here’s an example:
User-Agent: * Allow: /?display=wide Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /readme.html Disallow: /refer/ Sitemap: https://www.verandalounge.co.uk/post-sitemap.xml Sitemap: https://www.verandalounge.co.uk/page-sitemap.xml Sitemap: https://www.verandalounge.co.uk/deals-sitemap.xml
Note: If you are using a plugin like Yoast or All in One SEO, you may not need to add a sitemap section, because they try to do it automatically. If that fails, you can add it manually as in the example above.
What should I Disallow or Noindex?
Within the guidelines Google webmaster, they advise webmasters not to use their robots.txt file to hide low-quality content. So, using your robots.txt file to stop Google from indexing your categories, dates, and other archive pages may not be a wise choice.
Remember, the goal of robots.txt is to instruct bots what to do with the content they crawl on your site. This doesn’t stop them from crawling your site.
Also, you don’t need to add a WordPress login page, admin directory, or registration page to robots.txt because the login and registration pages have a noindex tag which is automatically added by WordPress.
However, I recommend that you disallow the readme.html file in your robots.txt file. This readme file can be used by someone trying to find out which version of WordPress you are using. Because anyone can easily access files by just browsing there. Dana also puts a “Disallow” tag to block malicious attacks.
Apart from that, you can also disallow (Disallow) your WordPress plugins directory to increase the security of your site against anyone looking for a vulnerable plugin to exploit.
How do I submit a WordPress robots.txt file to Google?
After you have updated or created your robots.txt file, you can submit it to Google using Google Search Console.
However, you should first test using Google’s robots.txt test tool here.
If you don’t see the version you created here, you will need to re-upload the robots.txt file you created to the root folder of your WordPress site.