How to create robots.txt file for your WordPress site

What is robots.txt ?

A robots.txt file is a file that contains specific instructions that inform the bot what it can and can’t look at. Since the bot is set to crawl everything by default anyway, a robots.txt file is essentially the means through which you can apply a set of specific crawling restrictions. So this robots.txt file plays a major role in search engine ranking and helps to block search engine bots to index and crawl important part of our blog.

why is it important ?

Since the bot is set to crawl everything by default anyway, a robots.txt file is essentially the means through which you can apply a set of specific crawling restrictions. Robots.txt file helps search engine robots to direct which part to crawl and which part to avoid. When Search bot or spider of Search Engine comes to your site and wants to index your site, they follow Robots.txt file first. Search bot or spider follows this files direction for index or no index any page of your website.

How to make  robots.txt file in WordPress?

Robots is generally a text file with extension of (.txt). So if you have FTP access and like to write use your hand just create a txt file name robots.txt. The follow the code according to you site preference which you would like to allow or disallow.

For example:

Allow Everything

User-agent: *
Disallow:

Block WP-Admin and WP-Includes

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

The standard approach

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-login.php
Disallow: /wp-register.php
Allow: /wp-content/uploads/

The another good and easy options to use plugin. This WP ROBOTS Plugin can help to do this all in admin settings. After successful installation you can see the settings in Dashboard -> Settings -> Reading

So here you can edit/changes your robot text. To view your robot text file just write on url: yourdomain/robots.txt

Some example of good sites:

http://www.google.com/robots.txt
https://www.facebook.com/robots.txt
http://www.ebay.com/robots.txt

Finally i would to say, it is recommended to use robots.txt file for SEO optimize your site. Even i am not a SEO person but in terms of essentials what I’ve included above is sure to improve the performance of your site.

That’s all here. If you still have any issue please put a comment below, we will come back to you shortly. Thanks.

This post has already been read 7083 times!

Mehedi Hasan

Cool WordPress Developer having much agile experience to develop any kind of WordPress sites & plugins. Also good in troubleshooting, fixing & making any kind of tweaks for WP site.

More Posts