A robots.txt file is a file that contains specific instructions that inform the bot what it can and can’t look at. Since the bot is set to crawl everything by default anyway, a robots.txt file is essentially the means through which you can apply a set of specific crawling restrictions. So this robots.txt file plays a major role in search engine ranking and helps to block search engine bots to index and crawl important part of our blog.
Since the bot is set to crawl everything by default anyway, a robots.txt file is essentially the means through which you can apply a set of specific crawling restrictions. Robots.txt file helps search engine robots to direct which part to crawl and which part to avoid. When Search bot or spider of Search Engine comes to your site and wants to index your site, they follow Robots.txt file first. Search bot or spider follows this files direction for index or no index any page of your website.
Robots is generally a text file with extension of (.txt). So if you have FTP access and like to write use your hand just create a txt file name robots.txt. The follow the code according to you site preference which you would like to allow or disallow.
For example:
User-agent: * Disallow:
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/
User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/cache/ Disallow: /wp-content/themes/ Disallow: /wp-login.php Disallow: /wp-register.php Allow: /wp-content/uploads/
The another good and easy options to use plugin. This WP ROBOTS Plugin can help to do this all in admin settings. After successful installation you can see the settings in Dashboard -> Settings -> Reading
So here you can edit/changes your robot text. To view your robot text file just write on url: yourdomain/robots.txt
http://www.google.com/robots.txt
http://www.ebay.com/robots.txt
Finally i would to say, it is recommended to use robots.txt file for SEO optimize your site. Even i am not a SEO person but in terms of essentials what I’ve included above is sure to improve the performance of your site.
That’s all here. If you still have any issue please put a comment below, we will come back to you shortly. Thanks.
This post has already been read 7088 times!
What is Lead Capture: In general sense lead capture designates a process or way of…
Introduction Unlimited Modal POPUPS on MouseClick: Everyone knows about popup during site visits and most…
Brief about Pregnancy Calculator : This WordPress Pregnancy Calculator plugin allows users to display an…
Introduction About this Service Plugin: This Visual Composer Service Plugin works as an addon to…
Introduction of Dynamic Query String : This plugin allows user to set dynamic query string…
There is an occasional bug for iPhone and iPad even they are designed as well and…