What is Robots.txt File?
If you have an blog then you can create/post on your blog or a site, a robots.txt file is auto generated by Blogger. The purpose of this Robots.txt file is inform the incoming robots to your blog (spiders, crawlers and ETC sent by search engines like Google, Bing, Yahoo and ETC). This is a simple text in which the website or blog owner use to write the coding for the search engine crawlers. It means that with the help of these Robots.txt file your blog is easily found by search engines. You would like certain pages of your blog or site to be indexed by search engines, while others you might prefer not to be indexed, like a About Us Page, Contact Us Page or any other.
Where is Robots.txt File Located?
Well, gaining some knowledge about this file now, you can easily see your Robots.txt file on your browser by adding /robots.txt after your blog URL like http://nextbloggertricks.blogspot.com/robots.txt or if you have a websites type this link in your browser address bar www.nextbloggertricks.com/robots.txt but before view your robots.txt file first you can enable this file from your blogger.
How To Enable Custom Robots.txt File In Blogger Blogs?
- Go to Blogger Dashboard>>Settings>>Search Prefences
- Look For Custom Robots.txt Area in the bottom.
- Now you can see a check box tick mark "Yes" and a box will appear where you have to insert your Robots.txt file.
- Copy the following Code and paste in that box.
User-agent: Mediapartners-GoogleDisallow:User-agent:*Disallow:/searchAllow:/Sitemap:http://nextbloggertricks.com/feeds/posts/default?orderby=UPDATED
- Note : The first red color line is for Google Adsense Approved accounts. If you are using Google Adsense in your blog then you can not changes otherwise remove it.
- Replace last blue color line with include my website link and you can change with your blog or site link.
- Click "Save Changes".
Explanations:-
After adding the Robots.txt file now you must be wanted to gain more knowledge its parts and how it work on your blog. So, I have listed every line meaning with detailed below.
- User-agent: Mediapartners-Google : This is first command of this Robots.txt file which is for those blogs and websites are using Google Adsense if you are not using Google Adsense then remove this line. otherwise not.
- Disallow : This line work for the search engine's crawler to do not index or crawl label and navigation pages.
- User-agent:* : This line work for calling the robots and * is for all the search engine's robots like Google, Bing, Yahoo ETC.
- Disallow:/search : This line work tells for the search engine's crawler not to crawl the search results pages.
- Allow:/ : This is allows your index the homepage or your blogger blog and websites.
- Sitemap : This is last line tells the search engine's crawlers to index the every new posts of your blog or your last time published post.
How to see Your Blog's Robots.txt File?
As explained above, each lines meaning and each commands explanation now let's time to check your blog robots.txt file it work? and see?, Simply type your blog url in address bar of your browser and add /robots.txt at the end of your blog URL. Just like this example.
www.nextbloggertricks.com/robots.txt
Once you view your robots.txt file, you will see this code which are using in your custom Robots.txt file. See the below image.
Final Words
So friends, it was our robots.txt file tutorial which is going to help you definitely. I will back with great and useful posts and tutorials. So be in touch with also if you're facing problem or getting error during this tutorial so leave comments for solutions and keep visiting us and don't forget to like our Facebook Page.
Happy Blogging!
No Comment
Comment Policy
We’re eager to see your comment. However, Please Keep in mind that all comments are moderated manually by our human reviewers according to our comment policy, and all the links are nofollow. Using Keywords in the name field area is forbidden. Let’s enjoy a personal and evocative conversation.