Search & Find

Sponsor Post

Best Telegram Groups List Collection - Updated

Sunday 28 August 2016

How to Enable Custom Robots file in Blogger blog with Pictures

Hello friends welcome to the Techtspot Blog, today I will show you how to enable a custom Robots.txt file and robots header tags in the Blogger blog with pictures.
Enable Custom Robots file in Blogger

What is the Robots.txt file?

Robots.txt file tells the crawler which pages, posts, products, and other URLs they want to crawl to index in the search engines result page (SERP).

Why we need to enable a custom robots.txt file and robots header tags in the blog?

When we are creating a blog, website, and online store and we upload some content on it then we are adding our online asset to the search engine console or Webmaster tools, that help us to crawl or index our pages, posts, products, and all URLs in the search engines like - Bing, Qwant, Google, Duckduckgo, Startpage, etc..., and without robots.txt file our all content URLs (unwanted & useless URLs) will be index in the search engine without any restriction.

Now I give you some reasons to enable the robots.txt file in your blog or website, let's see some points >>
# we don't want to index some of our pages or post like we are working on a particular page or post to improve the design.
# we don't want to index some specific page in the search engine, like the DEMO page, testing pages, WP-ADMIN page, etc... 
# we don't want to index our blog Label or Category URL, like -
> mydomainname.com/lable/mobile-phone
> mydomainname.com/category /mobile-games

Now you will understand how the robots.txt file help our online assets (blog, website, online store) be crawled and indexed in the search engine and the main thing is every crawler have their time to spend in every blog or website then they are automatically redirected or jump to another blog or website so in the particular time frame, the crawler crawls the maximum pages or posts URLs and the visit another website or blog. 

But if our website or blog has a lot of pages, posts, and product URLs then the crawler cannot crawl the whole website or blog so we are enabling the robots.txt file in our blog, website, and online store to tell the crawler which URLs they want to crawls and index, For Example:-
you can visit some given URL in your web browser then you can see the settings of the robots.txt file.
https://facebook.com/robots.txt
https://www.techtspot.com/robots.txt

Now in the Blogger platform, the blog by default crawl all content URLs in the search engine results page (SERP), so here we are enabling custom robots txt file and header tag to tell all search engines how they can crawl & index our blog in their SERP.

Custom robots.txt file explanation:-
#User-agent: Mediapartners-Google:- this for google to Provide better ads on blogger blog.
#User-agent: *:- this means our blogger blog or website can crawl all search engines. It means visible to all search engines.
#Disallow: /search:- this line tells all search engines do not crawl this post who related /search post and pages.
#Allow: /:- this line tells all search engines to crawl and Allow homepage.
#Sitemap:- we add our Blogger blog sitemap, if you don't know what is sitemap I just explain the sitemap in another post you can visit and check the post.

Custom robots header tag explanation:-
#All:- it crawls all Blogger blog posts and pages and this is enabled by default.
#noindex:- this option tells all search engines to not index Blogger blog pages and post in the search area.
#nofollow:- it tells not to follow the links on the pages.
#none:- it means we have selected both noindex, nofollow.
#noarchive:- this option can remove the cache link from search results.
#nosnippet:- this option can remove a snippet of the Blogger blog post and pages.
#noodp:- this option can not allow search engines to use metadata from the DMOZ directory.
#notranslate:- this option tells Spiders not to allow to translate the Blogger blog post and pages.
#noimageindex:- this option can not allow to index images into all search engines.

Now we are going to blogger blog and do these configurations in our blog, now let's start.

These are the Following Steps:-
Step 1First go to the Blogger blog and click the Setting option, here we scroll down and find the Crawlers and indexing section then we On the button for the Enable custom robots.txt option.
Enable Custom Robots file in Blogger 1
Step 2Now a new pop-up window appears middle of the device screen, here we paste the robots.txt file code and click the Save button.
Enable Custom Robots file in Blogger 2
Custom Robots.txt file Code:-
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://blog-name.blogspot.com/sitemap.xml
Step 3: If you provide all the code in the robots file then you no need to worry about the robots header tag option but if you're not sure then you can click the On button for the Custom robots header tag and one by one you can configure these options.
Enable Custom Robots file in Blogger 3
Step 4Here first I click the Home page tags option and then I enable all and noodp option then click the Save button.
Enable Custom Robots file in Blogger 4
Step 5: Now I click the second option Archive and search page tags, here I just enable noindex and noodp option then click the Save button.
Enable Custom Robots file in Blogger 5
Step 6: Now at last I click the Post and page tags option and I just enable all and noodp option then click the Save button.
Enable Custom Robots file in Blogger 6
Step 7: Now open the Blogger blog to check there is any problem.

Read More:- 

48 comments:

  1. I posted the codes but it couldn't save. what should I do?

    ReplyDelete
    Replies
    1. hello sir if u use this code in your blog and u not save it. don't worry this code is already add in webmaster tools

      Delete
  2. Nice Trick...That Helped Me A Lot...

    ReplyDelete
  3. google search always suggesting my policy page, contact us page when I search for my blog can you help me with this sir?thanks in advance.

    ReplyDelete
  4. thankyou sir for quick response. I will give a shot for those link. More power to you and to your blog sir.

    ReplyDelete
  5. This comment has been removed by the author.

    ReplyDelete
  6. just one more thing sir can I have favor? Can you visit my blog jztechshare.com and give me some advice sir why adsense always disapproved my application. thanks again sir more power.

    ReplyDelete
  7. yes, A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.

    ReplyDelete
  8. When I enabled robot.txt and added the code in the box.
    And searched for myown-research.Blogspot.com/robots.txt
    The page said ""There's Nothing Here""

    ReplyDelete
    Replies
    1. hello sir try this https://myown-research.blogspot.com/robots.txt

      Delete
  9. Thanks for the post. this article is really nice one. Robots.txt is to guide how to crawl and index page of your website.

    ReplyDelete
  10. this is good like i did it on
    www.entertaintricks.blogspot.com

    ReplyDelete
  11. Nice guide... I just implemented the code on my blog now.

    Thanks

    ReplyDelete
  12. Help me my new blogger is not working

    ReplyDelete
  13. I have created a new blogger but this is not working like your blogger but i'm new in blogger, if you help me, I will be happy.

    ReplyDelete
  14. Can I add it like this,

    User-agent: Mediapartners-Google

    Disallow:

    User-agent: *

    Disallow: / search

    Allow: /

    Sitemap: http://www.bhihai.com/feeds/posts/default?orderby=UPDATED

    ReplyDelete
    Replies
    1. i think you can and
      thanks for comment :)

      Delete
  15. Thank you so much. I am looking for this type of post.

    ReplyDelete
  16. Whenever I add robots.txt to the end of my domain name to check if it has worked, it is giving me an error 404 message. Please what to I do

    ReplyDelete
    Replies
    1. and please check this htts://your-website-name.com/robots.txt

      Delete
  17. Thanks Admin For Helpfull Content

    ReplyDelete
  18. Things you should do before applying for AdSense https://nigeriathings.blogspot.com/2018/06/things-you-should-do-before-applying.html?m=1

    ReplyDelete
    Replies
    1. Thanks for sharing this nice, helpful and informative article. Yes, this setting is really important for our blog.

      Delete
  19. thanks to a lot …
    for this article

    ReplyDelete
  20. Thank you for this lovely post and i read your article and i found this great

    Moviesforeview


    afilmywaps

    ReplyDelete
  21. Excellent content. I am very found of reading such type of content. Thanks for sharing.

    ReplyDelete