Hello friends welcome to the Techtspot Blog, today I will show you how to enable a custom Robots.txt file and robots header tags in the Blogger blog with pictures.
What is the Robots.txt file?
Robots.txt file tells the crawler which pages, posts, products, and other URLs they want to crawl to index in the search engines result page (SERP).
Why we need to enable a custom robots.txt file and robots header tags in the blog?
When we are creating a blog, website, and online store and we upload some content on it then we are adding our online asset to the search engine console or Webmaster tools, that help us to crawl or index our pages, posts, products, and all URLs in the search engines like - Bing, Qwant, Google, Duckduckgo, Startpage, etc..., and without robots.txt file our all content URLs (unwanted & useless URLs) will be index in the search engine without any restriction.
Now I give you some reasons to enable the robots.txt file in your blog or website, let's see some points >>
# we don't want to index some of our pages or post like we are working on a particular page or post to improve the design.
# we don't want to index some specific page in the search engine, like the DEMO page, testing pages, WP-ADMIN page, etc...
# we don't want to index our blog Label or Category URL, like -
> mydomainname.com/lable/mobile-phone
> mydomainname.com/category /mobile-games
Now you will understand how the robots.txt file help our online assets (blog, website, online store) be crawled and indexed in the search engine and the main thing is every crawler have their time to spend in every blog or website then they are automatically redirected or jump to another blog or website so in the particular time frame, the crawler crawls the maximum pages or posts URLs and the visit another website or blog.
But if our website or blog has a lot of pages, posts, and product URLs then the crawler cannot crawl the whole website or blog so we are enabling the robots.txt file in our blog, website, and online store to tell the crawler which URLs they want to crawls and index, For Example:-
you can visit some given URL in your web browser then you can see the settings of the robots.txt file.
https://facebook.com/robots.txt
https://www.techtspot.com/robots.txt
Now in the Blogger platform, the blog by default crawl all content URLs in the search engine results page (SERP), so here we are enabling custom robots txt file and header tag to tell all search engines how they can crawl & index our blog in their SERP.
Custom robots.txt file explanation:-
#User-agent: Mediapartners-Google:- this for google to Provide better ads on blogger blog.
#User-agent: *:- this means our blogger blog or website can crawl all search engines. It means visible to all search engines.
#Disallow: /search:- this line tells all search engines do not crawl this post who related /search post and pages.
#Allow: /:- this line tells all search engines to crawl and Allow homepage.
#Sitemap:- we add our Blogger blog sitemap, if you don't know what is sitemap I just explain the sitemap in another post you can visit and check the post.
Custom robots header tag explanation:-
#All:- it crawls all Blogger blog posts and pages and this is enabled by default.
#noindex:- this option tells all search engines to not index Blogger blog pages and post in the search area.
#nofollow:- it tells not to follow the links on the pages.
#none:- it means we have selected both noindex, nofollow.
#noarchive:- this option can remove the cache link from search results.
#nosnippet:- this option can remove a snippet of the Blogger blog post and pages.
#noodp:- this option can not allow search engines to use metadata from the DMOZ directory.
#notranslate:- this option tells Spiders not to allow to translate the Blogger blog post and pages.
#noimageindex:- this option can not allow to index images into all search engines.
Now we are going to blogger blog and do these configurations in our blog, now let's start.
These are the Following Steps:-
Step 1: First go to the Blogger blog and click the Setting option, here we scroll down and find the Crawlers and indexing section then we On the button for the Enable custom robots.txt option.
Step 2: Now a new pop-up window appears middle of the device screen, here we paste the robots.txt file code and click the Save button.Custom Robots.txt file Code:-
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://blog-name.blogspot.com/sitemap.xml
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://blog-name.blogspot.com/sitemap.xml
Step 3: If you provide all the code in the robots file then you no need to worry about the robots header tag option but if you're not sure then you can click the On button for the Custom robots header tag and one by one you can configure these options.
Step 4: Here first I click the Home page tags option and then I enable all and noodp option then click the Save button.
Step 5: Now I click the second option Archive and search page tags, here I just enable noindex and noodp option then click the Save button.
Step 6: Now at last I click the Post and page tags option and I just enable all and noodp option then click the Save button.
Step 7: Now open the Blogger blog to check there is any problem.
Read More:-
Add Stylish 404 Error Page to Blogger blog
Top 20 Best Free SEO Tools for Blogger Blog
How to Add Customize Blogger Comment Box
How to Design Blogger blog template Theme
I hope you enjoy this Post. If any problem comment it.
Top 20 Best Free SEO Tools for Blogger Blog
How to Add Customize Blogger Comment Box
How to Design Blogger blog template Theme
I hope you enjoy this Post. If any problem comment it.
I posted the codes but it couldn't save. what should I do?
ReplyDeletehello sir if u use this code in your blog and u not save it. don't worry this code is already add in webmaster tools
DeleteNice Trick...That Helped Me A Lot...
ReplyDeletegoogle search always suggesting my policy page, contact us page when I search for my blog can you help me with this sir?thanks in advance.
ReplyDeletehello sir
Deleteyou need to fill your meta description tag and meta keywords tag you know about meta description tag post visit this link and meta keywords tag visit this link.
hello JZtechshare
Deleteone more thing add your blog on Webmaster tools visit this link.
thankyou sir for quick response. I will give a shot for those link. More power to you and to your blog sir.
ReplyDeleteThis comment has been removed by the author.
ReplyDeletejust one more thing sir can I have favor? Can you visit my blog jztechshare.com and give me some advice sir why adsense always disapproved my application. thanks again sir more power.
ReplyDeletehere read Adsence Policy visit this link.
DeleteAnd you want to make money visit this link.
thank you for comment....
yes, A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.
ReplyDeletethanks for comment
DeleteWhen I enabled robot.txt and added the code in the box.
ReplyDeleteAnd searched for myown-research.Blogspot.com/robots.txt
The page said ""There's Nothing Here""
hello sir try this https://myown-research.blogspot.com/robots.txt
Deletethanks for post.. VV
ReplyDeleteThanks for the post. this article is really nice one. Robots.txt is to guide how to crawl and index page of your website.
ReplyDeleteThanks for your supports
ReplyDeletethis is good like i did it on
ReplyDeletewww.entertaintricks.blogspot.com
Nice guide... I just implemented the code on my blog now.
ReplyDeleteThanks
thanks for comment ( Komentoarigatō )
DeleteHelp me my new blogger is not working
ReplyDeletehello ask your question / problem ?
DeleteI have created a new blogger but this is not working like your blogger but i'm new in blogger, if you help me, I will be happy.
ReplyDeletehello sir watch this video click here for create a blog using blogger. and design it in Hindi
Deletemy blogger not indexed
ReplyDeletehello freeman
DeleteHere u will be add your blog on Webmaster tools visit this link.
Can I add it like this,
ReplyDeleteUser-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: / search
Allow: /
Sitemap: http://www.bhihai.com/feeds/posts/default?orderby=UPDATED
i think you can and
Deletethanks for comment :)
Thank you so much. I am looking for this type of post.
ReplyDeletethanks for comment :)
DeleteTHANKS G !
ReplyDeleteWhenever I add robots.txt to the end of my domain name to check if it has worked, it is giving me an error 404 message. Please what to I do
ReplyDeletethanks for comment
Deleteand please check this htts://your-website-name.com/robots.txt
DeleteThanks Admin For Helpfull Content
ReplyDeleteThings you should do before applying for AdSense https://nigeriathings.blogspot.com/2018/06/things-you-should-do-before-applying.html?m=1
ReplyDeletethanks for comment :)
DeleteThanks for sharing this nice, helpful and informative article. Yes, this setting is really important for our blog.
Deletethanks to a lot …
ReplyDeletefor this article
thanks for comment
DeleteThanks for sharing the procedure.
ReplyDeleteMotivational Quotes in Hindi
Positive Thinking Quotes in Hindi
thanks for comment :)
DeleteThank you for this lovely post and i read your article and i found this great
ReplyDeleteMoviesforeview
afilmywaps
thanks for comment :)
DeleteIt's helpful article.
ReplyDeletethanks for comment :)
DeleteExcellent content. I am very found of reading such type of content. Thanks for sharing.
ReplyDeletethanks for comment :)
Delete