Today I’am gonna show you guys How to Search Engine Optimize Robots.txt File to Get Organic Traffic.Each and everyone with a Worpress Blog have submitted his/her site to Google Webmaster Tools Right ? And that’s it Right ? Have you ever had a look inside Google Webmaster Tools Options ? There is a file called Robots.txt (Its a Text File).The default settings are not bad but also not good so in order to get maximum benefits from that file.I will show you guys How to Search Engine Optimize Robots.txt File to Get Organic Traffic.If you don’t know what is Robots.txt and what it does / how it helps then you can read it below.

How to Search Engine Optimize Robots.txt File to Get Organic Traffic :-

What is Robots.txt File and How it Helps ?

Robots txt and Search Engines

Image Credit :-

What is Robots.txt File :-

Robot Exclusion Standard which is also known as Robots Exclusion Protocol or in short known as robots.txt protocol actually its considered the best way to prevent web crawlers i.e spider  or web robots from accessing your site as you want actually Robots.txt file prevents such web crawlers (spiders) or web robots from accessing all or the part of your website which you don’t wanna get indexed or in other words the part of the site which you don’t wanna make it publicly viewable.Search Engines like Google, Yahoo, Bing etc use robots to scan websites on the Internet in order to categorize and archive web sites and its web pages.

Related Post :- How to Download Whatsapp Plus for Android Free !

Related Post :- How to Download Android Apps on Pc from Google PLay Store !

Related Post :- How to Run Whatsapp on Ipad !

Related Post :- How to Run Whatsapp On Pc (Windows and Mac Supported) !

How Robots.txt File Helps :-

Basically in short this file lets you control the web robots (bots) and web crawlers (spiders) Cool right ? So now you can control those Search Engines bots and crawlers and can command them to Index just the parts of your site which you want.Just Imagine if you have a WordPress Blog and you see on search engines all your wp-admin files getting Indexed how will it look ? Of course no one would like wp-admin personal files getting indexed on Google so Today I will show you guys How to Search Engine Optimize Robots.txt file by editing it and making it perfect by which you can Improve your SERP rankings i.e Search Engine Results Page ranking and ultimately better rankings will lead to Organic Traffic right ? Yeaaa So lets get started.

How to Search Engine Optimize your WordPress Robots.txt File :-

Search Engine Optimize Robots txt File

The Robots.txt file is always in  the root of your site i.e you can view my robots.txt file from here Robots.txt file would look different than mine.Now in order to Search Engine Optimize your Robots.txt file you will need to make it look it like my Robots.txt file as you can see in the Image below.

Crazy Tech Solutions Robots txt File

How to Edit and Search Engine Optimize your Robots.txt File :-

Edit Robots txt File in WordPress

1)First of all Login to your Hosting account Control Panel.I’am using Hostgator Hosting so my C-panel would look different if you are on any other host, after logging in to your C-panel, there you will see a option called as File Manager as you can see in the Image below so just click on that File Manager.File Manager is a place where your site files are stored.

Hostgator C-panel (File Manager)

2)As soon as you will click on Go you will be taken to a new tab where you will see your server or you can say your site’s files now there you must see Robots.txt file.If you can’t see that file then just click on New File from the Top and then just enter file name as robots.txt and click on Create New File.As you can see in the Image below Robots.txt file is being created.

C-panel Robots txt File Created

(Note:-If you already have Robots.txt file then don’t create a new one !)

3)Now just right click on that Robots.txt file and then click on Code Edit and then a New Screen will come As you can see in the Image below just select character encoding as utf-8 and then click on Edit.

C-panel Robots txt File Edited

4)Now you will be taken to your Robots.txt file and now just copy and paste the lines given below to your Robots.txt file.


User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /recommended/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: /wp-content/plugins/

User-agent: NinjaBot
Allow: /

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

Note :- Mostly Site’s have Sitemap URL as but still cross check your Sitemap URL and then enter the URL properly into your Robots.txt file 🙂

After Pasting those lines to your Robots.txt file just click on Save Changes.

Robots txt File Search Engine Optimized and Saved

Must Read :- How to Search Engine Optimize your WordPress Blog!

Must Read :- How to Index Posts Fast on Search Engines !

Must Read :-How To Remove Copied Content From Google Free Using Google DMCA !

Now as you have optimized your Robots.txt file so now you won’t have Bad Links or Unwanted Links Indexed but you will need to Remove old Bad Links so in order to know how to remove them just read my post by clicking on the link below.

Read This :- How to Find and Remove Bad URLs from Google to Increase Rankings !

Congratulations That’s it now you have successfully Search Engine Optimized your Robots.txt file 🙂 Now pls If you liked this post then your feedback will be appreciated in the comments below and also pls share this post and if you have any doubts then feel free to comment down below I will be glad to assist you,you can also Subscribe to my Newsletter below to stay updated with my latest posts which will right away delivered to your inbox as soon as I publish or you can also Follow me on Twitter or like me on Facebook or Add me to your Google Plus Circles.