Necessity of robots.txt

jeanm

Member
In recent times website traffic to my business site has decreased. I'm told by someone "knowledgeable" that my robots.txt file is turned off and that I need to turn it on immediately. A search of my website indicates I don't have a robots.txt file there at all. I think when I built the site many years ago that I thought everything on the site should be crawled so I never included one.

Should I include a robots.txt file that disallows nothing and allows everything or is that silly?

PS I'm aware my website is old style and not mobile-friendly.
 

chrishirst

Well-Known Member
Staff member
The ONLY thing a robots.txt file is required for is blocking access to parts of the site, if you do not need to do that a robots.txt file is completely unnecessary.
 

jeanm

Member
Thanks chrishirst. I'm now doubtful about other information this "knowledgeable" person is giving me.
 

chrishirst

Well-Known Member
Staff member
Yep, you should be doubtful about ALL the advice you get [including mine], but be especially doubtful of any body who proclaims themselves an "expert".
 

Kim Kiav

Member
Robots.xml file is only used to restrict search engines from crawling certain sections of your website. It absolutely doesn't affect website traffic
 
Top