Necessity of robots.txt

Discussion in 'Search Engine Optimization and Marketing' started by jeanm, Mar 14, 2018.

to remove all ads.
  1. jeanm

    jeanm Member

    Messages:
    150
    In recent times website traffic to my business site has decreased. I'm told by someone "knowledgeable" that my robots.txt file is turned off and that I need to turn it on immediately. A search of my website indicates I don't have a robots.txt file there at all. I think when I built the site many years ago that I thought everything on the site should be crawled so I never included one.

    Should I include a robots.txt file that disallows nothing and allows everything or is that silly?

    PS I'm aware my website is old style and not mobile-friendly.
     
  2. chrishirst

    chrishirst Well-Known Member Staff Member

    Messages:
    2,657
    The ONLY thing a robots.txt file is required for is blocking access to parts of the site, if you do not need to do that a robots.txt file is completely unnecessary.
     
  3. jeanm

    jeanm Member

    Messages:
    150
    Thanks chrishirst. I'm now doubtful about other information this "knowledgeable" person is giving me.
     
  4. chrishirst

    chrishirst Well-Known Member Staff Member

    Messages:
    2,657
    Yep, you should be doubtful about ALL the advice you get [including mine], but be especially doubtful of any body who proclaims themselves an "expert".
     
  5. Kim Kiav

    Kim Kiav Member

    Messages:
    34
    Robots.xml file is only used to restrict search engines from crawling certain sections of your website. It absolutely doesn't affect website traffic
     
  6. chrishirst

    chrishirst Well-Known Member Staff Member

    Messages:
    2,657
    Nope, robots.xml is not a thing for excluding search bots.
     

Share This Page