HRobertson
New Member
I'm a little nervous about adding restrictions on my robots.txt file. I want to allow all but a couple of pages from being indexed by all robots. Could someone please verify that the following will work:
User-Agent: *
Disallow: /hierarchy1/
Disallow: /westernphilosophy/
Want I need to ensure is that everything But the URLs containing the above mentioned strings are crawled and hopefully indexed. Will this work?
Any help is greatly appreciated. Thank you...
User-Agent: *
Disallow: /hierarchy1/
Disallow: /westernphilosophy/
Want I need to ensure is that everything But the URLs containing the above mentioned strings are crawled and hopefully indexed. Will this work?
Any help is greatly appreciated. Thank you...