So the site I'm working on has an admin page. Seems to me that this should be made unavailable to search engines. I already put it in an .htAccess controlled directory, but still, I want it hidden from spiders and such. any idea how?
Hey GRY086, be aware that there are spiders out there that will completely ignore your robots.txt (some even exploit it by seeing what you want ignored) so pay special attention to the link 3dmagical posted when they speak about "bad robots"
I was surprised to see my website show up on google when I searched my name which was also in the html. Actually go by my middle name now for that reason.
You can also use .htaccess as an extra layer or protection for your admin folder (generate one login username and password and admins will be able to access the login page but search engines wont be able to