What's this "robots.txt" file I'm seeing in my error logs?

Search engines look for files called "robots.txt" to tell them which pages in a website to index, and which they should leave alone. It's a pretty simple process, and you can read much more about web robots, how search engines use them, and how they interact with your website by going to The Web Robots Pages, which has a ton of really good information.

Interestingly enough, the robots.txt protocol isn't an official standard (few things on the Internet are, after all), but it is supported by most search engines, especially the bigger ones.

If you want to take advantage of this functionality, or even just to make that "file not found: robots.txt" error message disappear from your logs, you can create a robots.txt file that tells search engine spiders and robots which pages to ignore and place it in your /htdocs directory.

We'll be adding a FAQ on how exactly to go about doing this very soon, but for now, just drop us a line in email letting us know you're interested, and we'll be happy to walk you through the process.

If you don't feel like bothering with a robots.txt file, but you have webpages you'd prefer search engines not to index, you can keep web robots (or spiders, as they're also called) from crawling all over your website by using a meta tag. It's very simple: just cut and paste the following tag into the HTML of any page you'd like to protect. For best results, place it between the <head> and </head> tags.

Do keep in mind that this will only protect those webpages against nosy robots—nosy people may still be able to find them. If you have pages you'd really like to protect, you'll want to look into password protecting areas of your website with .htaccess (which is another feature Hosting for Humans provides free of charge with every web hosting account).

We'll also be adding a FAQ on how to use .htaccess soon, but for now, send us email telling us you're interested in password protection, and we'll tell you all about it.

As always, if you have any questions or problems, contact us at support@hostingforhumans.com and we’ll get right back to you.

home hosting contact faqs about order