10 common mistakes using robots.txt on your website

One of the things Cricket Walker teaches in her SEO Training Class, is to add a robots.txt file to your website. Web site owners use the robots.txt file to give instructions to the web bots about their site. A basic file might consist of the following lines of code which tells on web bots NOT to index this site.

User-agent: *
Disallow: /

Why would you NOT want your site indexed? Perhaps it is a site you are working on and it is not yet ready for indexing.

To allow access to your entire site, the file would contain the following lines of code:

User-agent: *

The robots.txt file is located in the top level of your site, the root directory.  Additional resources on creating and understanding the robots.txt file:

10 Common Mistakes using robots.txt on your website include:

  1. Adding robots.txt NOT under the root directory
  2. Wrong syntax in robots.tx
  3. Adding comment at the end of the sentence instead of at the beginning
  4. Empty robots.txt file almost like not having one
  5. Blocking the pages which you need to get indexed
  6. URL’s Paths are case sensitive
  7. Misspelled robots/user agent names
  8. Don’t add all the files in one single line
  9. No allow command in robots.txt
  10. Missing the colon

Read 10 common mistakes using robots.txt on your website. Do you have a robots.txt file as part of your website?

Speak Your Mind