One of the things Cricket Walker teaches in her SEO Training Class, is to add a robots.txt file to your website. Web site owners use the robots.txt file to give instructions to the web bots about their site. A basic file might consist of the following lines of code which tells on web bots NOT to index this site.
User-agent: *
Disallow: /
Why would you NOT want your site indexed? Perhaps it is a site you are working on and it is not yet ready for indexing.
To allow access to your entire site, the file would contain the following lines of code:
User-agent: *
Disallow:
The robots.txt file is located in the top level of your site, the root directory. Additional resources on creating and understanding the robots.txt file:
10 Common Mistakes using robots.txt on your website include:
- Adding robots.txt NOT under the root directory
- Wrong syntax in robots.tx
- Adding comment at the end of the sentence instead of at the beginning
- Empty robots.txt file almost like not having one
- Blocking the pages which you need to get indexed
- URL’s Paths are case sensitive
- Misspelled robots/user agent names
- Don’t add all the files in one single line
- No allow command in robots.txt
- Missing the colon
Read 10 common mistakes using robots.txt on your website. Do you have a robots.txt file as part of your website?
