Free Website Analysis
Google Webmaster Tools, Sitemaps, and Robots.txtOn a recent project I found that the index page, along with a few other key pages were not indexed by Google. I logged into Google Webmaster Tools and verified the site. The crawl stats interestingly enough said that the it had been prohibited from reading these important files because of the robots.txt file. I checked the file and there was no sign of anything that could have cause that, but whatever, so i just cleared out everything. Better to let Google read all the non password protected files than to prohibit it from the index page. Now I've submitted a Google sitemap with the help of XML Sitemaps. This should clear everything up.Tip: after you generate an XML sitemap file, make sure to modify the importance of the first 20 urls, especially if they are updated regularly or link to other important pages. Default importance is .5 and the highest is 1. Wed, 15 Aug 2007 11:44:11 Comments Related Articles |
Add Comment