Increse Traffic with Fewer Indexed Pages
By restricting the number of pages of your site that Google indexes, you can actually increase your traffic. Essentially, when there are tons of pages on your site, Google can become somewhat confused as to what the content of your site is all about. By restricitng auxilliary pages in your robot.tx file, you can help Google develop a better picture of what your site is all about. zz Online Marketing explains:
If you have a website about Elvis with 3,000 members and 1,000 pages of article content about Elvis, you may not be doing yourself a favour by showing Google all of this information.
When Google performs a crawl, it can say “okay, I’ve found 1,000 pages about Elvis, I’m not sure what these 3,000 pages are about, but they have a lot of links going to them so they must be fairly important”. The result being Google is slightly confused about the content of your site, it knows it’s about Elvis, but not quite sure the whole site is about that. Links to these member pages are also passing Pagerank (and authority) to these pages which are not important to your search.
By using the robots.txt file to exclude these pages from crawling, you might nudge your PR up a notch by passing the authority to relevant pages and Google will have a better holistic view of the content of your website. Result should be increased visibility for searches around your content!
As an example, zz Online Marketing suggests taking a look at Shoemoney's robot.txt file.



















Comments
I'd look at roboting your feed, Google seems to love sucking those things up - but adds noise around your actual content pages :-)
Thanks for stopping by!
Posted by: MarkZZ | February 5, 2007 11:51 AM