Preventing Search Engines From Crawling Your Web PagesWriting by Nick Stamoulis on Thursday, 10 of January , 2008 at 7:51 am
Matt Cutts has a good video today on Google Webmaster Central explaining how to prevent certain pages on your website from being crawled by the search engines.
You really need to be familiar with four methods of preventing the spiders from crawling your pages:
htaccessnoindexnofollowrobots.txtpassword protect
Tags: Crawling, Engines, From, Pages, Preventing, Search, YourPreventing Robots From Crawling Your PagesWriting by Nick Stamoulis on Saturday, 10 of November , 2007 at 4:13 pm
There might be a time when you do not want the search engines to crawl certain pages. You can prevent them from crawling certain pages by simply inserting the right code in its proper place. Contrary to some people’s thinking, you wouldn’t use the “no follow” tag.
The no follow tag is used to tell the search engines not to transfer your PageRank to the pages your links connect with. That’s a different thing altogether. The search engines will still crawl those links and index the pages you link to. But if you don’t want them indexing the pages on the other side of those links what would you do?
Tags: Crawling, From, Pages, Preventing, Robots, Your