Google pushes for an official web crawler standard

One of the cornerstones of Google's business (and really, the web at large) is the robots.txt file that sites use to exclude some of their content from the search engine's web crawler, Googlebot. It minimizes pointless indexing and sometimes keeps s…
Source: Engadget | Google pushes for an official web crawler standard