Robots.txt file

From ThaiiS Note (Wiki)

A Standard for Robot Exclusion

from : http://www.robotstxt.org/orig.html

Status of this document

This document represents a consensus on 30 June 1994 on the robots mailing list (robots-request@nexor.co.uk), between the majority of robot authors and other people with an interest in robots. It has also been open for discussion on the Technical World Wide Web mailing list (www-talk@info.cern.ch). This document is based on a previous working draft under the same title.

It is not an official standard backed by a standards body, or owned by any commercial organisation. It is not enforced by anybody, and there no guarantee that all current and future robots will use it. Consider it a common facility the majority of robot authors offer the WWW community to protect WWW server against unwanted accesses by their robots.

The latest version of this document can be found on http://www.robotstxt.org/wc/robots.html.

Introduction

WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. For more information see the robots page.

In 1993 and 1994 there have been occasions where robots have visited WWW servers where they weren't welcome for various reasons. Sometimes these reasons were robot specific, e.g. certain robots swamped servers with rapid-fire requests, or retrieved the same files repeatedly. In other situations robots traversed parts of WWW servers that weren't suitable, e.g. very deep virtual trees, duplicated information, temporary information, or cgi-scripts with side-effects (such as voting).

These incidents indicated the need for established mechanisms for WWW servers to indicate to robots which parts of their server should not be accessed. This standard addresses this need with an operational solution.

The Method

The method used to exclude robots from a server is to create a file on the server which specifies an access policy for robots. This file must be accessible via HTTP on the local URL "/robots.txt". The contents of this file are specified [#format below].

This approach was chosen because it can be easily implemented on any existing WWW server, and a robot can find the access policy with only a single document retrieval.

A possible drawback of this single-file approach is that only a server administrator can maintain such a list, not the individual document maintainers on the server. This can be resolved by a local process to construct the single file from a number of others, but if, or how, this is done is outside of the scope of this document.

The choice of the URL was motivated by several criteria:

The filename should fit in file naming restrictions of all common operating systems.
The filename extension should not require extra server configuration.
The filename should indicate the purpose of the file and be easy to remember.
The likelihood of a clash with existing files should be minimal.

The Format

The format and semantics of the "/robots.txt" file are as follows:

The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL). Each record contains lines of the form "<field>:<optionalspace><value><optionalspace>". The field name is case insensitive.

Comments can be included in file using UNIX bourne shell conventions: the '#' character is used to indicate that preceding space (if any) and the remainder of the line up to the line termination is discarded. Lines containing only a comment are discarded completely, and therefore do not indicate a record boundary.

The record starts with one or more User-agent lines, followed by one or more Disallow lines, as detailed below. Unrecognised headers are ignored.

User-agent: The value of this field is the name of the robot the record is describing access policy for. If more than one User-agent field is present the record describes an identical access policy for more than one robot. At least one field needs to be present per record.The robot should be liberal in interpreting this field. A case insensitive substring match of the name without version information is recommended.If the value is '*', the record describes the default access policy for any robot that has not matched any of the other records. It is not allowed to have multiple such records in the "/robots.txt" file.
Disallow: The value of this field specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved. For example, Disallow: /help disallows both /help.html and /help/index.html, whereas Disallow: /help/ would disallow /help/index.html but allow /help.html. Any empty value, indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record.

The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.

Examples

The following example "/robots.txt" file specifies that no robots should visit any URL starting with "/cyberworld/map/" or "/tmp/", or /foo.html:

# robots.txt for http://www.example.com/
 
 User-agent: *
 Disallow: /cyberworld/map/ # This is an infinite virtual URL space
 Disallow: /tmp/ # these will soon disappear
 Disallow: /foo.html

This example "/robots.txt" file specifies that no robots should visit any URL starting with "/cyberworld/map/", except the robot called "cybermapper":

# robots.txt for http://www.example.com/
 
 User-agent: *
 Disallow: /cyberworld/map/ # This is an infinite virtual URL space
 
 # Cybermapper knows where to go.
 User-agent: cybermapper
 Disallow:

This example indicates that no robots should visit this site further:

# go away
 User-agent: *
 Disallow: /

Example Code

Although it is not part of this specification, some example code in Perl is available in norobots.pl. It is a bit more flexible in its parsing than this document specificies, and is provided as-is, without warranty.

Note: This code is no longer available. Instead I recommend using the robots exclusion code in the Perl libwww-perl5 library, available from CPAN in the LWP directory.

Author's Address

Martijn Koster

Collection of Robots.txt Files

from : http://www.dailyblogtips.com/collection-of-robotstxt-files/

The implementation of a suitable robots.txt file is very important for search engine optimization. There is plenty of advice around the Internet for the creation of such files (if you are looking for an introduction on this topic read “Creat a robots.txt file“), but what if instead of looking at what people say we could look at what people do?

That is what I did, collecting the robots.txt files from a wide range of blogs and websites. Below you will find them. Key Takeaways

Only 2 out of 30 websites that I checked were not using a robots.txt file
Even if you don’t have any specific requirements for the search bots, therefore, you probably should use a simple robots.txt file
Most people stick to the “User-agent: *” attribute to cover all agents
The most common “Disallowed” factor is the RSS Feed
Google itself is using a combination of closed folders (e.g., /searchhistory/) and open ones (e.g., /search), which probably means they are treated differently
A minority of the sites included the sitemap URL on the robots.txt file

The Minimalistic Guys

Problogger.net

   User-agent: *
   Disallow:

Marketing Pilgrim

   User-agent: *
   Disallow:

Search Engine Journal

   User-agent: *
   Disallow:

Matt Cutts

   User-agent: *
   Allow:
   User-agent: *
   Disallow: /files/

Pronet Advertising

   User-agent: *
   Disallow: /mt
   Disallow: /*.cgi$

TechCrunch

   User-agent: *
   Disallow: /*/feed/
   Disallow: /*/trackback/

The Structured Ones

Online Marketing Blog

   User-agent: Googlebot
   Disallow: */feed/

   User-agent: *
   Disallow: /Blogger/
   Disallow: /wp-admin/
   Disallow: /stats/
   Disallow: /cgi-bin/
   Disallow: /2005x/

Shoemoney

   User-Agent: Googlebot
   Disallow: /link.php
   Disallow: /gallery2
   Disallow: /gallery2/
   Disallow: /category/
   Disallow: /page/
   Disallow: /pages/
   Disallow: /feed/
   Disallow: /feed

Scoreboard Media

   User-agent: *
   Disallow: /cgi-bin/
   User-agent: Googlebot
   Disallow: /category/
   Disallow: /page/
   Disallow: */feed/
   Disallow: /2007/
   Disallow: /2006/
   Disallow: /wp-*

SEOMoz.org

   User-agent: *
   Disallow: /blogdetail.php?ID=537
   Disallow: /blog?page
   Disallow: /blog/author/
   Disallow: /blog/category/
   Disallow: /tracker
   Disallow: /ugc?page
   Disallow: /ugc/author/
   Disallow: /ugc/category/

Wolf-Howl

   User-agent: *
   Disallow: /cgi-bin/
   Disallow: /images/
   Disallow: /noindex/
   Disallow: /privacy-policy/
   Disallow: /about/
   Disallow: /company-biographies/
   Disallow: /press-media-room/
   Disallow: /newsletter/
   Disallow: /contact-us/
   Disallow: /terms-of-service/
   Disallow: /terms-of-service/
   Disallow: /information/comment-policy/
   Disallow: /faq/
   Disallow: /contact-form/
   Disallow: /advertising/
   Disallow: /information/licensing-information/
   Disallow: /2005/
   Disallow: /2006/
   Disallow: /2007/
   Disallow: /2008/
   Disallow: /2009/
   Disallow: /2004/
   Disallow: /*?*
   Disallow: /page/
   Disallow: /iframes/

John Chow

   sitemap: http://www.johnchow.com/sitemap.xml
   User-agent: *
   Disallow: /cgi-bin/
   Disallow: /go/
   Disallow: /wp-admin/
   Disallow: /wp-includes/
   Disallow: /author/
   Disallow: /page/
   Disallow: /category/
   Disallow: /wp-images/
   Disallow: /images/
   Disallow: /backup/
   Disallow: /banners/
   Disallow: /archives/
   Disallow: /trackback/
   Disallow: /feed/
   User-agent: Googlebot-Image
   Allow: /wp-content/uploads/
   User-agent: Mediapartners-Google
   Allow: /
   User-agent: duggmirror
   Disallow: /

Smashing Magazine

   Sitemap: http://www.smashingmagazine.com/sitemap.xml
   User-agent: Mediapartners-Google*
   Disallow:
   User-agent: *
   Disallow: /styles/
   Disallow: /inc/
   Disallow: /tag/
   Disallow: /cc/
   Disallow: /category/
   User-agent: MSIECrawler
   Disallow: /
   User-agent: psbot
   Disallow: /
   User-agent: Fasterfox
   Disallow: /
   User-agent: Slurp
   Crawl-delay: 200

Gizmodo

   User-Agent: Googlebot
   Disallow: /index.xml$
   Disallow: /excerpts.xml$
   Allow: /sitemap.xml$
   Disallow: /*view=rss$
   Disallow: /*?view=rss$
   Disallow: /*format=rss$
   Disallow: /*?format=rss$
   Sitemap: http://gizmodo.com/sitemap.xml

Lifehacker

   User-Agent: Googlebot
   Disallow: /index.xml$
   Disallow: /excerpts.xml$
   Allow: /sitemap.xml$
   Disallow: /*view=rss$
   Disallow: /*?view=rss$
   Disallow: /*format=rss$
   Disallow: /*?format=rss$
   Sitemap: http://lifehacker.com/sitemap.xml

The Mainstream Media

Wall Street Journal

   User-agent: *
   Disallow: /article_email/
   Disallow: /article_print/
   Disallow: /PA2VJBNA4R/
   Sitemap: http://online.wsj.com/sitemap.xml

ZDNet

   User-agent: *
   Disallow: /Ads/
   Disallow: /redir/
   # Disallow: /i/ is removed per 190723
   Disallow: /av/
   Disallow: /css/
   Disallow: /error/
   Disallow: /clear/
   Disallow: /mac-ad
   Disallow: /adlog/
   # URS per bug 239819, these were expanded
   Disallow: /1300-
   Disallow: /1301-
   Disallow: /1302-
   Disallow: /1303-
   Disallow: /1304-
   Disallow: /1305-
   Disallow: /1306-
   Disallow: /1307-
   Disallow: /1308-
   Disallow: /1309-
   Disallow: /1310-
   Disallow: /1311-
   Disallow: /1312-
   Disallow: /1313-
   Disallow: /1314-
   Disallow: /1315-
   Disallow: /1316-
   Disallow: /1317-

NY Times

   # robots.txt, www.nytimes.com 6/29/2006
   #
   User-agent: *
   Disallow: /pages/college/
   Disallow: /college/
   Disallow: /library/
   Disallow: /learning/
   Disallow: /aponline/
   Disallow: /reuters/
   Disallow: /cnet/
   Disallow: /partners/
   Disallow: /archives/
   Disallow: /indexes/
   Disallow: /thestreet/
   Disallow: /nytimes-partners/
   Disallow: /financialtimes/
   Allow: /pages/
   Allow: /2003/
   Allow: /2004/
   Allow: /2005/
   Allow: /top/
   Allow: /ref/
   Allow: /services/xml/
   User-agent: Mediapartners-Google*
   Disallow:

YouTube

   # robots.txt file for YouTube
   User-agent: Mediapartners-Google*
   Disallow:
   User-agent: *
   Disallow: /profile
   Disallow: /results
   Disallow: /browse
   Disallow: /t/terms
   Disallow: /t/privacy
   Disallow: /login
   Disallow: /watch_ajax
   Disallow: /watch_queue_ajax

Bonus

Google

   User-agent: *
   Allow: /searchhistory/
   Disallow: /news?output=xhtml&
   Allow: /news?output=xhtml
   Disallow: /search
   Disallow: /groups
   Disallow: /images
   Disallow: /catalogs
   Disallow: /catalogues
   Disallow: /news
   Disallow: /nwshp
   Disallow: /?
   Disallow: /addurl/image?
   Disallow: /pagead/
   Disallow: /relpage/
   Disallow: /relcontent
   Disallow: /sorry/
   Disallow: /imgres
   Disallow: /keyword/
   Disallow: /u/
   Disallow: /univ/
   Disallow: /cobrand
   Disallow: /custom
   Disallow: /advanced_group_search
   Disallow: /advanced_search
   Disallow: /googlesite
   Disallow: /preferences
   Disallow: /setprefs
   Disallow: /swr
   Disallow: /url
   Disallow: /default
   Disallow: /m?
   Disallow: /m/search?
   Disallow: /wml?
   Disallow: /wml/search?
   Disallow: /xhtml?
   Disallow: /xhtml/search?
   Disallow: /xml?
   Disallow: /imode?
   Disallow: /imode/search?
   Disallow: /jsky?
   Disallow: /jsky/search?
   Disallow: /pda?
   Disallow: /pda/search?