diff options
| author | Tim-Philipp Müller <tim.muller@collabora.co.uk> | 2010-01-03 12:35:02 +0000 |
|---|---|---|
| committer | Tim-Philipp Müller <tim.muller@collabora.co.uk> | 2010-01-03 12:35:02 +0000 |
| commit | a457ac67e0238187360da2f9f6491db9a8d374b3 (patch) | |
| tree | 34a948200b777290221c29cc3c25f5c93a6a5c39 | |
| parent | 533cb02c013a66eea168833c99787dcc0f3a327c (diff) | |
robots: hide non-head docs from google and yahoo bots
Attempt to make google & co show the latest GStreamer docs rather
than the oldest they can find (until someone comes up with some
redirect rules or a better solution). Allow/Disallow use here may
not be supported by all user agents, so restricted to these two
for now.
| -rw-r--r-- | src/copy/robots.txt | 10 |
1 files changed, 10 insertions, 0 deletions
diff --git a/src/copy/robots.txt b/src/copy/robots.txt index b1c11ec0..8cbc3020 100644 --- a/src/copy/robots.txt +++ b/src/copy/robots.txt @@ -2,3 +2,13 @@ # don't want them to show up when people search for 'gstreamer windows' User-agent: * Disallow: /pkg/windows/ + +# don't want ancient docs to show up in e.g. Google or Yahoo +# (maybe we should just remove the old docs from the server?) +User-agent: Slurp +Allow: /data/doc/gstreamer/head/ +Disallow: /data/doc/gstreamer/ + +User-agent: Googlebot +Allow: /data/doc/gstreamer/head/ +Disallow: /data/doc/gstreamer/ |