diff options
| author | Danilo G. Baio <dbaio@FreeBSD.org> | 2021-08-08 14:12:03 +0000 |
|---|---|---|
| committer | Danilo G. Baio <dbaio@FreeBSD.org> | 2021-08-13 22:51:16 +0000 |
| commit | 6af77691110d19019b389fad3a0700286fdf0ef1 (patch) | |
| tree | 8d441513f0424f8abcd268c0290ced0bfc97a1d8 | |
| parent | 09ad8f3a81aef4928007292007b896182b47cd8b (diff) | |
documentation/robots.txt: Disallow Past Releases Documentation
It was noticed that Google/DDG sometimes are sending users to the past
releases documentation. To avoid misinformation, disallow this archive
from the search engines.
This robots.txt only covers docs.FreeBSD.org and not www.FreeBSD.org, so
also remove unnecessary items.
Differential Revision: https://reviews.freebsd.org/D31459
| -rw-r--r-- | documentation/themes/beastie/layouts/robots.txt | 19 |
1 files changed, 1 insertions, 18 deletions
diff --git a/documentation/themes/beastie/layouts/robots.txt b/documentation/themes/beastie/layouts/robots.txt index b94bcd43cf..f59c629169 100644 --- a/documentation/themes/beastie/layouts/robots.txt +++ b/documentation/themes/beastie/layouts/robots.txt @@ -1,19 +1,2 @@ -# $FreeBSD$ - User-agent: * -#Crawl-delay: 10 -Disallow: /cgi/confirm-code.cgi -Disallow: /cgi/cvsweb.cgi -Disallow: /cgi/dosendpr.cgi -Disallow: /cgi/getmsg.cgi -Disallow: /cgi/mailindex.cgi -Disallow: /cgi/man.cgi?apropos=2& -Disallow: /cgi/mid.cgi -Disallow: /cgi/mirror.cgi -Disallow: /cgi/missing_handler.cgi -Disallow: /cgi/ports.cgi -Disallow: /cgi/query-pr-summary.cgi -Disallow: /cgi/query-pr.cgi -Disallow: /cgi/search.cgi -Disallow: /cgi/url.cgi -Disallow: /statistic +Disallow: /doc/ |