From: Joey Hess <joey@kodama.kitenet.net>
Date: Mon, 10 Nov 2008 23:08:09 +0000 (-0500)
Subject: add a robots.txt
X-Git-Tag: 2.69~18
X-Git-Url: http://git.vanrenterghem.biz/git.ikiwiki.info.git/commitdiff_plain/1ea66c3d3f0a33bc3f04d073457b525a70380c37?hp=-c

add a robots.txt

Yahoo! has been pounding on ikiwiki.cgi again. While I'd prefer ikiwiki to
generate sites that avoided robots hitting it in other ways, I'm adding a
robots.txt, at least temporarily.
---

1ea66c3d3f0a33bc3f04d073457b525a70380c37
diff --git a/doc/robots.txt b/doc/robots.txt
new file mode 100644
index 000000000..7be87f9bd
--- /dev/null
+++ b/doc/robots.txt
@@ -0,0 +1,2 @@
+User-Agent: *
+Disallow: /ikiwiki.cgi