[Python-Dev] Googlebot and the mail.python.org python-dev archive (original) (raw)
Antoine Pitrou solipsis at pitrou.net
Sat Feb 28 11:37:09 CET 2009
- Previous message: [Python-Dev] Googlebot and the mail.python.org python-dev archive
- Next message: [Python-Dev] Googlebot and the mail.python.org python-dev archive
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Georg Brandl <g.brandl gmx.net> writes:
Guido van Rossum schrieb: > I think the better syntax would be to add site:mail.python.org to the > query, but you're right, that doesn't seem to find recent messages. > Maybe the absence of a robots.txt file on mail.python.org could be a > partial explanation? Doesn't the absence of a robots.txt mean "you may index everything"?
It does. However, pages such as: http://mail.python.org/pipermail/python-dev/ (and, it seems, all other pipermail-generated archive pages) have the following HTML tag in them: which explicitly instructs Web spiders not to index contents nor follow links.
Regards
Antoine.
- Previous message: [Python-Dev] Googlebot and the mail.python.org python-dev archive
- Next message: [Python-Dev] Googlebot and the mail.python.org python-dev archive
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]