| msg218308 - (view) |
Author: Raymond Hettinger (rhettinger) *  |
Date: 2014-05-12 01:35 |
| Resources: * http://en.wikipedia.org/wiki/Robots_exclusion_standard#Nonstandard_extensions * https://support.google.com/webmasters/answer/183669?hl=en * https://github.com/seomoz/reppy |
|
|
| msg218318 - (view) |
Author: Berker Peksag (berker.peksag) * |
Date: 2014-05-12 09:26 |
| There is a patch for Crawl-delay in issue 16099. |
|
|
| msg252528 - (view) |
Author: Berker Peksag (berker.peksag) * |
Date: 2015-10-08 10:09 |
| The Crawl-delay part(issue 16099) is now committed. |
|
|
| msg253027 - (view) |
Author: Peter Wirtz (pwirtz) * |
Date: 2015-10-15 03:10 |
| I would like to tackle this issue. Should I wait for to be resolved first? |
|
|
| msg253035 - (view) |
Author: Berker Peksag (berker.peksag) * |
Date: 2015-10-15 08:23 |
| issue 25400 is not a blocker of this, so feel free to write a patch. |
|
|
| msg253063 - (view) |
Author: Peter Wirtz (pwirtz) * |
Date: 2015-10-15 19:51 |
| Here is a patch that provides support for the Sitemap extension. |
|
|
| msg255225 - (view) |
Author: Stéphane Wirtel (matrixise) * |
Date: 2015-11-23 20:50 |
| Add a test with your patch. Thank you |
|
|
| msg255228 - (view) |
Author: Berker Peksag (berker.peksag) * |
Date: 2015-11-23 21:17 |
| Peter didn't write a test because issue 25497 (test_robotparser rewrite) needs to be committed first. See in issue 25400 for more information. |
|
|
| msg311416 - (view) |
Author: Stéphane Wirtel (matrixise) * |
Date: 2018-02-01 10:10 |
| Hi @berker and @pwirtz. could you write a test for this issue? thanks |
|
|
| msg315362 - (view) |
Author: Steven Steven (stevensalbert) |
Date: 2018-04-16 18:29 |
| Kindly add a test for this issue |
|
|
| msg316740 - (view) |
Author: Lady Red (mcscope@gmail.com) * |
Date: 2018-05-15 22:06 |
| I wrote a test for this as it seems to have been abandoned, and opened a PR. https://github.com/python/cpython/pull/6878 |
|
|
| msg316743 - (view) |
Author: Lady Red (mcscope@gmail.com) * |
Date: 2018-05-15 22:09 |
| Sorry, wrong PR number. it is 6883, and attached to this ticket |
|
|
| msg316811 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2018-05-16 14:52 |
| New changeset 5db5c0669e624767375593cc1a01f32092c91c58 by Ned Deily (Christopher Beacham) in branch 'master': bpo-21475: Support the Sitemap extension in robotparser (GH-6883) https://github.com/python/cpython/commit/5db5c0669e624767375593cc1a01f32092c91c58 |
|
|
| msg316813 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2018-05-16 14:54 |
| Thanks for the patch, Peter, and thanks for the PR and test, Lady Red! Merged for release in 3.8.0. |
|
|