[3.15] gh-153404: Silently ignore non-decimal digits in robots.txt Crawl-delay and Request-rate (GH-153405) (#153830)
hugovk <[email protected]> Fri, 31 Jul 2026 05:28:28 -0400 (EDT)
| Newsgroups | gmane.comp.python.cvs |
|---|---|
| Message-ID | <[email protected]> |
aHR0cHM6Ly9naXRodWIuY29tL3B5dGhvbi9jcHl0aG9uL2NvbW1pdC8yNzM3MmMyOWFjN2Q5ODZl MjdlOWFjNDM2ZmQ4ZDgxOTdjMzMzMmM2DQpjb21taXQ6IDI3MzcyYzI5YWM3ZDk4NmUyN2U5YWM0 MzZmZDhkODE5N2MzMzMyYzYNCmJyYW5jaDogMy4xNQ0KYXV0aG9yOiBNaXNzIElzbGluZ3RvbiAo Ym90KSA8MzE0ODg5MDkrbWlzcy1pc2xpbmd0b25AdXNlcnMubm9yZXBseS5naXRodWIuY29tPg0K Y29tbWl0dGVyOiBodWdvdmsgPDEzMjQyMjUraHVnb3ZrQHVzZXJzLm5vcmVwbHkuZ2l0aHViLmNv bT4NCmRhdGU6IDIwMjYtMDctMzFUMTI6Mjg6MTMrMDM6MDANCnN1bW1hcnk6DQoNClszLjE1XSBn aC0xNTM0MDQ6IFNpbGVudGx5IGlnbm9yZSBub24tZGVjaW1hbCBkaWdpdHMgaW4gcm9ib3RzLnR4 dCBDcmF3bC1kZWxheSBhbmQgUmVxdWVzdC1yYXRlIChHSC0xNTM0MDUpICgjMTUzODMwKQ0KDQpD by1hdXRob3JlZC1ieTogdG9uZ2h1YXJvb3QgKOerpeivnSkgPHRvbmdodWFyb290QGdtYWlsLmNv bT4NCg0KZmlsZXM6DQpBIE1pc2MvTkVXUy5kL25leHQvTGlicmFyeS8yMDI2LTA3LTA5LTE2LTM1 LTQ3LmdoLWlzc3VlLTE1MzQwNC5kYVJrZXMucnN0DQpNIExpYi90ZXN0L3Rlc3Rfcm9ib3RwYXJz ZXIucHkNCk0gTGliL3VybGxpYi9yb2JvdHBhcnNlci5weQ0KDQpkaWZmIC0tZ2l0IGEvTGliL3Rl c3QvdGVzdF9yb2JvdHBhcnNlci5weSBiL0xpYi90ZXN0L3Rlc3Rfcm9ib3RwYXJzZXIucHkNCmlu ZGV4IGNkMTQ3NzAzN2U5NGI3NC4uMWVjNjRkYTA2NGQ1MmI3IDEwMDY0NA0KLS0tIGEvTGliL3Rl c3QvdGVzdF9yb2JvdHBhcnNlci5weQ0KKysrIGIvTGliL3Rlc3QvdGVzdF9yb2JvdHBhcnNlci5w eQ0KQEAgLTE4OCw2ICsxODgsOCBAQCBkZWYgdGVzdF9yZXF1ZXN0X3JhdGUoc2VsZik6DQogICAg ICAgICAgICAgICAgICAgICAgICAgcGFyc2VkX3JlcXVlc3RfcmF0ZS5zZWNvbmRzLA0KICAgICAg ICAgICAgICAgICAgICAgICAgIHNlbGYucmVxdWVzdF9yYXRlLnNlY29uZHMNCiAgICAgICAgICAg ICAgICAgICAgICkNCisgICAgICAgICAgICAgICAgZWxzZToNCisgICAgICAgICAgICAgICAgICAg IHNlbGYuYXNzZXJ0SXNOb25lKHBhcnNlZF9yZXF1ZXN0X3JhdGUpDQogDQogDQogY2xhc3MgRW1w dHlGaWxlVGVzdChCYXNlUmVxdWVzdFJhdGVUZXN0LCB1bml0dGVzdC5UZXN0Q2FzZSk6DQpAQCAt MjQ2LDYgKzI0OCwzMiBAQCBjbGFzcyBJbnZhbGlkQ3Jhd2xEZWxheVRlc3QoQmFzZVJvYm90VGVz dCwgdW5pdHRlc3QuVGVzdENhc2UpOg0KICAgICBiYWQgPSBbXQ0KIA0KIA0KK2NsYXNzIE5vbkRl Y2ltYWxEaWdpdHNUZXN0KEJhc2VSZXF1ZXN0UmF0ZVRlc3QsIHVuaXR0ZXN0LlRlc3RDYXNlKToN CisgICAgIyBOb24tZGVjaW1hbCBVbmljb2RlIGRpZ2l0cyBwYXNzIHN0ci5pc2RpZ2l0KCkgYnV0 IGludCgpIHJlamVjdHMNCisgICAgIyB0aGVtLCBzbyB0aGUgZGlyZWN0aXZlIG11c3QgYmUgc2ls ZW50bHkgaWdub3JlZCwgbm90IHJhaXNlLg0KKyAgICByb2JvdHNfdHh0ID0gIiIiXA0KK1VzZXIt QWdlbnQ6ICoNCitEaXNhbGxvdzogL3RtcC8NCitDcmF3bC1kZWxheTogwrINCitSZXF1ZXN0LXJh dGU6IMKyLzUNCisgICAgIiIiDQorICAgIGdvb2QgPSBbJy9mb28uaHRtbCddDQorICAgIGJhZCA9 IFsnL3RtcC8nXQ0KKyAgICBjcmF3bF9kZWxheSA9IE5vbmUNCisgICAgcmVxdWVzdF9yYXRlID0g Tm9uZQ0KKw0KKw0KK2NsYXNzIE5vbkRlY2ltYWxEZW5vbWluYXRvclRlc3QoQmFzZVJlcXVlc3RS YXRlVGVzdCwgdW5pdHRlc3QuVGVzdENhc2UpOg0KKyAgICByb2JvdHNfdHh0ID0gIiIiXA0KK1Vz ZXItYWdlbnQ6ICoNCitEaXNhbGxvdzogL3RtcC8NCitSZXF1ZXN0LXJhdGU6IDUvwrINCisgICAg IiIiDQorICAgIGdvb2QgPSBbJy9mb28uaHRtbCddDQorICAgIHJlcXVlc3RfcmF0ZSA9IE5vbmUN CisgICAgYmFkID0gWycvdG1wLyddDQorDQorDQogY2xhc3MgQW5vdGhlckludmFsaWRSZXF1ZXN0 UmF0ZVRlc3QoQmFzZVJvYm90VGVzdCwgdW5pdHRlc3QuVGVzdENhc2UpOg0KICAgICAjIGFsc28g dGVzdCB0aGF0IEFsbG93IGFuZCBEaWFzYWxsb3cgd29ya3Mgd2VsbCB3aXRoIGVhY2ggb3RoZXIN CiAgICAgcm9ib3RzX3R4dCA9ICIiIlwNCmRpZmYgLS1naXQgYS9MaWIvdXJsbGliL3JvYm90cGFy c2VyLnB5IGIvTGliL3VybGxpYi9yb2JvdHBhcnNlci5weQ0KaW5kZXggMGMzZTVkOTI4OTA5MzU4 Li5kMjY3ZWQwMDM0NWM0OTAgMTAwNjQ0DQotLS0gYS9MaWIvdXJsbGliL3JvYm90cGFyc2VyLnB5 DQorKysgYi9MaWIvdXJsbGliL3JvYm90cGFyc2VyLnB5DQpAQCAtMTQzLDE1ICsxNDMsMTUgQEAg ZGVmIHBhcnNlKHNlbGYsIGxpbmVzKToNCiAgICAgICAgICAgICAgICAgICAgICAgICAjIGJlZm9y ZSB0cnlpbmcgdG8gY29udmVydCB0byBpbnQgd2UgbmVlZCB0byBtYWtlDQogICAgICAgICAgICAg ICAgICAgICAgICAgIyBzdXJlIHRoYXQgcm9ib3RzLnR4dCBoYXMgdmFsaWQgc3ludGF4IG90aGVy d2lzZQ0KICAgICAgICAgICAgICAgICAgICAgICAgICMgaXQgd2lsbCBjcmFzaA0KLSAgICAgICAg ICAgICAgICAgICAgICAgIGlmIGxpbmVbMV0uc3RyaXAoKS5pc2RpZ2l0KCk6DQorICAgICAgICAg ICAgICAgICAgICAgICAgaWYgbGluZVsxXS5zdHJpcCgpLmlzZGVjaW1hbCgpOg0KICAgICAgICAg ICAgICAgICAgICAgICAgICAgICBlbnRyeS5kZWxheSA9IGludChsaW5lWzFdKQ0KICAgICAgICAg ICAgICAgICAgICAgICAgIHN0YXRlID0gMg0KICAgICAgICAgICAgICAgICBlbGlmIGxpbmVbMF0g PT0gInJlcXVlc3QtcmF0ZSI6DQogICAgICAgICAgICAgICAgICAgICBpZiBzdGF0ZSAhPSAwOg0K ICAgICAgICAgICAgICAgICAgICAgICAgIG51bWJlcnMgPSBsaW5lWzFdLnNwbGl0KCcvJykNCiAg ICAgICAgICAgICAgICAgICAgICAgICAjIGNoZWNrIGlmIGFsbCB2YWx1ZXMgYXJlIHNhbmUNCi0g ICAgICAgICAgICAgICAgICAgICAgICBpZiAobGVuKG51bWJlcnMpID09IDIgYW5kIG51bWJlcnNb MF0uc3RyaXAoKS5pc2RpZ2l0KCkNCi0gICAgICAgICAgICAgICAgICAgICAgICAgICAgYW5kIG51 bWJlcnNbMV0uc3RyaXAoKS5pc2RpZ2l0KCkpOg0KKyAgICAgICAgICAgICAgICAgICAgICAgIGlm IChsZW4obnVtYmVycykgPT0gMiBhbmQgbnVtYmVyc1swXS5zdHJpcCgpLmlzZGVjaW1hbCgpDQor ICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFuZCBudW1iZXJzWzFdLnN0cmlwKCkuaXNkZWNp bWFsKCkpOg0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICBlbnRyeS5yZXFfcmF0ZSA9IFJl cXVlc3RSYXRlKGludChudW1iZXJzWzBdKSwgaW50KG51bWJlcnNbMV0pKQ0KICAgICAgICAgICAg ICAgICAgICAgICAgIHN0YXRlID0gMg0KICAgICAgICAgICAgICAgICBlbGlmIGxpbmVbMF0gPT0g InNpdGVtYXAiOg0KZGlmZiAtLWdpdCBhL01pc2MvTkVXUy5kL25leHQvTGlicmFyeS8yMDI2LTA3 LTA5LTE2LTM1LTQ3LmdoLWlzc3VlLTE1MzQwNC5kYVJrZXMucnN0IGIvTWlzYy9ORVdTLmQvbmV4 dC9MaWJyYXJ5LzIwMjYtMDctMDktMTYtMzUtNDcuZ2gtaXNzdWUtMTUzNDA0LmRhUmtlcy5yc3QN Cm5ldyBmaWxlIG1vZGUgMTAwNjQ0DQppbmRleCAwMDAwMDAwMDAwMDAwMDAuLjA3NDZkM2YxMjE0 OWQ1Mg0KLS0tIC9kZXYvbnVsbA0KKysrIGIvTWlzYy9ORVdTLmQvbmV4dC9MaWJyYXJ5LzIwMjYt MDctMDktMTYtMzUtNDcuZ2gtaXNzdWUtMTUzNDA0LmRhUmtlcy5yc3QNCkBAIC0wLDAgKzEsNCBA QA0KKzpjbGFzczpgdXJsbGliLnJvYm90cGFyc2VyLlJvYm90RmlsZVBhcnNlcmAgbm93IHNpbGVu dGx5IGlnbm9yZXMgYQ0KK2BgQ3Jhd2wtZGVsYXlgYCBvciBgYFJlcXVlc3QtcmF0ZWBgIHZhbHVl IHdyaXR0ZW4gd2l0aCBub24tZGVjaW1hbCBkaWdpdHMNCisoc3VjaCBhcyBgYFUrMDBCMiBTVVBF UlNDUklQVCBUV09gYCkgaW5zdGVhZCBvZiByYWlzaW5nIDpleGM6YFZhbHVlRXJyb3JgDQorYW5k IGFib3J0aW5nIHRoZSBwYXJzZSBvZiB0aGUgd2hvbGUgYGByb2JvdHMudHh0YGAgZmlsZS4NCg0K X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KUHl0aG9uLWNo ZWNraW5zIG1haWxpbmcgbGlzdCAtLSBweXRob24tY2hlY2tpbnNAcHl0aG9uLm9yZwpUbyB1bnN1 YnNjcmliZSBzZW5kIGFuIGVtYWlsIHRvIHB5dGhvbi1jaGVja2lucy1sZWF2ZUBweXRob24ub3Jn Cmh0dHBzOi8vbWFpbC5weXRob24ub3JnL21haWxtYW4zLy9saXN0cy9weXRob24tY2hlY2tpbnMu cHl0aG9uLm9yZwpNZW1iZXIgYWRkcmVzczogcHl0aG9uLXB5dGhvbi1jaGVja2luc0BtLmdtYW5l LW14Lm9yZwo=