Not using a fake user-agent as the default one. Making user-agent configurable. by kazet · Pull Request #18 · rfc-st/humble (original) (raw)

Hi!,

Thanks for your contribution; yes, the option to send a customized User-Agent is something I had been thinking about for some time: in the tests I have done these years many of the requested URLs expect (at least) that this header is sent with a 'recognizable' content to return a valid response (ex. combination of OS + Browser + Engine, etc) and to prevent the request from being associated with a web crawler, bot or scanner.

In fact, I think that sending only the User-Agent in an HTTP request, even if it is well-formed and 'real', is clearly insufficient for some of these URLs ... but that's another issue that I will also improve when I have time :).

By the way: the User-Agent that includes 'humble' by default ('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36') is real :), you can check it by parsing it in https://explore.whatismybrowser.com/useragents/parse/

I have to check calmly your PR, as I am convinced that if the User-Agent is simply the literal 'humble' there will be problems with some URLs (being in this case a minimal value unrelated to any OS/Browser).

Give me a few days, please, to think about this (I also want to include some minimal validation of the customized User-Agent) and we'll talk.

Thank you.

P.S: good luck with Artemis! :).

Best regards,