Issue 1500504: Alternate RFC 3986 compliant URI parsing module (original) (raw)

Created on 2006-06-04 14:50 by ncoghlan, last changed 2022-04-11 14:56 by admin.

Messages (16)
msg50411 - (view)	Author: Alyssa Coghlan (ncoghlan) *	Date: 2006-06-04 14:50
Inspired by (and based on) Paul Jimenez's uriparse module (http://python.org/sf/1462525), urischemes tries to put a cleaner interface in front of the URI parsing engine. Most of the module works with a URI subclass of tuple that is always a 5-tuple (scheme, authority, path, query, fragment). The authority component is either None, or a URIAuthority subclass of tuple that is always a 4-tuple (user, password, host, port). The function make_uri will create a URI string from the 5 constituent components of a URI. The components do not need to be strings - if they are not strings, str() will be invoked on them (this allows the URIAuthority tuple subclass to be used transparently instead of a string for the authority component). The result is checked to ensure it is an RFC-compliant URI. The function split_uri accepts a string and returns a URI object with strings as the individual elements. Invoking str() on this object will recreate a URI string using make_uri(). The regex underlying this operation is now broken out and available as module level attributes like URI_PATTERN. The functions split_authority and make_authority are similar, only working solely on the authority component rather than the whole URI. The function parse_uri digs into the internal structure of a URI, also parsing the components. This will replace a non-empty URI authority component string with a URIAuthority tuple subclass. Depending on the scheme, it may also replace other components (e.g. for mailto links, the path is replaced with a (user, host) tuple subclass). The main parsing engine is still URIParser (much the same as Paul's), but the root of the internal parser hierarchy is now SchemeParser. This has two subclasses, URLParser and MailtoParser. The various URL flavours are now different instances of URLParser rather than subclasses. All of the actual parsers are available as module level attributes with the same name as the scheme they parse. Additionally, each parser knows the name of the scheme it is intended to parse. The parse() methods of the individual parsers are now expected to return a URI object (SchemeParser actually takes care of this). The parse() method also takes a dictionary of defaults, which can override the defaults supplied by the parser instance. The unparse() method is gone - instead, the scheme parser should ensure that all components returned are either strings or produce the right thing when __str__ is invoked (e.g. see _MailtoURIPath) The module level 'schemes' attribute is a mapping from scheme names to parsers that is automatically populated with all instances of SchemeParser that are found in the module globals() urljoin has been renamed to join_uri to match the style of the other names in the module.
msg50412 - (view)	Author: Alyssa Coghlan (ncoghlan) *	Date: 2006-06-05 13:53
Logged In: YES user_id=1038590 Updated version attached which addresses some issues raised by Mike Brown in private mail (the difference between a URI and a URI reference and some major differences between URI paths and posix paths). Also settled on split/join for the component separation and recombination operations and made the join methods all take a tuple so that join_x(split_x(uri)) round trips. Based on the terminology in the RFC, the function to combine a URI reference with a base URI is now called "resolve_uriref".
msg50413 - (view)	Author: Alyssa Coghlan (ncoghlan) *	Date: 2006-06-06 15:46
Logged In: YES user_id=1038590 Uploaded version 0.3 which passes all the RFC tests, as well as the failing 4Suite tests Mike sent me based on version 0.1 and 0.2. The last 4suite failure went away when I realised those tests expected to operate in strict mode :)
msg50414 - (view)	Author: Alyssa Coghlan (ncoghlan) *	Date: 2006-06-08 12:11
Logged In: YES user_id=1038590 Uploaded version 0.4 This version cleans up the logic in resolve_uripath a bit (use a separate loop to strip the leading dot segments, add comments explaining meaning of if statements when dealing with dot segments). It also exposes EmailPath (along with split_emailpath and join_emailpath) as public objects, rather than treating them as internal to the MailtoSchemeParser.
msg50415 - (view)	Author: Alyssa Coghlan (ncoghlan) *	Date: 2007-02-14 09:11
Removed all versions prior to 0.4
msg83920 - (view)	Author: Daniel Diniz (ajaksu2) *	Date: 2009-03-21 03:38
I'll collect open issues that would be solved by this.
msg86301 - (view)	Author: Alyssa Coghlan (ncoghlan) *	Date: 2009-04-22 15:38
The code itself is no longer the hard part here (hence the easy tag). The problem is the fact that getting something like this into the standard library is a tough sell on python-dev because it isn't really a field tested module, but once people start downloading things from PyPI, they're more likely to go for something like 4Suite rather than a mere URI parsing module. What the issue really needs is someone to champion the benefits of having this in the standard library. Now that it is available, it would also be worth looking at updating the module to use collection.named_tuple instead of creating its own variant of the same thing.
msg86308 - (view)	Author: Daniel Diniz (ajaksu2) *	Date: 2009-04-22 17:24
ISTM that gathering the issues where this would help is a good start, but I haven't had the time to do it yet.
msg86348 - (view)	Author: Senthil Kumaran (orsenthil) *	Date: 2009-04-23 02:06
I am willing to review this/work on it. But I wonder if this can be categorized as easy task. 1) Integration to Standard Library will involve compatibility with existing parsing, which will invariably involve certain tweaks (with discussions/buy-in from others). 2) There are other patches which tries to achieve this purpose; consolidation is required.
msg109963 - (view)	Author: Senthil Kumaran (orsenthil) *	Date: 2010-07-11 05:18
A new way for parsing URI. I have not reviewed it even after saying I would like to, but having the dependency issue resolved, I think it is good to look at it again, especially if it leads some helpful approaches to parsing IRI.
msg109966 - (view)	Author: Alyssa Coghlan (ncoghlan) *	Date: 2010-07-11 06:22
"accepted" is a little too strong for the current status of this :) I've removed the easy tag as well (making the case for this or something like it in the standard library it going to involve a fair bit of effort - the coding was actually the comparatively easy part).
msg120245 - (view)	Author: Éric Araujo (eric.araujo) *	Date: 2010-11-02 19:20
Is this still relevant? Can’t the improvements make it into urllib.parse?
msg120264 - (view)	Author: Alyssa Coghlan (ncoghlan) *	Date: 2010-11-02 22:09
I still like the higher level API concept, although I might not do it exactly as presented here any more. Independently of introducing a new parsing API, it would be worthwhile extracting the parsing tests from the attached module to make sure the *existing* parser can handle them all correctly.
msg120344 - (view)	Author: Éric Araujo (eric.araujo) *	Date: 2010-11-03 20:24
Sure, adding tests is a no-brainer. Regarding the module, I’m a bit reluctant. I see value in providing low-level building blocks (think OS calls) and high-level utilities for regular use, but here it seems that urllib.parse and urischemes are at the same level. I’m not opposed to the functionality itself—I would like to use a class simply named “URI” (and generally get better names, that is RFC names instead of specific inventions), have components normalization and such goodies—but I think the existing module can get fixes and improvements. I fear the confusion that could be caused by having two modules for the same task, unless you want to propose that the new module deprecate urllib.parse. Senthil, what is your opinion?
msg120349 - (view)	Author: Alyssa Coghlan (ncoghlan) *	Date: 2010-11-03 21:47
Just to be clear, even *I* don't think adding urischemes as it stands is a particularly great idea, and I wrote it. The only reason I haven't closed the issue is because I'd like to see it mined for additional tests in test_urlparse and perhaps even implementation or API enhancements in url.parse first. (The latter becomes a lot more likely if the urischemes implementation passes tests that url.parse fails) I also think, since I wrote this, the various urllib parsing methods were updated to return named tuple instances with properties, so a lot of the awkwardness of extracting partial values went away. (i.e. returning structured objects already raised the level of the urllib APIs from the "tuple-of-strings" level they used to be sitting at) I do still assert that urischemes is slightly "higher level" than the current incarnation of similar functionality in urllib.parse. Universal Resource Identifiers are more encompassing than Universal Resource Locators and Universal Resource Names, and the new APIs explicitly deal with both kinds of URI. There are subtle differences in the assumptions you're allowed to make when you may have a URN rather than a URL, so I believe the current module sometimes does the wrong thing when given one of the former. That said, it's been a long time since I've needed to remember the details, so I don't recall exactly where the current module gets URI handling wrong (or at least, did back in 2006). The intro to RFC 3986 is a good place to start in learning the differences though - Sir Tim writes good docs :)
msg202728 - (view)	Author: A.M. Kuchling (akuchling) *	Date: 2013-11-13 00:22
Here's a slightly modified version of urischeme.py that can be run under Python 3 and compares its results with urllib.parse, printing out the mismatches. The major differences seem to be 1) urischeme fills in the default port if it's not explicitly provided, e.g. http urls have the port set to 80, 2) the path is returned as '/', not the empty string, for the URL http://host, 3) urllib.parse.urljoin() doesn't get rid of ./ and ../ in URLs. 3) seems like something worth fixing in urllib.parse. The others probably present some backward-compatibility issues.