bpo-29729: uuid.UUID now accepts bytes-like object by vstinner · Pull Request #3801 · python/cpython (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation24 Commits8 Checks0 Files changed

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

uuid.UUID doesn't require the 'bytes' argument to be an instance of
bytes: accept bytes-like types, bytearray and memoryview for example.

https://bugs.python.org/issue29729

uuid.UUID doesn't require the 'bytes' argument to be an instance of bytes: accept bytes-like types, bytearray and memoryview for example.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a "versionchanged" directive in the module documentation.

Hmm, memoryview is not accepted as bytes_le. And the length check works correctly only with memoryview with byte format.

• Accept also bytes-like for bytes_le: convert memoryview to bytes to use slices. Check also the length after the conversion to integer to raise a TypeError even if the length is not 16.
• Fix unit tests: pass bytes to 'bytes' and 'bytes_le' parameters
• Add more tests on Unicode passed to bytes and bytes_le
• Document UUID changes

Hmm, memoryview is not accepted as bytes_le

Oh, I didn't look at bytes_le. I also modified bytes_le to accept bytes-like objects.

And the length check works correctly only with memoryview with byte format.

I took care of such issues in my updated PR.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add int.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why bytes_le should accept an int? Do you have an example please?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should not! But accepts. Add a test for bytes_le=16. 😉

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... and 'bytes_le'

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, fixed.

A ValueError is raised if 'bytes_le' is a str of len != 16. To solve this duplicate int.from_bytes() for bytes_le instead of reusing the code for bytes.

A ValueError is raised if 'bytes_le' is a str of len != 16.

I added an unit tests to ensure that a TypeError is raised if you pass a str of any length to bytes or bytes_le.

Please don't merge my PR until PR #3796 is merged. IMHO PR #3796 is more important :-)

I added an unit tests to ensure that a TypeError is raised if you pass a str of any length to bytes or bytes_le.

Ah, yes, a bytes constructor raises an error.

But still there is a problem with memoryview with non-byte format. bytes_le=memoryview(b'\0'*16).cast('I') works, but bytes=memoryview(b'\0'*16).cast('I') is rejected and bytes=memoryview(b'\0'*64).cast('I') is accepted by mistake.

PR #3796 has been merged. I merged master into my branch.

But still there is a problem with memoryview with non-byte format. memoryview(b'\0'*64).cast('I') is accepted by mistake.

I don't think that we should overengineer the code here. We are consenting adults here. You can also find a value to skip input validations. It's really hard to write the perfect validation. If you can a random memoryview, yeah, maybe we get a valid UUID object whereas you shouldn't. Well, I don't think that we should detect this corner case. Why would you do that on purpose?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may not be worth supporting every kind of invalid input (unless you have to handle untrusted input). But it is important to implement what the documentation promises (or fix the documentation). As of rev a169e89, bytes has to also support len.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By above in the comment, do you mean this check, which is actually below the comment? :)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the comment.

@vadmium: "It may not be worth supporting every kind of invalid input (unless you have to handle untrusted input). But it is important to implement what the documentation promises (or fix the documentation). As of rev a169e89, bytes has to also support len."

I'm sorry, but I don't understand what you are expecting from me. What do you mean by "bytes has to also support len"? Do you expect a nicer error message if len() fails?

For your code to work, the object passed as the bytes argument has to implement __len__ returning 16. But your documentation promises any bytes-like object should work (as long as it is a string of 16 bytes). Since the glossary definition of bytes-like objects has no requirements about implementing __len__, I would expect len to not be used on the argument. You could change the code for bytes to make a copy like you do for bytes_le:

bytes = bytes_(bytes) # Convert bytes-like to bytes object if len(bytes) != 16 . . .

Or you could use memoryview:

with memoryview(bytes) as view: if view.nbytes != 16 . . .

Or you could change the documentation:

bytes_le now accepts any bytes-like object, and bytes now accepts any bytes-like object for which len returns 16.

Or you could change the documentation:

UUID constructor documentation is explicit:

   Create a UUID from either a string of 32 hexadecimal digits, a string of 16
   bytes as the *bytes* argument, a string of 16 bytes in little-endian order as
   the *bytes_le* argument, ...

I guess that your remarks was on my "versionchanged" markup and the NEWS entry. I added "of 16 bytes" there.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

>>> a = array.array('i', [-1]*4)
>>> x = int.from_bytes(a, byteorder='big')
>>> len(a)
4
>>> x.bit_length()/8
16.0

And conversely:

>>> a = array.array('i', [-1]*16)
>>> x = int.from_bytes(a, byteorder='big')
>>> len(a)
16
>>> x.bit_length()/8
64.0

Yes I was talking about the versionchanged note, which now says "any bytes-like objects of 16 bytes". But I don't think this is clear enough. Antoine and Serhiy already gave examples of bytes-like objects containing exactly 16 bytes where len returns a different number.

Since you already copy bytes_le into a true bytes object, why not parallel this for the bytes parameter?

if bytes is not None: # Check for int so that bytes=16 triggers an exception below if not isinstance(bytes, (bytes_, bytearray, int_)): bytes = bytes_(bytes) # Copy any bytes-like object if len(bytes) != 16: raise ValueError('bytes is not a 16-byte string') int = int_.from_bytes(bytes, 'big')

I decided to work on https://bugs.python.org/issue29729 because I understand that the proposed change was just to remove an assertion. That's all.

It seems like you expect much more changes, much more complex code. It's already my 6th version of my pull request.

Honestly, I don't care much of the uuid module. I don't think that anyone ever tried to call UUID(bytes=value) with a type different than bytes (ok, apart the bug reporter ;-)). If I would really need to pass non-bytes, having to cast manually with bytes(value) wouldn't burn my hands.

For all these reasons, I abandon my change. Feel free to reuse my code if someone wants to pursue the effort on UUID enhancement ;-)

The problem with adding support of bytes-like objects is that in different cases this term has different meaning. The definition in the glossary is too wide, in many particular cases we need the support of not just the buffer protocol, but len(), indexing, slicing, iterating, startswith() or even lower().

I think we need different terms for different levels of "bytesness".

I think that we should replace the assertion with a regular if and raise a proper TypeError, rather than trying to support "bytes-like" objects. It's fine to require bytes on UUID. UUID is not a common type, and passing bytes to UUID is even less common (IMHO).