| msg210306 - (view) |
Author: Gustavo J. A. M. Carneiro (gustavo) * |
Date: 2014-02-05 11:52 |
| If you try the attached program, you will find that for every iteration the uuid.uuid4() call generates objects that contain reference cycles and need the help of the garbage collector. This is not nice. If I make the ctypes module not able to import, then no garbage is generated. This problem appears in 2.7, 3.3, and 3.4, at least. |
|
|
| msg210316 - (view) |
Author: R. David Murray (r.david.murray) *  |
Date: 2014-02-05 14:10 |
| I'm not sure that this is a real problem, but have you identified what the cycle is? |
|
|
| msg210320 - (view) |
Author: Gustavo J. A. M. Carneiro (gustavo) * |
Date: 2014-02-05 14:28 |
| Well, this isn't a big problem, but I have an application that needs to run with the GC disabled, since it causes pauses of a couple of seconds each time a full collection runs (we have a few million objects allocated). I will run the GC only once every 3 hours. So it would be nice if this uuid call didn't generate cycles. No, I didn't identify the cycle. All I know is that the garbage below is produced. If the ctypes module is unavailable, uuid still works but doesn't generate the garbage. >>> gc.garbage [(<type '_ctypes.Array'>,), {'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '__module__': 'ctypes', '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, '__doc__': None, 'value': <attribute 'value' of 'c_char_Array_16' objects>}, <class 'ctypes.c_char_Array_16'>, <attribute '__dict__' of 'c_char_Array_16' objects>, <attribute '__weakref__' of 'c_char_Array_16' objects>, (<class 'ctypes.c_char_Array_16'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>, (<type '_ctypes.Array'>,), {'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '__module__': 'ctypes', '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, '__doc__': None, 'value': <attribute 'value' of 'c_char_Array_16' objects>}, <class 'ctypes.c_char_Array_16'>, <attribute '__dict__' of 'c_char_Array_16' objects>, <attribute '__weakref__' of 'c_char_Array_16' objects>, (<class 'ctypes.c_char_Array_16'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>, (<type '_ctypes.Array'>,), {'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '__module__': 'ctypes', '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, '__doc__': None, 'value': <attribute 'value' of 'c_char_Array_16' objects>}, <class 'ctypes.c_char_Array_16'>, <attribute '__dict__' of 'c_char_Array_16' objects>, <attribute '__weakref__' of 'c_char_Array_16' objects>, (<class 'ctypes.c_char_Array_16'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>, (<type '_ctypes.Array'>,), {'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '__module__': 'ctypes', '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, '__doc__': None, 'value': <attribute 'value' of 'c_char_Array_16' objects>}, <class 'ctypes.c_char_Array_16'>, <attribute '__dict__' of 'c_char_Array_16' objects>, <attribute '__weakref__' of 'c_char_Array_16' objects>, (<class 'ctypes.c_char_Array_16'>, <type '_ctypes.Array'>, <type '_ctypes._CData'>, <type 'object'>), <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>] |
|
|
| msg210322 - (view) |
Author: Gustavo J. A. M. Carneiro (gustavo) * |
Date: 2014-02-05 14:58 |
| I have narrowed it down to one line of code: ctypes.create_string_buffer(16) That is enough to create 7 objects that have reference cycles. [<class 'ctypes.c_char_Array_16'>, {'__module__': 'ctypes', '__doc__': None, '__weakref__': <attribute '__weakref__' of 'c_char_Array_16' objects>, 'raw': <attribute 'raw' of 'c_char_Array_16' objects>, '_length_': 16, '_type_': <class 'ctypes.c_char'>, 'value': <attribute 'value' of 'c_char_Array_16' objects>, '__dict__': <attribute '__dict__' of 'c_char_Array_16' objects>}, (<class 'ctypes.c_char_Array_16'>, <class '_ctypes.Array'>, <class '_ctypes._CData'>, <class 'object'>), <attribute '__weakref__' of 'c_char_Array_16' objects>, <attribute 'raw' of 'c_char_Array_16' objects>, <attribute 'value' of 'c_char_Array_16' objects>, <attribute '__dict__' of 'c_char_Array_16' objects>] So maybe the bug is in ctypes itself, not the uuid module. |
|
|
| msg210328 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2014-02-05 15:44 |
| Yes, I was pretty sure it was in cytpes, from looking at the UUID code. |
|
|
| msg210331 - (view) |
Author: Gustavo J. A. M. Carneiro (gustavo) * |
Date: 2014-02-05 16:16 |
| Regardless, if you don't mind, take this patch for Python 3.5 to avoid ctypes, at least in the Linux case (I don't have Windows to test). Creating a proper extension module is safer and really not that hard... |
|
|
| msg210335 - (view) |
Author: R. David Murray (r.david.murray) * |
Date: 2014-02-05 16:57 |
| Thanks. This looks like a good idea, but I'll leave it to someone with more experience with this module to review it. Let's change this issue to an enhancement request for uuid instead. If someone wants to work on the cycle-in-ctypes problem they can open a new issue. |
|
|
| msg210346 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2014-02-05 20:08 |
| See also . |
|
|
| msg251509 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2015-09-24 11:38 |
| In the Python stdlib, we try to avoid ctypes to use instead a C extension module. So I like the idea of a new optional _uuid module. I reviewed the attached patch. |
|
|
| msg251510 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2015-09-24 11:38 |
| @Gustavo: Can you please take my remarks in account and rebase your change on the default branch? |
|
|
| msg253493 - (view) |
Author: Gustavo J. A. M. Carneiro (gustavo) * |
Date: 2015-10-26 19:14 |
| This patch fixes the Mac OS X issue @haypo pointed out. |
|
|
| msg253494 - (view) |
Author: Gustavo J. A. M. Carneiro (gustavo) * |
Date: 2015-10-26 19:25 |
| One issue of note is regarding generate_time(). Originally I found ctypes bindings for this function, so I wrapped it as well in the extension module. However, it doesn't appear to be used... |
|
|
| msg253511 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2015-10-27 02:35 |
| New review (question for Windows). |
|
|
| msg253598 - (view) |
Author: Gustavo J. A. M. Carneiro (gustavo) * |
Date: 2015-10-28 13:36 |
| New patch that: 1. adds assert(sizeof(uuid_t) == 16); to the extension module; 2. fixes the code path when ctypes has to be used instead of the extension module (needed a bit more refactoring, apologies if it makes the diff harder to read); 3. Adjusts the uuid module tests to account for the possibility of ctypes not being imported. |
|
|
| msg303219 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-09-28 11:18 |
| https://github.com/python/cpython/pull/3796 updates the patch for 3.7 and improves on it a bit by making initialization lazy. |
|
|
| msg303233 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-09-28 13:29 |
| There are too many uuid open issues proposing similar changes. I mark this issue as a duplicate of bpo-11063 to avoid splitted discussions. Please continue the discussion there! |
|
|
| msg303280 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-09-28 21:03 |
| New changeset a106aec2ed6ba171838ca7e6ba43c4e722bbecd1 by Antoine Pitrou in branch 'master': bpo-11063, bpo-20519: avoid ctypes and improve import time for uuid (#3796) https://github.com/python/cpython/commit/a106aec2ed6ba171838ca7e6ba43c4e722bbecd1 |
|
|