Issue 26519: Cython doesn't work anymore on Python 3.6 (original) (raw)
Created on 2016-03-09 16:01 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Messages (9)
Author: STINNER Victor (vstinner) * 
Date: 2016-03-09 16:01
I tried to run numpy test suite on Python 3.6 compiled in debug mode. I got an assertion error. Then I wanted to try the Git version of numpy, but the "Cythonizing sources" step of the numpy installer fails with:
Traceback (most recent call last): File "bin/cython", line 9, in load_entry_point('Cython==0.23.4', 'console_scripts', 'cython')() (...) File "lib/python3.6/site-packages/Cython/Compiler/Nodes.py", line 6585, in generate_execution_code self.body.generate_execution_code(code) File "lib/python3.6/site-packages/Cython/Compiler/Nodes.py", line 7026, in generate_execution_code fresh_finally_clause().generate_execution_code(code) File "lib/python3.6/site-packages/Cython/Compiler/Nodes.py", line 7013, in fresh_finally_clause node_copy = copy.deepcopy(node) File "/home/haypo/prog/python/default/Lib/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) (...) File "/home/haypo/prog/python/default/Lib/copy.py", line 314, in _reconstruct item = deepcopy(item, memo) File "/home/haypo/prog/python/default/Lib/copy.py", line 174, in deepcopy rv = reductor(4) TypeError: can't pickle Argument objects
Argument class indirectly comes from Cython.Compiler.Nodes.ExprStatNode.
Command to reproduce the bug:
tar -xf mtrand.tar.gz # download mtrand.tar.gz attached to this issue python -m venv ENV ENV/bin/python -m pip install cython ENV/bin/python ENV/bin/cython --fast-fail -o mtrand.c mtrand.pyx
It looks like cython behaviour failed after this change:
changeset: 99677:b8d108a2a38e parent: 99675:80d1faa9735d parent: 99676:0cd2de69fb66 user: Serhiy Storchaka <storchaka@gmail.com> date: Fri Dec 25 21:05:35 2015 +0200 files: Lib/test/test_csv.py Misc/NEWS Objects/typeobject.c description: Issue #22995: Instances of extension types with a state that aren't subclasses of list or dict and haven't implemented any pickle-related methods (reduce, reduce_ex, getnewargs, getnewargs_ex, or getstate), can no longer be pickled. Including memoryview.
Sorry, I don't understand yet if it's a regression in Python 3.6 or a real bug in Cython.
Author: STINNER Victor (vstinner) *
Date: 2016-03-09 16:06
Oops, my commands to reproduce the bug were wrong.
Commands to reproduce the bug:
tar -xf mtrand.tar.gz # download mtrand.tar.gz attached to this issue
./python -m venv ENV ENV/bin/python -m pip install cython ENV/bin/python ENV/bin/cython --fast-fail -o mtrand/mtrand.c mtrand/mtrand.pyx
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2016-03-09 16:31
May be this is the same issue as reported in . This was a bug in Cython and it is fixed now.
Author: STINNER Victor (vstinner) *
Date: 2016-03-09 16:39
"May be this is the same issue as reported in . This was a bug in Cython and it is fixed now."
Oh, I wasn't aware of the issue #22995. I will read it.
The bug isn't fixed yet in the latest release of Cython (0.23.4, released at 2015-10-10). I guess that it's fixed in the development version.
Author: STINNER Victor (vstinner) *
Date: 2016-03-09 16:51
This was a bug in Cython and it is fixed now.
If the change is deliberate, can you please mention it in "What's New in Python 3.6" doc? Especially in the "Porting to Python 3.6" section: https://docs.python.org/dev/whatsnew/3.6.html#porting-to-python-3-6
Can you suggest a fix in the doc? I guess that a method should be added: reduce(), reduce_ex() or getstate()?
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2016-03-09 17:54
Cython 0.23.4 was released at 2015-10-10. The fix was applied 2016-01-16. Try to use Cython from trunk. May be this is not exactly the same bug and Cython needs other fix.
The change is deliberate. It helps to find existing bugs.
Except trivial case when your extension class has no any fields in additional to pickleable parent class, the class needs to provide some combinations of getstate, setstate, getnewargs/getnewargs_ex and reduce/reduce_ex to be pickleable. This was said in PEP 307. Previously you silently produced invalid pickle and got an exception or invalid object when unpickle it. Now you more likely get an exception at pickling time.
Author: Serhiy Storchaka (serhiy.storchaka) *
Date: 2016-03-12 06:28
Stefan, could you please check that this issue is fixed and there are no other cases of pickling or copying unpickleable classes?
Author: Stefan Behnel (scoder) *
Date: 2016-03-17 12:29
Our CI build server says it's all fine. The fix will eventually be released, certainly before Py3.6 comes out.
Author: STINNER Victor (vstinner) *
Date: 2016-03-17 16:31
Our CI build server says it's all fine. The fix will eventually be released, certainly before Py3.6 comes out.
Ok, fine :-)
History
Date
User
Action
Args
2022-04-11 14:58:28
admin
set
github: 70706
2016-03-17 16:31:34
vstinner
set
status: open -> closed
resolution: third party
messages: +
2016-03-17 12:29:28
scoder
set
messages: +
2016-03-12 06:28:24
serhiy.storchaka
set
messages: +
2016-03-09 17:54:51
serhiy.storchaka
set
messages: +
2016-03-09 16:51:05
vstinner
set
messages: +
2016-03-09 16:39:59
vstinner
set
nosy:scoder, vstinner, serhiy.storchaka
messages: +
2016-03-09 16:31:50
serhiy.storchaka
set
nosy: + scoder
messages: +
2016-03-09 16:06:56
vstinner
set
messages: +
2016-03-09 16:01:57
vstinner
create