| msg68964 - (view) |
Author: kai zhu (kaizhu) |
Date: 2008-06-29 20:06 |
| this patch touches only Python/ceval.c. 1. the only existing thing it modifies is PyEval_EvalFrameEx (adds 7 extra cases for the new 3.0 opcodes, doesn't mess w/ any of the existing ones, or anything else as a matter of fact) 2. that, plus it defines (the new opcodes) #define SET_ADD 17 #define STORE_LOCALS 69 #define LOAD_BUILD_CLASS 34 // 71 in py3k #define MAKE_BYTES 35 // unused in py3k #define POP_EXCEPT 36 // 89 in py3k #define UNPACK_EX 94 #define BUILD_SET 109 // 104 in py3k and some backported vars & helper functions only 2 contiguous areas of ceval.c is patched (1. & 2. - areas are delimited by the comments '// py3k') this simple patch seems sufficient in theory to allow the interpreter to natively execute most of 3.0's language syntax (nonlocal, print, extended unpacking, ... have been tested to work) *the one exception being pep3102 - keyword-only arguments i wrote 2 small scripts which gives an interactive function 'compile_py3k' similar to __builtin__.compile. its a wrapper function which queries the byte-compiling task to a python 3.0 server via pipe io. example demonstrating pep3132 extended unpacking syntax: a,b,*c = 1,2,3,4 (note the backported 3.0 opcode used in line2 of the disassembly) Python 2.5.2 (r252:60911, Jun 27 2008, 21:19:51) [GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import py3to2 py3to2 server restarting with io: (4, 5) py3to2 server: Python 3.0b1 (r30b1:64395, Jun 24 2008, 21:53:55) py3to2 server: [GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2 py3to2 server: Type "help", "copyright", "credits" or "license" for more information. py3to2 server: >>> src = "a,b,*c = 1,2,3,4" >>> codeobject = py3to2.compile_py3k(src,"","exec") 1 0 LOAD_CONST 5 ((1, 2, 3, 4)) 3 UNPACK_EX_py3k 2 6 STORE_NAME 0 (a) 9 STORE_NAME 1 (b) 12 STORE_NAME 2 (c) 15 LOAD_CONST 4 (None) 18 RETURN_VALUE >>> exec(codeobject) >>> print a,b,c 1, 2, [3, 4] >>> u can go c http://pypi.python.org/pypi?name=py3to2&version=20080628&:action=display for more info on the script anyway, i think it would b a great boost for python 3.0 (which i think is very cool) if developers can test/develop it under the 2.x environment. this patch plus some scripts to emulated 3k extensions (bytes, bytearray, ...) can go a long way in making 3.0 available for the masses. ps. i've also attempted the reverse (forward-port 2.x opcodes to 3.0), & its a bit more complicated (namely the PRINTxxx opcodes). if there's significant interest in that, i could work on it a bit more, but my current interest is in extending 3.0 functionality to the vast 2.x software base. |
|
|
| msg68971 - (view) |
Author: Martin v. Löwis (loewis) *  |
Date: 2008-06-29 21:43 |
| The first beta release for 2.6 has been made, so no new features can be accepted here. Deferring this to 2.7. |
|
|
| msg68988 - (view) |
Author: kai zhu (kaizhu) |
Date: 2008-06-30 02:43 |
| ok |
|
|
| msg69006 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2008-06-30 14:27 |
| Some remarks: - I don't think doing a bulk backport of ceval.c is the right approach. Instead, each functionality should be considered and backported independently, if desired. - Guido already said that 3.0 should be mostly clean from compatibility-related code, so forward-porting 2.x opcodes is out of the question. There's a reason they removed in the first place :-) - Some functionalities shouldn't be backported, e.g. introducing SETUP_EXCEPT was motivated by the different semantics of exception cleanup in py3k, backporting it would probably break some 2.x code. - Compatibility between the two eval loops (2.x and py3k) is probably the tip of the iceberg. |
|
|
| msg69023 - (view) |
Author: kai zhu (kaizhu) |
Date: 2008-06-30 22:39 |
| ideally that may be true. but its quite frustrating testing/developing new py3k software when many modules/extensions we take for granted in 2.x isn't available. in the meantime, this patch serves as a very convenient stopgap for developers (even w/ its bugs), for writing py3k migration code in 2.x (& access to all its extensions) for example, its a difficult task to port a big project like numpy to py3k all @ once. but this patch could allow u to piece-wise transform it, one script @ a time, to py3k language syntax compliance. |
|
|
| msg69116 - (view) |
Author: Collin Winter (collinwinter) * |
Date: 2008-07-02 20:26 |
| I don't know why this is assigned to me. |
|
|
| msg69461 - (view) |
Author: kai zhu (kaizhu) |
Date: 2008-07-09 11:27 |
| update: these 3k language features have been tested to work in python 2.5.2 w/ the backported opcodes pep3104 Access to Names in Outer Scopes pep3105 Make print a function pep3111 Simple input built-in in Python 3000 pep3113 Removal of Tuple Parameter Unpacking pep3114 Renaming iterator.next() to .__next__() pep3115 Metaclasses in Python 3000 pep3127 Integer Literal Support and Syntax pep3129 Class Decorators pep3132 Extended Iterable Unpacking had to backport __build_class__ in bltinmodule.c to get metaclasses working. u can go to http://www-rcf.usc.edu/~kaizhu/work/py3to2/ for more info install/usage summary: 1. build python 2.5.2 w/ patched ceval.c & bltinmodule.c 2. download scripts: py3to2.py & _py3to2.py 3. ln -s python3.0 python3k 4. run python2.5 & type 'import py3to2' # it will automatically run the pep tests the script provides 3 functions similar to those in __builtin__: compile_py3k, exec_py3k, eval_py3k right now, working on backporting pep 3102 & 3107 - annotations & keyword-only arguments. apparently appending the co_kwonlyargcount attr to codeobject in 2.5.2 doesn't seem to affect the build |
|
|
| msg69611 - (view) |
Author: kai zhu (kaizhu) |
Date: 2008-07-13 16:07 |
| import/reload now works. accomplished by adding 5 lines in parse_source_module (import.c) to 1st check for the hook __builtins__.parse_source_module_py3k. the hook will automatically compile in py3k format if it finds the magic comment: "# import as py3k\n" * below is a working python 3.0 script integrated w/ numpy. also added: pep3102 Keyword-Only Arguments pep3112 Bytes literals in Python 3000 download: http://www-rcf.usc.edu/~kaizhu/work/py3to2/current/ patched files: ceval.c (unchanged from last) bltinmodule.c (unchanged from last) import.c (added 5 continuous lines to parse_source_module) there r 7 unimplemented pep's remaining: any suggested solutions? pep3107 Function Annotations (easy, just haven't gotten around yet) pep3109/3110 Exceptions in Python 3000 (hard?) pep3120 Using UTF-8 as the default source encoding pep3123 Making PyObject_HEAD conform to C (hard/who cares for 2.x?) pep3131 Supporting Non-ASCII Identifiers (looks emulable) pep3138 String representation in Python 3000 @ any rate, i think its feature complete enough to b useful in certain areas (for me its scientific computing). ################################################################################ """ numpy_py3k.py this is a py3to2 demo showing a python3.0 script being run under python 2.5 & utilizing numpy, a python2.5 extension module. add the magic comment '# import as py3k\n' to import / reload a script in py3k format interactive usage: >>> import py3to2 >>> import numpy_py3k >>> reload(numpy_py3k) commandline usage: python -c 'import py3to2; import numpy_py3k' """ # import as py3k import numpy print('pep3102 Keyword-Only Arguments') # nth order polynomial fit of multiple y data def polyfits(nth, x, *ys, rcond = None, full = False): return [numpy.polyfit(x, y, nth, rcond, full) for y in ys] fits = polyfits(2, # 2nd order fit numpy.arange(16), # x data numpy.random.rand(16), numpy.random.rand(16), # multiple y data rcond = numpy.MachAr().eps, # precision full = False, # return only coeffs ) print('fits', fits); print('#'*64) print('pep3112 Bytes literals in Python 3000') x = bytes( numpy.arange(256, dtype = numpy.int8).tostring() ) print('bytes', x); print('#'*64) print('pep3114 Renaming iterator.next() to .__next__()') x = (x for x in numpy.arange(16)) print('x.__next__()', x.__next__(), x.__next__(), x.__next__()); print('#'*64) print('pep3132 Extended Iterable Unpacking') a,b,*c = numpy.random.rand(4) print('a = %s, b = %s, c = %s'%(a,b,c)); print('#'*64) ################################################################################ |
|
|
| msg69612 - (view) |
Author: Benjamin Peterson (benjamin.peterson) * |
Date: 2008-07-13 17:01 |
| This isn't dealing with the 2to3 conversion tool. Please do not add it. |
|
|
| msg69618 - (view) |
Author: kai zhu (kaizhu) |
Date: 2008-07-13 19:01 |
| why not? it allows developers to migrate 2.x scripts one-by-one to working 3.0 conformant ones while maintaining backwards-compatibility w/ existing 2.x scripts & extension modules (eg. numpy, PIL, zope, ...) py3to2 can transparently import & mix & match 2.x & 3.0 scripts (but builtins/extensions must b 2.x - hence its a 2to3 migration tool). @ the moment, every script compilable by py3to2 should b 3.0 language conformant, or otherwise it would fail the syntax check & byte-compile stage performed by the python 3.0 slave interpreter (see Mechanism for details). |
|
|
| msg69619 - (view) |
Author: Benjamin Peterson (benjamin.peterson) * |
Date: 2008-07-13 19:05 |
| On Sun, Jul 13, 2008 at 2:03 PM, kai zhu <report@bugs.python.org> wrote: > > kai zhu <davidbranniganz@gmail.com> added the comment: > > why not? it allows developers to migrate 2.x scripts one-by-one to > working 3.0 conformant ones while maintaining backwards-compatibility w/ > existing 2.x scripts & extension modules (eg. numpy, PIL, zope, ...) > > py3to2 can transparently import & mix & match 2.x & 3.0 scripts (but > builtins/extensions must b 2.x - hence its a 2to3 migration tool). Yes, I realize that, but there is another tool called 2to3 that preforms syntax transformations on 2.x code. That's what the "2to3 conversion" tool component is for. |
|
|
| msg69624 - (view) |
Author: kai zhu (kaizhu) |
Date: 2008-07-13 20:58 |
| then the 2 can complement each other well ;) i don't c any competition between them as they have completely different objectives 2to3 -> convert 2x scripts to 3k py3to2 <- use the newly created 3k scripts in existing 2x environ u have to admit migrating to 3k is quite painful & slow w/ its lack of 2x's vast extension base. given py3to2's potential to alleviate this problem for migrating developers, don't u think that possibly merits its mention in the 2to3 category? btw, py3to2's patches minimally affects 2.x compatibility: ceval.c - backports missing 3.0 opcodes (8 in python 2.5.2) bltinmodule.c - backports missing 3.0 function __build_class__ import.c - adds a 5 line hook to function parse_source_module allowing automatic checking for and compiling of 3k scripts during import/reload i just ran 'make test' w/ a patched build of python 2.5.2 w/ following results on redhat, x86_64: 285 tests OK. 37 tests skipped: test_aepack test_al test_applesingle test_bsddb185 test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses test_dl test_gdbm test_gl test_imageop test_imgfile test_linuxaudiodev test_macfs test_macostools test_normalization test_ossaudiodev test_pep277 test_plistlib test_rgbimg test_scriptpackages test_socket_ssl test_socketserver test_startfile test_sunaudiodev test_timeout test_urllib2net test_urllibnet test_winreg test_winsound test_zipfile64 1 skip unexpected on linux2: test_gdbm not bad considering it now supports 11 of the 18 (soon to b 12 & likely more...) 3000 series pep's ^_^ |
|
|
| msg69627 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2008-07-13 21:09 |
| > don't u think that possibly merits its mention in the 2to3 category? Please trust our judgment on how to use the bug tracker. This isn't a competition for publicity; it's a tool to help organizing work among committers and other contributors. |
|
|
| msg75016 - (view) |
Author: kai zhu (kaizhu) |
Date: 2008-10-20 23:39 |
| ported to python-2.6 & is a bit more stable (written a few py3k programs w/ it). the patches have been simplified & consolidated to a single file: ceval.c u can also test out scripts generated by 2to3 by adding to them the magic line: "from __future__ import py3k_syntax" an example "python-2.6" script using "python-3.0" language syntax (& numpy if its installed) is provided @: http://www-rcf.usc.edu/~kaizhu/work/py3to2/current/example_py3k.py new features added by patch to python-2.6 (under py3k_syntax mode): pep3102 Keyword-Only Arguments pep3104 Access to Names in Outer Scopes pep3107 Function Annotations pep3111 Simple input built-in in Python 3000 pep3113 Removal of Tuple Parameter Unpacking pep3114 Renaming iterator.next() to iterator.__next__() pep3115 Metaclasses in Python 3000 pep3127 Integer Literal Support and Syntax - oct() functions differently in py3k_syntax mode pep3129 Class Decorators pep3132 Extended Iterable Unpacking pep3135 New Super misc exec becomes a function sans unicode & py3k's builtins module, the above (plus __future__) implements pretty much all of py3k's language syntax py3k_syntax mode: a special bitflag is embedded in codeobj->co_argcount, which triggers PyEval_EvalFrameEx to enable the backported opcodes (which r disabled by default) & use an alternate py3k __builtin__. codeobj->co_argcount now limited to range 0-0xffff with upper bits reserved for py3k flags codeobj->co_kwonlyargcount (limited to range 0-0xff) is embedded in upper bits of codeobj->co_argcount most of the trouble have been gettin kwonlyarg, annotations, & classes to work. there is one unrelated opcode (LOAD_ATTR_py3k) added to ceval.c for my personal use (curries attributes as a universal function), but can b safely disabled by commenting its case out of PyEvalFrameEx in ceval.c & deleting it from dict opnew_py2x in py3to2.py |
|
|
| msg75018 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2008-10-21 05:03 |
| Recommend rejecting this. I believe that it wasn't backported for a reason. Guido, care to pronounce? |
|
|
| msg75020 - (view) |
Author: kai zhu (kaizhu) |
Date: 2008-10-21 06:56 |
| hi, i'm happy i got a response on this :) anyway, can u elaborate on the "reason" y something like this was never done? yes, i kno the patch is rough right now (& will never get accepted in its current state), but its now proven to work, & gives migrating developers *something* to test their py3k_syntax code while *still* maintaining compatibility w/ python2.x extension modules. isn't that the spirit of python 2.7? in the meantime, i'm in no hurry to have anything committed. |
|
|
| msg75025 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2008-10-21 14:23 |
| This doesn't sound like it will ever be reliable. Kai Zhu, a better strategy would be for you to maintain your own experimental port of Python 2.6+3.0 based upon these patches. |
|
|
| msg75044 - (view) |
Author: kai zhu (kaizhu) |
Date: 2008-10-21 20:59 |
| k |
|
|
| msg75174 - (view) |
Author: kai zhu (kaizhu) |
Date: 2008-10-24 18:17 |
| post fyi: here's moderately complex python-3.0 script (plus curry extension) i wrote using numpy, PIL, & scipy.weave 2.6 extensions. it takes a jpeg, gif... image file & outputs it in colorized ascii art. the actual purpose is for colorized 3-d scientific plottin in text terminal (screenshots of image conversion & 3d plots in putty terminal included) usage: python -c "import py3to2; import img2txt; img2txt.img2txt.test()" python -c "import py3to2; import img2txt; img2txt.tplot3d.test()" |
|
|
| msg75185 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2008-10-24 21:58 |
| > Added file: http://bugs.python.org/file11875/mario.jpg Please don't use the bug tracker to upload unrelated code (such as Numpy code - Numpy is not part of Python core). I'm tempted to mark all these images as spam. |
|
|
