[Python-ideas] New 3.x restriction on number of keyword arguments (original) (raw)

M.-A. Lemburg mal at egenix.com
Sat Oct 23 00:36:30 CEST 2010

Cesare Di Mauro wrote:

2010/10/22 M.-A. Lemburg <mal at egenix.com>

Cesare Di Mauro wrote: I think that having more than 255 arguments for a function call is a very rare case for which a workaround (may be passing a tuple/list or a dictionary) can be a better solution than having to introduce a brand new opcode to handle it.

It's certainly rare when writing applications by hand, but such limits can be reached with code generators wrapping external resources such as database query rows, spreadsheet rows, sensor data input, etc. We've had such a limit before (number of lines in a module) and that was raised for the same reason.

Changing the current opcode(s) is a very bad idea, since common cases will slow down. I'm sure there are ways to avoid that, e.g. by using EXTENDEDARG for such cases. -- Marc-Andre Lemburg eGenix.com I've patched Python 3.2 alpha 3 with a rough solution using EXTENDEDARG for CALLFUNCTION* opcodes, raising the arguments and keywords limits to 65535 maximum. I hope it'll be enough. :)

Sure, we don't have to raise it to 2**64 :-) Looks like a pretty simple fix, indeed.

I wish we could get rid off all the byte shifting and div'ery use in the byte compiler - I'm pretty sure that such operations are rather slow nowadays compared to working with 16-bit or 32-bit integers and dropping the notion of taking the word "byte" in byte code literally.

In ast.c:

astforarguments: if (nposargs > 65535 || nkwonlyargs > 65535) { asterror(n, "more than 65535 arguments"); return NULL; } astforcall: if (nargs + ngens > 65535 || nkeywords > 65535) { asterror(n, "more than 65535 arguments"); return NULL; }

In compile.c: opcodestackeffect: #define NARGS(o) (((o) & 0xff) + ((o) >> 8 & 0xff00) + 2*(((o) >> 8 & 0xff) + ((o) >> 16 & 0xff00))) case CALLFUNCTION: return -NARGS(oparg); case CALLFUNCTIONVAR: case CALLFUNCTIONKW: return -NARGS(oparg)-1; case CALLFUNCTIONVARKW: return -NARGS(oparg)-2; #undef NARGS #define NARGS(o) (((o) % 256) + 2*(((o) / 256) % 256)) case MAKEFUNCTION: return -NARGS(oparg) - ((oparg >> 16) & 0xffff); case MAKECLOSURE: return -1 - NARGS(oparg) - ((oparg >> 16) & 0xffff); #undef NARGS compilercallhelper: int len; int code = 0; len = asdlseqLEN(args) + n; n = len & 0xff | (len & 0xff00) << 8;_ _VISITSEQ(c, expr, args);_ _if (keywords) {_ _VISITSEQ(c, keyword, keywords);_ _len = asdlseqLEN(keywords);_ _n |= (len & 0xff | (len & 0xff00) << 8) << 8;_ _}_ _In ceval.c:_ _PyEvalEvalFrameEx:_ _TARGETWITHIMPL(CALLFUNCTIONVAR, callfunctionvarkw)_ _TARGETWITHIMPL(CALLFUNCTIONKW, callfunctionvarkw)_ _TARGET(CALLFUNCTIONVARKW)_ _callfunctionvarkw:_ _{_ _int na = oparg & 0xff | oparg >> 8 & 0xff00; int nk = (oparg & 0xff00 | oparg >> 8 & 0xff0000) >> 8; callfunction: int na = oparg & 0xff | oparg >> 8 & 0xff00; int nk = (oparg & 0xff00 | oparg >> 8 & 0xff0000) >> 8; A quick example: s = '''def f(*Args, **Keywords): print('Got', len(Args), 'arguments and', len(Keywords), 'keywords') def g(): f(''' + ', '.join(str(i) for i in range(500)) + ', ' + ', '.join('k{} = {}'.format(i, i) for i in range(500)) + ''') g() ''' c = compile(s, '', 'exec') eval(c) from dis import dis dis(g) The output is: Got 500 arguments and 500 keywords 5 0 LOADGLOBAL 0 (f) 3 LOADCONST 1 (0) 6 LOADCONST 2 (1) [...] 1497 LOADCONST 499 (498) 1500 LOADCONST 500 (499) 1503 LOADCONST 501 ('k0') 1506 LOADCONST 1 (0) 1509 LOADCONST 502 ('k1') 1512 LOADCONST 2 (1) [...] 4491 LOADCONST 999 ('k498') 4494 LOADCONST 499 (498) 4497 LOADCONST 1000 ('k499') 4500 LOADCONST 500 (499) 4503 EXTENDEDARG 257 4506 CALLFUNCTION 16905460 4509 POPTOP 4510 LOADCONST 0 (None) 4513 RETURNVALUE The dis module seems to have some problem displaying the correct extended value, but I have no time now to check and fix it. Anyway, I'm still unconvinced of the need to raise the function def/call limits.

It may seem strange to have functions, methods or object constructors with more than 255 parameters, but as I said: when using code generators, the generators don't care whether they use 100 or 300 parameters. Even if just 10 parameters are actually used later on. However, the user will care a lot if the generators fail due such limits and then become unusable.

As example, take a database query method that exposes 3-4 parameters for each query field. In more complex database schemas that you find in e.g. data warehouse applications, it is not uncommon to have 100+ query fields or columns in a data table.

With the current limit in function/call argument counts, such a model could not be mapped directly to Python. Instead, you'd have to turn to solutions based on other data structures that are not automatically checked by Python when calling methods/functions.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Oct 22 2010)

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

More information about the Python-ideas mailing list