Issue 2677: Argument rules for callables do not apply when function implementation uses PyArg_ParseTuple (original) (raw)

Issue2677

Created on 2008-04-24 10:01 by garden, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (12)
msg65712 - (view)	Author: Ludovico Gardenghi (garden)	Date: 2008-04-24 10:01
(It seems strange to me that this issue hasn't been raised in the past, maybe I just failed to find it in the BTS. In that case please excuse me and please point me to the original discussion.) The Language Reference, section 5.3.4, states that, for every callable object: "[...] If keyword arguments are present, they are first converted to positional arguments, as follows. First, a list of unfilled slots is created for the formal parameters. [...] Next, for each keyword argument, the identifier is used to determine the corresponding slot (if the identifier is the same as the first formal parameter name, the first slot is used, and so on). [...]" This is not true if the function is defined using the C function PyArg_ParseTuple, and this happens a lot in the standard library. I discovered it trying to call os.open("filename", flag=os.O_RDONLY), just to make an example. In this case it seems useless to specify the keyword, but I have to write a generic "wrapping" function that takes a function name and its arguments (as keyword arguments) and call the original function after having changed the content of some of the arguments. Apart from the reason, I believe that this behavior is inconsistent with the language definition and should be fixed. I'm very new to Python, but maybe the format string of ParseTuple should be extended in order to accept also the name of the positional arguments? Or something like ParseTupleAndKeywords should be used instead? I tried only on Python 2.4 and 2.5 but I took a look at the source code of 2.6 and 3.0 and I believe the issue is still there.
msg65713 - (view)	Author: Martin v. Löwis (loewis) *	Date: 2008-04-24 10:22
I fail to see the problem. The open function really doesn't have a named parameter called flags; the positional parameters are unnamed. So there is no violation of the language reference, AFAICT. Perhaps it would be useful to point out that some parameters are available only for positional passing, as they are unnamed.
msg65714 - (view)	Author: Ludovico Gardenghi (garden)	Date: 2008-04-24 10:35
I'd believe you when you say "positional parameters are unnamed", but: - the language reference contains terms such as "first formal parameter name". This means that positional parameters *may* have a name but may also have no name? - if you define a python function as "def f(a, b, c):" you can call it using keyword arguments instead of positional (e.g. f(1, c=3, b=2)). Could you please explain me what I'm still missing? (I repeat - I met python for the first time 2 weeks ago, so it may very well be the case that I'm completely wrong)
msg65715 - (view)	Author: Martin v. Löwis (loewis) *	Date: 2008-04-24 10:50
> I'd believe you when you say "positional parameters are unnamed", but: > > - the language reference contains terms such as "first formal parameter > name". This means that positional parameters *may* have a name but may > also have no name? Correct (although that is actually my interpretation of the implementation; it's probably not a feature of the language itself). > - if you define a python function as "def f(a, b, c):" you can call it > using keyword arguments instead of positional (e.g. f(1, c=3, b=2)). Unnamed positional parameters are only available in C code. You cannot write a function with unnamed parameters in Python. > Could you please explain me what I'm still missing? (I repeat - I met > python for the first time 2 weeks ago, so it may very well be the case > that I'm completely wrong) It's just how PyArg_ParseTuple works: it doesn't receive any parameter names for the positional parameters, so it can't possibly do the matching that the language prescribes for positional parameters. Instead, PyArg_ParseTuple operates *just* by position, hence it effectively implements unnamed positional parameters. You are not completely wrong. It's just that this detail is something most people recognize at some point and accept as a fact, regardless of what the language specification says (and, as I claim, that text isn't incorrect - or the implementation isn't incorrect -- it's just underspecified, failing to mention a detail specific to CPython)
msg65716 - (view)	Author: Ludovico Gardenghi (garden)	Date: 2008-04-24 11:08
> You are not completely wrong. It's just that this detail is something > most people recognize at some point and accept as a fact, regardless > of what the language specification says (and, as I claim, that text > isn't incorrect - or the implementation isn't incorrect -- it's just > underspecified, failing to mention a detail specific to CPython) Ok. I think I'll end up accepting it as a fact, too :) and work around the issue. IMHO it would be perfectly acceptable to say "if you use CPython and extend python with some C functions you must expect this behavior", but it's slightly less acceptable that different modules from the standard library have different behaviors (depending on which language has been used to implement them): - open(mode='r', name='filename') works - os.open(flag=os.O_RDONLY, filename='filename') does not work - calendar.weekday(2008, day=24, month=4) works - math.fmod(x=10, y=3) does not work - ... From the point of view of someone who writes python code there should be no difference between the behavior of these calls, as long as they are included in the standard python library. IMHO, again. Maybe yes, the easier but probably harmless solution is to change the documentation and point out that "in general, you can't". Maybe this somehow leans towards promoting a bug to the rank of feature? ;-)
msg65717 - (view)	Author: Martin v. Löwis (loewis) *	Date: 2008-04-24 11:14
> Maybe yes, the easier but probably harmless solution is to change the > documentation and point out that "in general, you can't". Maybe this > somehow leans towards promoting a bug to the rank of feature? ;-) The language spec is stuck between saying what the abstract Python language is supposed to do, and describing what CPython precisely does. You shouldn't use keyword arguments to pass non-optional positional arguments, IMO, but the text describes precisely what happens if you do - for Python functions. The more vague specification then shouldn't say "you can't" (because that would indicate that you get an exception when you try), but "it's unspecified", then going on to say what CPython happens to do in some release.
msg65718 - (view)	Author: Ludovico Gardenghi (garden)	Date: 2008-04-24 11:29
At present, "unspecified" is surely better than "you can't", that's a good point. I understand the difficulties of balancing the reference between the abstract definition and the actual implementation. But I still believe that this should not be an unspecified behavior, either in one direction or another. At least somewhere in the future. Anyway, thanks for the explanation :)
msg65719 - (view)	Author: Martin v. Löwis (loewis) *	Date: 2008-04-24 11:42
Making it a documentation issue; I don't think the implementation should change. Georg, if you don't see the need for action, feel free to close it.
msg65820 - (view)	Author: Terry J. Reedy (terry.reedy) *	Date: 2008-04-25 22:20
I consider the difference between builtin and def'ed functions to be something of an implementation wart -- one that I would like to see someday removed if sensibly possible. How is a beginner to know that the parameter names used in the docs and help() responses are not really parameter names? In the meanwhile, I think something like the following in the doc would help: "(Note: an implementation may provide builtin functions whose positional parameters do not have names, even if they are 'named' for the purpose of documentation, and which therefore cannot be supplied by keyword.)" Also in the meanwhile, the OP can def-wrap builtins import builtins def abs(number): return builtins.abs(number) # but some like int require more care with its no-default option
msg65853 - (view)	Author: Georg Brandl (georg.brandl) *	Date: 2008-04-26 18:34
I'd love to add to the documentation, but I can't seem to find a proper location - except the Tutorial?
msg65858 - (view)	Author: Terry J. Reedy (terry.reedy) *	Date: 2008-04-27 00:09
Ludovico quoted from LangReg 5.3.4 of old style doc. The same paragraph about keyword args is under Expressions/Primaries/Calls in the 2.6/3.0 docs
msg65882 - (view)	Author: Georg Brandl (georg.brandl) *	Date: 2008-04-27 09:40
Ah sorry, I should really read the OP next time. Added a note in r62521.