Issue 1683368: object.init shouldn't allow args/kwds (original) (raw)
Created on 2007-03-19 03:32 by blakeross, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Messages (38)
Author: Blake Ross (blakeross)
Date: 2007-03-19 03:32
object.init currently allows any amount of args and keywords even though they're ignored. This is inconsistent with other built-ins, like list, that are stricter about what they'll accept. It's also inconsistent with object.new, which does throw if any are provided (if the default init is to be used).
To reproduce:
object.init(object(), foo, bar=3)
This is a slight irritation when using cooperative super calling. I'd like each class' init to cherry-pick keyword params it accepts and pass the remaining ones up the chain, but right now I can't rely on object.init to throw if there are remaining keywords by that point.
Author: Georg Brandl (georg.brandl) * 
Date: 2007-03-19 08:56
I don't really understand either why object_new() checks the arguments, not object_init():
""" static int object_init(PyObject *self, PyObject *args, PyObject *kwds) { return 0; }
/* If we don't have a tp_new for a new-style class, new will use this one. Therefore this should take no arguments/keywords. However, this new may also be inherited by objects that define a tp_init but no tp_new. These objects WILL pass argumets to tp_new, because it gets the same args as tp_init. So only allow arguments if we aren't using the default init, in which case we expect init to handle argument parsing. */ static PyObject * object_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { if (type->tp_init == object_init && (PyTuple_GET_SIZE(args) || (kwds && PyDict_Check(kwds) && PyDict_Size(kwds)))) { PyErr_SetString(PyExc_TypeError, "default new takes no parameters"); return NULL; } return type->tp_alloc(type, 0); } """
Author: Guido van Rossum (gvanrossum) *
Date: 2007-03-19 20:01
I'll try to explain why I did it this way. I was considering the single inheritance case implementing an Immutable object, which overrides new but has no need to override init (since it's too late to do anything in init for an Immutable object). Since the init still gets called it would be annoying to have to override it just to make the error go away if there was a check in init. The other case is overriding init without overriding new, which is the most common way of doing Mutable objects; here you wouldn't want new to complain about extra args. So the only time when you'd want complaints is if both new and init are the defaults, in which case it doesn't really matter whether you implement this in init or in new, so I arbitrarily chose new.
I wasn't thinking of your use case at the time though (cooperative super calls to init, which still isn't something I engage in on a day-to-day basis). I wonder if the right thing to do wouldn't be to implement the same check both in init and in new.
Am I makign sense?
Author: Blake Ross (blakeross)
Date: 2007-03-19 22:45
Makes sense. I don't think we can ever be completely correct here since we're inferring intent from the presence of init/new that's liable to be wrong in some cases, but it's likely correct often enough that it's worth doing.
If I understand correctly, we want to be more forgiving iff one of the two methods is used, so it seems like we should be complaining if both are used or if neither is used. After all, I could add a new to my coop use case and I'd still want object to complain. If that's the case, both object_new and object_init should be complaining if ((tp->tp_new == object_new && tp->tp_init == object_init) || (tp->tp_new != object_new && tp->tp_init != object_init)).
Of course, for the paranoid, there's always the risk that new will modify these class functions and change the outcome :) For instance, if a class had a new and no init and its new changed new back to object.new, object_init on that run would be fooled into thinking it's using the defaults for both and would complain. I think this could only be fixed in type_call, which is rather ugly...but then, this is a special case of the "call init after new" behavior, and we're trying to solve it within the methods themselves. Perhaps this last point is academic enough to be ignored...I don't know why anyone would do this, although the language makes it possible.
Author: Guido van Rossum (gvanrossum) *
Date: 2007-03-19 23:31
Attached is a patch that implements this proposal, adding copious commentary. It doesn't seem to break anything in the test suite.
I wonder if we should even make the check more rigid: check the argument list if either the current method is overridden or the other one is not overridden. This would make super calls check the arguments even if the other method is overridden. What do you think? File Added: new_init.patch
Author: Guido van Rossum (gvanrossum) *
Date: 2007-03-19 23:35
This smells enough like a new feature that it couldn't go into 2.5.
Author: Blake Ross (blakeross)
Date: 2007-03-20 01:27
I think making the check more rigid is a good idea, since this should throw:
class a(object): def init(self, foo): super(a, self).init(foo) def new(cls, foo): return object.new(cls) a(1)
(minor typo in the patch: "solution it" -> "solution is")
Author: Guido van Rossum (gvanrossum) *
Date: 2007-03-20 20:54
Here's a stricter version. Unfortunately it breaks a couple of standard modules; this is a confirmation of my doubts whether the style of cooperative super calling of init that you use is really the most common or "best practice".
So far I have only fixed string.py (which would otherwise prevent extensions from being built); I haven't looked into why the other tests fail: test_array, test_cpickle, test_descr, test_pickle (and maybe more?).
My conclusion: this would probably break too much code to be worth it. So I'll have to revert to the previous version. But anyway, here it is for your perusal. File Added: new_init_strict.patch
Author: Guido van Rossum (gvanrossum) *
Date: 2007-03-20 21:24
I should mention that if we can't get the strict version of this in 2.6, we should be able to get it into 3.0.
Author: Blake Ross (blakeross)
Date: 2007-03-21 22:03
Looks good. I skimmed briefly the tests you mentioned. The issue with test_array appears to be exactly the kind of bug this is intended to identify: it calls array.init(...), but array doesn't have its own initializer, so object's is used.
I'd guess that the others are failing due to whatever the problem with pickling is (test_descr uses pickling). I haven't looked into that yet.
I'm sure cooperative super calling of init isn't all that common (it seems like the mechanism itself isn't used much yet, and may not be until it's done via keyword) but it doesn't seem like such a bad practice, especially when mixins are in the picture. There doesn't seem to be a great alternative.
Author: Guido van Rossum (gvanrossum) *
Date: 2007-03-21 22:42
Well, but since it's been like this for a long time, I don't want to gratuitously break code. At least not in 2.6. So I'm rejecting the stricter patch for 2.6. (However, if you want to submit patches that would fix these breakages anyway, be my guest.)
Author: Blake Ross (blakeross)
Date: 2007-03-21 22:48
Holding the strict version for 3 makes sense to me. Let me know if you need anything more on my end... thanks for the fast turnaround.
Author: Terry J. Reedy (terry.reedy) *
Date: 2007-03-21 23:16
I ask myself, what should I expect from the documentation...
object.init.doc 'x.init(...) initializes x; see x.class.doc for signature' object.class <type 'type'> type.doc "type(object) -> the object's type\ntype(name, bases, dict) -> a new type"
and I still don't know ;-).
Author: Adam Olsen (Rhamphoryncus)
Date: 2007-03-22 00:44
I think the avoidance of super() is largely because of this kind of leniency. Consider this snippet on python 2.3:
class Base(object): def init(self, foo=None, *args, **kwargs): super(Base, self).init(foo, *args, **kwargs) Base(foo='bar') Base('bar', 42) Base('bar', 42, x=7)
All pass silently, no error checking. Now consider a python 3.0 version, with a strict object.init:
class Base: def init(self, foo=None, *, y='hi', **kwargs): super(Base, self).init(**kwargs) Base(foo='bar') # Valid, accepted Base('bar', 42) # Raises exception Base('bar', x=7) # Raises exception
The error checking added by this bug/patch and the error checking added by PEP 3102 (keyword-only arguments) make super a lot more sane.
I think it would also help if calling a method via super() didn't allow positional arguments. If the base class's arguments can't be given as keyword args then you probably should call it explicitly, rather than relying on super()'s MRO.
I was also going to suggest super() should automagically create empty methods your parent classes don't have, but then I realized you really should have a base class that asserts the lack of such an automagic method:
class Base(object): def mymethod(self, myonlyarg='hello world'): assert not hasattr(super(Base, self), 'mymethod')
By the time you reach this base class you will have stripped off any extra arguments that your subclasses added, leaving you with nothing to pass up (and nothing to pass to). Having two mixins with the same method name and without a common parent class is just not sane.
Author: Guido van Rossum (gvanrossum) *
Date: 2007-03-22 04:28
I think it would also help if calling a method via super() didn't allow positional arguments.
That's absurd, except for init(), where it could make sense depending on the style of cooperation used. But not enough to enforce this in the language; in Py3k you will be able to enforce this on a per-class basis.
Having two mixins with the same method name and without a common parent class is just not sane.
Right. This is a cornerstone of cooperative multiple inheritance that sometimes seems to be forgotten; there is a big difference between defining a method and extending a method, and only extending methods can make super calls.
The init case is an exception, because there's no requirement that a subclass have a signature compatible with the superclass (if you don't get this, read up on constructors in C++ or Java).
Author: Adam Olsen (Rhamphoryncus)
Date: 2007-03-22 04:46
I think it would also help if calling a method via super() didn't allow positional arguments.
That's absurd, except for init(), where it could make sense depending on the style of cooperation used. But not enough to enforce this in the language; in Py3k you will be able to enforce this on a per-class basis.
The vast majority of "positional" arguments can also be given via name. The rare exceptions (primarily C functions) may not cooperate well anyway, so you're trading a relatively obscure limitation for better error detection.
Perhaps not that important though, since it could be taught as bad style unless absolutely needed.
Having two mixins with the same method name and without a common parent class is just not sane.
Right. This is a cornerstone of cooperative multiple inheritance that sometimes seems to be forgotten; there is a big difference between defining a method and extending a method, and only extending methods can make super calls.
The init case is an exception, because there's no requirement that a subclass have a signature compatible with the superclass (if you don't get this, read up on constructors in C++ or Java).
I understand the desire for it to be an exception, I fail to see how it actually is one. The namespace/signature conflicts exist just the same.
The only way I can see to handle incompatible signatures is to add a flag that says "I am the ONLY class allowed to subclass X" (triggering an error if violated), have super() entirely bypass it, and then call X.init() directly. Even that doesn't handle X's superclasses being subclassed more than once, and it looks pretty complicated/obscure anyway.
Author: Guido van Rossum (gvanrossum) *
Date: 2007-03-23 05:01
Committed revision 54539.
The committed version issues warnings rather than errors when both methods are overridden, to avoid too much breakage.
The string.py change was necessary to avoid spurious warnings (with no module/lineno!) and breakage of test_subprocess.py. Something fishy's going on -- is string.Template() used by the warnings module or by site.py???
I'm leaving this bug open but changing the category to Py3k so remind me it needs to be merged and then changed there.
Author: Raymond Hettinger (rhettinger) *
Date: 2007-04-02 23:12
FWIW, this change will be somewhat pervasive and will affect anything inheriting object.init including immutable builtins (like tuple, float, and frozenset) as well as user-defined new-style classes that do not define their own init method (perhaps using new instead). Here are the warnings being thrown-off by the current test suite:
/py26/Lib/test/test_array.py:731: DeprecationWarning: object.init() takes no parameters array.array.init(self, 'c', s)
/py26/Lib/copy_reg.py:51: DeprecationWarning: object.init() takes no parameters base.init(obj, state)
/py26/Lib/test/test_descr.py:2308: DeprecationWarning: object.init() takes no parameters float.init(self, value)
Author: Guido van Rossum (gvanrossum) *
Date: 2007-04-02 23:55
That's one way of looking at it. You could also say that it found two legitimate problems:
since array doesn't define init() there's no point in calling it
similarly, float doesn't define init()
The copy_reg warning is more subtle, and needs a work-around. I've checked in all three fixes.
Author: Benjamin Peterson (benjamin.peterson) *
Date: 2008-04-12 20:59
Can this be closed?
Author: Gregory P. Smith (gregory.p.smith) *
Date: 2010-02-02 06:01
FYI - A discussion on why this change may have been a bad idea and breaks peoples existing code:
http://freshfoo.com/blog/object__init__takes_no_parameters
Author: Jonas H. (jonash) *
Date: 2010-04-01 22:47
What exactly is the correct solution with Python 2.6 to avoid this warning? My use case is something like
class myunicode(unicode): def init(self, *args, **kwargs): unicode.init(self, *args, **kwargs) self.someattribute = calculate_attribute_once()
Shall I overwrite new rather than init? Or what :-)
Author: Terry J. Reedy (terry.reedy) *
Date: 2010-04-01 23:16
@dauerbaustelle I believe your question is a separate issue and that it should have been asked on Python list. However, yes, subclasses of immutables must override new. For more, do ask on the list, not here.
Author: Roundup Robot (python-dev) 
Date: 2012-03-17 05:06
New changeset 25b71858cb14 by Benjamin Peterson in branch 'default': make extra arguments to object.init/new to errors in most cases (finishes #1683368) http://hg.python.org/cpython/rev/25b71858cb14
Author: Guido van Rossum (gvanrossum) *
Date: 2012-03-17 05:09
Please don't add python-dev@python.org to the nosy list.
Author: Benjamin Peterson (benjamin.peterson) *
Date: 2012-03-17 05:11
python-dev is just the name of the robot which notes records changesets.
Author: Jason R. Coombs (jaraco) *
Date: 2013-01-14 21:08
For reference, I encountered an issue due to this change and didn't quite understand what was going on. I distilled the problem down and posted a question on stack overflow:
The answer led me here, so now I understand. I wanted to share this use-case for posterity.
I didn't find anything in the "what's new" documents for Python 3.3 or 3.0. Was this fundamental signature change to all objects documented anywhere? Any objection if I draft a change to the docs?
Author: Terry J. Reedy (terry.reedy) *
Date: 2013-01-15 02:47
First, What's New " explains the new features in Python". This issue is a bugfix. AFAIK, object() has always been documented as having no parameters. The fact that passing extra args should raise a TypeError is no secret.
Second, this is documented. The third sentence of http://docs.python.org/3/whatsnew/3.3.html is " For full details, see the changelog." We really mean that ;-). The changelog is derived from Misc/NEWS in the repository. It says "Issue #1683368: object.new and object.init raise a TypeError if they are passed arguments and their complementary method is not overridden." That is prefixed by "Issue #1683368:", which links to this issue. This entry is easily found by searching for 'object.init' (or a sufficient prefix thereof).
For 3.2, the What's New sentence was "For full details, see the Misc/NEWS file" and the link went to the raw repository file. http://hg.python.org/cpython/file/3.2/Misc/NEWS My impression is that this issue played a role in including the prettified version, instead of just the repository link, in the on-line version of the docs. What's New for 2.7 does not even have the link.
In any case, any bugfix breaks code that depends on the bug. Hence the decision to make the full changelog more available and more readable.
I realize that the change to the header for What's New is hard to miss. But what are we to do? Add a new What's New in What's New doc for one release? Put the change in flashing red type?
Author: Jason R. Coombs (jaraco) *
Date: 2013-01-15 03:34
Aah. Indeed, that's where I should have looked. Thanks for the pointer.
Author: Jason R. Coombs (jaraco) *
Date: 2014-05-28 05:52
I recently ran into this error again. I was writing this class to provide backward-compatible context manager support for zipfile.ZipFile on Python 2.6 and 3.1:
class ContextualZipFile(zipfile.ZipFile): """ Supplement ZipFile class to support context manager for Python 2.6 """
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
self.close()
def __new__(cls, *args, **kwargs):
"""
Construct a ZipFile or ContextualZipFile as appropriate
"""
if hasattr(zipfile.ZipFile, '__exit__'):
return zipfile.ZipFile(*args, **kwargs)
return super(ContextualZipFile, cls).__new__(cls, *args, **kwargs)At the point where super is called, the author is unaware of the details of the function signature for zipfile.ZipFile.new, so simply passes the same arguments as were received by the derived class. However, this behavior raises a DeprecationWarning on Python 2.6 and 3.1 (and would raise an error on Python 3.2 if the code allowed it).
What's surprising is that the one cannot simply override a constructor or initializer without knowing in advance which of those methods are implemented (and with what signature) on the parent class.
It seems like the construction (calling of new) is special-cased for classes that don't implement new.
What is the proper implementation of ContextualZipFile.new? Should it use super but omit the args and kwargs? Should it call object.new directly? Should it check for the existence of new on the parent class (or compare it to object.new)?
Author: Guido van Rossum (gvanrossum) *
Date: 2014-05-28 15:05
If you don't know enough about the base class you shouldn't be subclassing it. In this particular case you should be overriding init, not new.
Author: Terry J. Reedy (terry.reedy) *
Date: 2014-05-28 15:23
From what I see, you do not need to change either new or init, just add enter and exit , and you only need to do that in 2.6. Since Zipfile is written in Python, you could monkey-patch instead of subclassing, if that is easier in your particular case.
Author: Eric Snow (eric.snow) *
Date: 2014-05-28 15:39
If you don't know enough about the base class you shouldn't be subclassing it.
That's important when overriding any API in subclass and absolutely always essential when it comes to new and init! That's something that isn't very obvious at first. :(
In this particular case you should be overriding init, not new.
Jason's code is doing something like OSError.new does now, which returns an instance of a subclass depending on the errno. However, while the language supports it, I see that as a viable hack only when backward-compatibilty is a big concern. Otherwise I find factory classmethods to be a much better solution for discoverability and clarity of implementation.
Author: Guido van Rossum (gvanrossum) *
Date: 2014-05-28 17:42
Sorry, I didn't realize why new was being used. But what Jason's code is doing isn't any cleaner than monkey-patching.
Author: Jason R. Coombs (jaraco) *
Date: 2014-05-28 21:15
Maybe I should have focused on a more trivial example to demonstrate the place where my expectation was violated. The use of a real-world example is distracting from my intended point. Consider instead this abstract example:
class SomeClass(SomeParentClass): def new(cls, *args, **kwargs): return super(SomeClass, cls).new(cls, *args, **kwargs)
def __init__(self, *args, **kwargs):
super(SomeClass, self).__init__(*args, **kwargs)Ignoring for a moment the incongruity of the invocation of new with 'cls' due to new being a staticmethod, the naive programmer expects the above SomeClass to work exactly like SomeParentClass because both overrides are implemented as a trivial pass-through.
And indeed that technique will work just fine if the parent class implements both init and new, but if the parent class (or one of its parents) does not implement either of those methods, the technique will fail, because the fall through to 'object' class.
I believe this incongruity stems from the fact that new and init are special-cased not to be called if they aren't implemented on the class.
Therefore, to write SomeClass without knowledge of the SomeParentClass implementation, one could write this instead:
class SomeClass(SomeParentClass): def new(cls, *args, **kwargs): super_new = super(SomeClass, cls).new if super_new is object.new: return super_new(cls) return super_new(cls, *args, **kwargs)
def __init__(self, *args, **kwargs):
super_init = super(SomeClass, self).__init__
if super_init.__objclass__ is object:
return
super_init(*args, **kwargs)Now that implementation is somewhat ugly and perhaps a bit brittle (particularly around use of objclass). Ignoring that for now, it does have the property that regardless of the class from which it derives, it will work, including:
SomeParentClass = datetime.datetime # implements only new SomeParentClass = zipfile.ZipFile # implements only init class SomeParentClass: pass # implements neither init nor new
While I would prefer a language construct that didn't require this dance for special casing (or similarly require the programmer to hard-code the dance to a specific implementation of a specific parent class as Guido recommends), at the very least I would suggest that the documentation better reflect this somewhat surprising behavior.
Currently, the documentation states [https://docs.python.org/2/reference/datamodel.html#object.new] effectively "Typical implementations of new invoke the superclass’ new() method with appropriate arguments." It's left as an exercise to the reader to ascertain what 'appropriate arguments' are, and doesn't communicate that the introduction or omission of new or init to a class hierarchy affects the process by which a class is constructed/initialized.
Greg Smith's blog demonstrates some even more dangerous cases. I don't understand why his concerns weren't addressed, because they seem legitimate, and I agree with his conclusion that the older behavior is more desirable, despite the concerns raised by the OP.
Author: Jason R. Coombs (jaraco) *
Date: 2014-05-28 21:47
Based on the example above, I've created a blog post to publish my recommendation for overriding these special methods in a way that's safe regardless of the parent implementation, given the status quo:
http://blog.jaraco.com/2014/05/how-to-safely-override-init-or-new-in.html
Author: Guido van Rossum (gvanrossum) *
Date: 2014-05-28 22:57
Hrm. I've always thought that the key point of cooperative MI was the term cooperative. Consider a regular (non-constructor) method. You must have a common base class that defines this method, and that method shouldn't be calling the super-method (because there isn't one). All cooperative classes extending this method must derive from that base class.
It's the same for init and new, except that you may treat each (keyword) argument as a separate method. But you must still have a point in the tree to "eat" that argument, and that point must not pass it up the super call chain.
If in a particular framework you want unrecognized keyword arguments to the constructor to be ignored, you should define a common base class from which all your cooperative subclasses inherit. But given the prevalence of single inheritance, 'object' shouldn't be that common base class.
Author: Raymond Hettinger (rhettinger) *
Date: 2014-06-03 07:05
Jason, I made some recommendations on this subject in my blog post a few years ago: http://rhettinger.wordpress.com/2011/05/26/super-considered-super/
''' A more flexible approach is to have every method in the ancestor tree cooperatively designed to accept keyword arguments and a keyword-arguments dictionary, to remove any arguments that it needs, and to forward the remaining arguments using **kwds, eventually leaving the dictionary empty for the final call in the chain.
Each level strips-off the keyword arguments that it needs so that the final empty dict can be sent to a method that expects no arguments at all (for example, object.init expects zero arguments):
class Shape: def init(self, shapename, **kwds): self.shapename = shapename super().init(**kwds)
class ColoredShape(Shape): def init(self, color, **kwds): self.color = color super().init(**kwds)
cs = ColoredShape(color='red', shapename='circle')
'''
History
Date
User
Action
Args
2022-04-11 14:56:23
admin
set
github: 44742
2015-05-13 16:27:58
terry.reedy
set
type: behavior
stage: resolved
2014-06-03 07:05:43
rhettinger
set
messages: +
2014-05-28 22:57:52
gvanrossum
set
messages: +
2014-05-28 21:47:10
jaraco
set
messages: +
2014-05-28 21:15:59
jaraco
set
messages: +
2014-05-28 17:42:15
gvanrossum
set
messages: +
2014-05-28 15:39:11
eric.snow
set
nosy: + eric.snow
messages: +
2014-05-28 15:23:55
terry.reedy
set
messages: +
2014-05-28 15:05:09
gvanrossum
set
messages: +
2014-05-28 15:03:45
terry.reedy
set
messages: +
2014-05-28 15:02:28
terry.reedy
set
messages: -
2014-05-28 15:02:20
gvanrossum
set
messages: -
2014-05-28 05:52:09
jaraco
set
messages: +
2014-05-28 05:45:02
jaraco
set
messages: +
2013-01-15 03:34:27
jaraco
set
messages: +
2013-01-15 02:47:36
terry.reedy
set
messages: +
2013-01-14 21:08:39
jaraco
set
nosy: + jaraco
messages: +
2012-03-17 05:11:55
benjamin.peterson
set
messages: +
2012-03-17 05:09:50
gvanrossum
set
nosy: - python-dev
messages: +
2012-03-17 05:06:18
python-dev
set
nosy: + python-dev
messages: +
2010-04-01 23:16:11
terry.reedy
set
messages: +
2010-04-01 22:47:13
jonash
set
nosy: + jonash
messages: +
2010-02-02 06:01:19
gregory.p.smith
set
nosy: + gregory.p.smith
messages: +
2009-04-20 10:35:10
KayEss
set
nosy: + KayEss
2008-10-13 12:41:32
jcea
set
nosy: + jcea
2008-04-14 20:38:47
gvanrossum
set
status: open -> closed
2008-04-12 20:59:56
benjamin.peterson
set
nosy: + benjamin.peterson
messages: +
2008-01-06 22:29:46
admin
set
keywords: - py3k
versions: Python 3.0
2007-08-30 00:22:14
gvanrossum
set
versions: + Python 3.0
2007-03-19 03:32:33
blakeross
create