Adding Deep Immutability (original) (raw)
Following the discussions today at the Python Language Summit, we would like to circulate the first of the four-ish PEPs: peps/pep-9999.rst at main · TobiasWrigstad/peps · GitHub
For context, we are proposing making Python data-race free through a series of PEPs ultimately delivering a Rust-like dynamically checked ownership system. You can find more information about the whole project here — Fearless Concurrency for Python | Project Verona — including the slides from the aforementioned talk.
AA-Turner (Adam Turner) May 15, 2025, 12:01am 2
For those of us not at PyCon / the Language Summit, could you briefly précis the intent for the ‘four-ish PEPs’? A comment arising from this document may be answered by a draft PEP you’ve already prepared, so better to spare everyone the trouble.
A
stw (Tobias Wrigstad) May 15, 2025, 1:38am 3
Yes, of course! There is a better and more comprehensive outline in the PEP linked in the post, but briefly:
1st PEP — Deep Immutability; objects can be made deeply immutable after which mutation attempts throw an exception. (This is the PEP linked in the post.)
2nd PEP: Add support for managing cyclic immutable garbage using a shared reference counter per strongly connected component of immutable objects plus adding support for atomic reference counting of immutable objects. This will permit immutable objects to be shared across threads and also across subinterpreters (by reference).
3rd PEP: Support for “sending” and “receiving” immutable objects between threads and subinterpreters + easy way to spawn parallel tasks that can run on subinterpreters and threads to facilitate opting in to parallelism.
4th PEP: Support for sharing mutable data between threads and subinterpreters using region-based ownership.
–Tobias
sirosen (Stephen Rosen) May 15, 2025, 3:44am 4
I wasn’t able to read it all right away, but I would be surprised if class and metaclass freezing didn’t pose an issue for users of __init_subclass__.
That hook is often used to create some class accessible registry. Freezing such a registry could make subclassing start to fail. Maybe that isn’t an issue? But it seems likely to be surprising to me.
Overall, very interesting – having built-in support for immutable types would be pretty great.
Tinche (Tin Tvrtković) May 15, 2025, 4:41am 5
How do you propose modeling this in the type system?
Nineteendo (Nice Zombies) May 15, 2025, 6:04am 6
Even though we might never add shallow freezing, wouldn’t it be better to call the function deepfreeze()?
blhsing (Ben Hsing) May 15, 2025, 8:48am 7
Nice approach to making most objects in Python immutable. One of the important properties of immutable objects to me though is hashability, and I don’t see any mention of it in the PEP or the reference implementation.
It should be feasible to make all deep-frozen objects hashable since all the primitives are hashable.
AndersMunch (Anders Munch) May 15, 2025, 12:20pm 8
Nice work.
Freezing functions, that’s some tricky stuff. I’d like to show you two pieces of code.
First, a normal class.
class C1:
__slots__ = ['a']
def __init__(self):
self.a = 0
def f(self):
self.a += 1
def g(self):
return self.a
c1 = C1()
c1.f()
print(c1.g())
Next, something functionally similar.
def C2():
a = 0
def f():
nonlocal a
a += 1
def g():
return a
return { 'f': f, 'g': g }
c2 = C2()
c2['f']()
print(c2['g']())
C2 is the sort of thing you might make in a programming language that has closures but doesn’t have classes. c1 and c2 work similarly, it’s just that where c1 uses attribute lookup, c2 uses indexing.
After freezing c1 and c2, c1.f() will raise an exception. But c2['f']() won’t, I guess?
mjp41 (Matthew Parkinson) May 15, 2025, 1:37pm 9
They will both raise an exception after freezing:
>>> freeze(c2)
>>>
>>> c2['f']()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in f
TypeError: cannot write to local variable 'a'
In this case, it freezes the dictionary, which in turn freezes the two function objects it contains. It then traverses the function objects fields, which includes the variables it has closed over and freezes those.
pf_moore (Paul Moore) May 15, 2025, 1:47pm 10
One thing that surprised and impressed me is that this already comes with a reference implementation. This definitely looks interesting.
sirosen (Stephen Rosen) May 15, 2025, 3:16pm 11
I was wondering about typing as well.
Is it feasible to make a new special type form, similar to Final, as an indicator for this?
i.e. freeze(x: T) makes x into Frozen[T]?
There are all sorts of ways that won’t capture things, since freeze is an imperative which modifies the type of the existing value and many other values around it. Plus, Frozen[T] is not a subtype of T, and naively T is a subtype of Frozen[T], so… yikes!
I have no clue if that can be made to work coherently. It seems sensible from the perspective of a Python typing user. It gets tricky fast – you’d think that isfrozen is of type TypeIs[Frozen[T]], but for that to work, inputs have to be of type T | Frozen[T] – if xs: list[int], isfrozen(xs) isn’t valid. xs would have to be typed as xs: list[int] | Frozen[list[int]].
I trust the typing experts to have clearer and stronger opinions here, but I can’t imagine that the type system can represent this without specialized support. typing might be able to give us something in the near term like Frozen which at least lets us annotate that a value is frozen. And maybe isfrozen gets typed as TypeIs[Frozen[T]], so that you can write functions which are annotated as accepting frozen or unfrozen inputs and handle those cases explicitly.
Tinche (Tin Tvrtković) May 15, 2025, 6:11pm 12
Yeah, I don’t think there’s a way to make it work with typing as proposed, hence my question. To me this indicates a larger issue - if any mutable object can become immutable at any time, reasoning about what is safe is going to be real tricky, typing or not.
Both the Python runtime system and the Python typing system already have a robust way of dealing with immutability - ints, floats, strings, bytes, tuples, namedtuples, frozensets, frozen dataclasses and frozen attrs classes. Getting those to share safely across threads and subinterpreters would be a great result though.
Another approach might be to make calling freeze() only type check if the argument is statically known to be deeply immutable already. It would still be tricky - Sequence[int] wouldn’t be enough, you’d need to use tuples explicitly, for example (because lists are also sequences).
I’m not concerned about the typing impact of this, there’s other typequalifiers that don’t imply a subtyping relationship, I don’t see how Frozen would be a problem here.
Frozen would likely just need to be a type qualifier that makes setattr and setitem (del too) disallowed for all attributes.
modeling methods that mutate an item would require more here[1], but there are many runtime constructs not expressible in typing yet, it’s fine for this to not be immediately suported.
- perhaps a negated intersection as
Self & ~Frozen[Self]to mark that a method is only safe when not frozen? ↩︎
Tinche (Tin Tvrtković) May 15, 2025, 6:29pm 14
Can you elaborate on how this could ever be supported? It’s the equivalent of having a function that turns a list into a tuple in-place. What about other, existing references to that object that are still lists in the eyes of the type system?
I believe the typing impact is only a distraction in this topic and don’t consider it a question requiring a detailed answer at this time. It is not a requirement of the language that all things are supported in typing, in fact that’s an explicit nongoal of typing as it has been accepted.
I also believe that the way people use immutable objects, they pass them to things where an immutable object is expected, and this is only to prevent a class of runtime mutability issues where mutation is accidental (and protecting against this being needed for concurrency, but also for interpreter safety), something already not modeled in the type system accurately due to preexisting problems, like variance compatibility not being a requirement of compatible subtyping. (see: Sequence MutableSequence subtyping relationship)
sayandipdutta (Sayandip Dutta) May 15, 2025, 6:44pm 16
Regarding copy-on-write, the pre-PEP writes this under a More Rejected Alternatives section:
Copy-on-Write Immutability: Instead of raising errors on mutation, create a modified copy. However, this changes object identity semantics and is less predictable. Support for copy-on-write may be added later, if a suitable design can be found.
Should this not be under a Deferred Ideas section?
sirosen (Stephen Rosen) May 15, 2025, 7:01pm 17
We don’t necessarily need to look at typing specifically, but I think the underlying question is a reasonable one: how will I reason about code in which objects which were mutable can become immutable via action at a distance? What developer tools will make this easy to reason about?
Part of the answer will surely be “don’t write code with unclear contracts around immutability”.
Typing is now part of how we reason about documentation and contracts – I think that’s why it arises here naturally as a topic.
I need to read the document more, but my area of interest right now with the proposed freeze API is how we can reason about the scope of the changes it makes. e.g., if I freeze an object, could that result in freezing the module in which it or its class was defined?
I hinted at my perspective in the prior post, but I don’t think this is actually all that important to have specific reasoning or developer tooling for. The use case for this is pretty clear in my mind: freeze things that you are only ever passing to things that should never mutate the item, use the other benefits of this to also safely interact with things like subinterpreters. If my function mutates something and someone freezes something before passing it to my function, that’s their error, not mine, and they’ll get an exception to handle it. They should also get that exception consistently for having frozen it, so code paths like this won’t be a common mistake or something every function author has to think about.
NeilGirdhar (Neil Girdhar) May 15, 2025, 8:14pm 19
I agree with you that there is a significant loss in terms of benefits provided by static type checking.
Would it be possible to explore some ways of recovering the tight static types? For example, just brainstorming:
def f(x: MutableSequence[T]):
freeze(x)
reveal_type(x) # Somehow loses MutableSequence and becomes Sequence in the eyes of type checkers?
Can freeze(x) make a deep copy, or would that cost too much? The benefit would be that any owners of x would not have x’s interface changed out from under them. If that doesn’t work, could some discussion of why not be added to the PEP?
MegaIng (Cornelius Krupp) May 15, 2025, 8:32pm 20
This still pushes the problem up to whoever called f.
What would actually be needed is something like a MightFreeze qualifier that indicates that a function will/might freeze it’s parameter, and you aren’t allowed to pass in objects without that qualifier you don’t own (whatever ownership means; That’s something for future PEPs in this series/related PEPs to clarify).