[python 3.14] Metaclasses: interact with annotations from namespace dict (original) (raw)
April 3, 2025, 4:15pm 1
Hi everyone,
After having read the python3.14 “What’s new” about annotations and both related PEPs 649 and 749, I have a question about how to interact with annotations from a metaclass’ __prepare__ or __new__.
The simplest first: How to retrieve annotations from a namespace dict? Is the following ok?
from annotationlib import Format
def get_namespace_annotations(namespace: dict[str, object]) -> dict[str, object]:
"""Get annotations from the namespace dict."""
if (ann := namespace.get("__annotate__", None)) is not None:
return ann(Format.VALUE) # Change to FORWARDREF when implemented (?)
return cast("dict[str, object]", namespace.get("__annotations__", {}))
Now the second question would be: How to manipulate its annotations (i.e. add/remove/modify)?
I guess the “legacy” way would be to just use get_namespace_annotations, then set __annotate__ to None, before using the obtained dict as in legacy. Is that even ok for a start?
And of course, is there a “cleaner” way, interacting directly with __annotate__?
I worry mainly because of this:
The behavior of accessing the
__annotations__and__annotate__attributes on classes with a metaclass other thanbuiltins.typeis unspecified. The documentation should warn against direct use of these attributes and recommend using theannotationlibmodule instead.Similarly, the presence of
__annotations__and__annotate__keys in the class dictionary is an implementation detail and should not be relied upon.
Thanks!
DavidCEllis (David Ellis) April 3, 2025, 5:13pm 2
I’m not sure about getting the __annotate__ function, I’m currently doing roughly the same as you in order to get the function itself in my own metaclass. If it stops being available in the namespace then I’m not sure how you’re intended to obtain them in __new__.
I’d note that you shouldn’t call it directly with Format.FORWARDREF in most cases, you’ll want to use annotationlib.call_annotate_function(ann, Format.FORWARDREF).
eliegoudout (Eliegoudout) April 3, 2025, 5:29pm 3
Oh that’s a very good point thanks! I missed that despite looking at the source lol. Without any owner I guess since it does not exist yet inside __new__?
eliegoudout (Eliegoudout) April 4, 2025, 4:37pm 4
I guess a possibility is to redefine __annotate__:
def new_annotate(added_annotations: dict[str, object]) -> Callable[[int], dict[str, object]]:
"""Adds annotations manually."""
def inner(mode: int) -> dict[str, object]:
return __annotate__(mode) | added_annotations
return inner
But again, I’m a bit lost on what the recommended practice is…
eliegoudout (Eliegoudout) April 9, 2025, 6:54pm 5
If this ever helps someone, for now I use the following
from annotationlib import Format, call_annotate_function # type: ignore[import-not-found]
from typing import cast
def get_namespace_annotations(namespace: dict[str, object]) -> dict[str, object]:
"""Get annotations from a namespace dict in python3.14."""
__annotations__ = cast("dict[str, object] | None", namespace.get("__annotations__"))
__annotate__ = namespace.get("__annotate__")
if __annotations__ is None:
if __annotate__ is None:
return {}
return call_annotate_function(__annotate__, Format.FORWARDREF)
# Avoid bad synchronization, suppress `__annotate__`
if __annotate__ is not None:
namespace["__annotate__"] = None
return __annotations__
which seems to work but does not sanitize __annotate__ and __annotations__.
Jelle (Jelle Zijlstra) April 10, 2025, 5:11am 6
This works with the current alpha, but in general I don’t want people to peek at the internal implementation details regarding where things are stored. I just put up a PR (gh-132261: Store annotations at hidden internal keys in the class dict by JelleZijlstra · Pull Request #132345 · python/cpython · GitHub) that will make it so you can call annotationlib.get_annotate_function(ns) to get the annotate function out of the namespace.
Your function also immediately calls the annotate function. This has the downside that if the class contains annotations that cannot resolve immediately, this will fail. You can use annotationlib.call_annotate_function(ann, Format.FORWARDREF) to address this. This is what I’d use if I need to access the annotations at class creation time. For example, this is what the implementation of typing.TypedDict does on current main.
I’d generally recommend to write a wrapper that evaluates the annotations and makes the changes you want. Example:
import annotationlib
class Meta(type):
def __new__(mcls, name, bases, ns):
annotate = annotationlib.get_annotate_function(ns)
typ = super().__new__(mcls, name, bases, ns)
def wrapped_annotate(format):
annos = annotationlib.call_annotate_function(annotate, format, owner=typ)
annos["Meta"] = "example"
return annos
typ.__annotate__ = wrapped_annotate
return typ
I will probably add some recipes to the annotationlib docs with examples like this.
General caution: We’re still in the alpha phase, and anything may still change.
eliegoudout (Eliegoudout) April 12, 2025, 8:24am 7
Thanks for your message. Did you take into account my last message just before yours? I think it essentially implements it the way you meant it?
Also, I just looked up your PR, it seems to check whether ns is a dict, shouldn’t it check for Mapping only since __prepare__ could choose any type of mapping?
Regarding modification I agree that it is a possible way, but it seems a bit wonky. I know this is in alpha phase but I’m honestly kind of surprised the PEPs were accepted in their current state, when they leave so much room for debate while breaking a lot of __annottions__ use.
sixpearls (Ben Margolis) February 10, 2026, 11:08pm 8
Is there anyway to access the annotations, in any format, at the __setitem__ of the object returned by __prepare__, before the next assignment in the class definition which may depend on the proper handling of the annotation? In 3.13 and earlier, I could intercept the assignment of the __annotations__ attribute to do what I need (and, ultimately, discard that annotation), which would occur between the annotated assignment and the next expression in the class. I have tried to capture assignment to e.g., __annotate_func__, but that does not appear to be assigned until pretty late in the class creation. I can do what is needed if the user code (with the annotation in the class) uses future, but would like it if I didn’t need to impose this on users and if I could make this work after it is deprecated.
Thanks,
Jelle (Jelle Zijlstra) February 11, 2026, 3:07am 9
There isn’t really a way. For reference below is the compiled bytecode for a class with annotations. The closest you could do would I guess be to find the code object for the annotations (no documented way to do it but should be possible) and then match up the annotations with the assignments. However, that would be very fragile.
Why do you need this?
>>> def f():
... class X:
... a: int = 1
... b: str = 2
...
>>> dis.dis(f)
1 RESUME 0
2 LOAD_BUILD_CLASS
PUSH_NULL
LOAD_CONST 0 (<code object X at 0x102812340, file "<python-input-3>", line 2>)
MAKE_FUNCTION
LOAD_CONST 1 ('X')
CALL 2
STORE_FAST 0 (X)
LOAD_CONST 2 (None)
RETURN_VALUE
Disassembly of <code object X at 0x102812340, file "<python-input-3>", line 2>:
-- MAKE_CELL 0 (__classdict__)
2 RESUME 0
LOAD_NAME 0 (__name__)
STORE_NAME 1 (__module__)
LOAD_CONST 0 ('f.<locals>.X')
STORE_NAME 2 (__qualname__)
LOAD_SMALL_INT 2
STORE_NAME 3 (__firstlineno__)
LOAD_LOCALS
STORE_DEREF 0 (__classdict__)
3 LOAD_SMALL_INT 1
STORE_NAME 4 (a)
4 LOAD_SMALL_INT 2
STORE_NAME 5 (b)
2 LOAD_FAST 0 (__classdict__)
BUILD_TUPLE 1
LOAD_CONST 1 (<code object __annotate__ at 0x102876d30, file "<python-input-3>", line 2>)
MAKE_FUNCTION
SET_FUNCTION_ATTRIBUTE 8 (closure)
STORE_NAME 6 (__annotate__)
LOAD_CONST 2 (())
STORE_NAME 7 (__static_attributes__)
LOAD_FAST 0 (__classdict__)
STORE_NAME 8 (__classdictcell__)
LOAD_CONST 3 (None)
RETURN_VALUE
Disassembly of <code object __annotate__ at 0x102876d30, file "<python-input-3>", line 2>:
-- COPY_FREE_VARS 1
2 RESUME 0
LOAD_FAST 0 (format)
LOAD_SMALL_INT 2
COMPARE_OP 132 (>)
POP_JUMP_IF_FALSE 3 (to L1)
NOT_TAKEN
LOAD_COMMON_CONSTANT 1 (NotImplementedError)
RAISE_VARARGS 1
3 L1: LOAD_CONST 0 ('a')
LOAD_DEREF 1 (__classdict__)
LOAD_FROM_DICT_OR_GLOBALS 0 (int)
4 LOAD_CONST 1 ('b')
LOAD_DEREF 1 (__classdict__)
LOAD_FROM_DICT_OR_GLOBALS 1 (str)
2 BUILD_MAP 2
RETURN_VALUE
sixpearls (Ben Margolis) February 11, 2026, 5:11am 10
I am the lead developer of Condor, where we use Python’s metaprogramming hooks to create a blackboard like environment so engineers can easily define their mathematical models. This requires manipulating the class’s namespace as it is being defined.
On a recent internal project, we wanted to take advantage of annotations to improve some documentation output. In Python 3.13, we can define models like this:
class Sys(condor.ExplicitSystem):
a = input()
b: output = a**2
c: output = b + 1.0
and we can manipulate what actually gets assigned to the b attribute before the b+1.0 is evaluated on the next line. The standard, non-annotation-based approach would work in Python 3.14 (or below)
class Sys(condor.ExplicitSystem):
a = input()
output.b = a**2
output.c = b + 1.0
For the purposes of improving documentation, perhaps we are better off automatically providing annotations for documentation rather than using the annotation syntax to define Condor models. It was cute that the annotation syntax worked, but maybe there isn’t actually much of a value added.
sixpearls (Ben Margolis) February 11, 2026, 8:55pm 11
I’m not familiar with the code objects as it relates to the creation of the class. Is the code object available before the __setitem__(“a”, 1) is called? How does the __annotate__ object “see” the annotation?
And to clarify, the Annotation semantics describes “deferred evaluation” as the default as of 3.14, but it is actually the only mechanism available (once __future__.annotation is removed)? You can defer the evaluation to earlier* but cannot be as eager as it was before?
The earliest possible evaluation seems to be triggered by entering __new__, when the annotationlib.get_annotate_from_class_namespace starts to work (after __annotate__ or __annotate_func__ is added to the namespace).
Still looking into the possibility of helping Sphinx without using the annotation expression, so hopefully we can work around this change. I’m just trying to understand.
Thanks so much for your help!
Jelle (Jelle Zijlstra) February 12, 2026, 4:43am 12
Your use case is highly unusual and didn’t come up in the voluminous discussions about PEP 649 for several years. I’m afraid it will not be supported.
Yes, deferred evaluation is the default in 3.14. from __future__ import annotations (which turns all annotations into strings) still exists, but will be removed in the distant future. Eager evaluation, which was the default in 3.13 and earlier, no longer exists.
If you try hard enough in Python everything is available. You can do something like this:
In [19]: class X:
...: code = sys._getframe().f_code.co_consts[-4]
...: print(code)
...: a: int
...:
<code object __annotate__ at 0x3a2c2e530, file "<ipython-input-19-47380f91972a>", line 1>
And then some more tricks to build a function object from the code object. But this will be fragile and not portable.
sixpearls (Ben Margolis) February 12, 2026, 1:27pm 13
And this is not even the core use for Condor, just something we stumbled into recently
while trying to figure out the change in 3.14, I did a lot of reading; “voluminous” is an understatement.
And that’s part of why I love it. Thanks for the suggestion, I did some experimenting and got to accessing the appropriate frame from __setitem__ and found the code object for the __annotate__. If we can’t make Sphinx (or some other documentation tool) do what we need I can look into continuing this path of constructing the function from the code object.
Thanks!
jsbueno (Joao S. O. Bueno) February 16, 2026, 10:54pm 14
It looks like following annotations as the class body is created really gets complicated with the new delayed annotations.
Changing the approach altogether, however, maybe could keep the “annotations syntax” for Condor
One way I can think of is to enable tracing (`sys.settrace`, usually good for debugging) in `_ _prepare__` so you can get a callback for each line of code inside the class body -
and then, really, maybe “get_source” to check the actual text, and use the annotation.
Since class parsing won’t likely be a performance critical region of the code, neither runs in concurrent code (threaded or async), probably that would work.
I put together a minimal example on setting `sys.settrace` in `_ _prepare__`:
import inspect
import sys
class Meta(type):
_creating_class = False
_source = None
_first_lno = 0
@classmethod
def __prepare__(mcls, name, bases, **kwargs):
sys.settrace(mcls.follow_class)
return super().__prepare__(name, bases, **kwargs)
@classmethod
def follow_class(mcls, frame, event, arg):
if event == "call" and not mcls._creating_class:
mcls._creating_class = True
mcls._code = frame
mcls._source = inspect.getsource(frame).split("\n")
mcls._first_lno = frame.f_lineno
# print(mcls._source)
return mcls.follow_class
elif event == "line" and mcls._creating_class:
source = inspect.getsource(frame)
print("to execute: ", mcls._source[frame.f_lineno - mcls._first_lno])
return mcls.follow_class
elif event == "return":
print("finishing up introspection")
mcls._code = None
mcls._creating_class = False
return None
output = object()
class Sys(metaclass=Meta):
a = 23
b: output = a**2
c: output = b + 1.0
print("all done")
blhsing (Ben Hsing) March 2, 2026, 4:07am 15
One relatively simple and robust workaround, compared to bytecode decoding or source parsing, is to execute the class body twice–the first pass to obtain the __annotations__ attribute after the class is created and the second pass to rerun the class body with the prior __annotations__ injected.
In order to rerun the class body we need the code object of the class body function, which can be obtained from the caller’s frame when the __setitem__ method of namespace dict gets called for setting __module__ and __qualname__, etc.
This approach works as long as the class body has no side effect and as long as its annotations aren’t defined conditionally or dynamically, which should be the case in most practical applications since annotations are typically meant to be static:
import sys
class AnnotationsNamespace(dict):
def __setitem__(self, name, value):
if getattr(self, 'frame', None) is None:
self.frame = sys._getframe(1)
super().__setitem__(name, value)
class AnnotationsMeta(type):
@classmethod
def __prepare__(metacls, name, bases):
return AnnotationsNamespace()
def __new__(metacls, name, bases, namespace, **kwargs):
cls = super().__new__(metacls, name, bases, namespace, **kwargs)
new_locals = {'__annotations__': cls.__annotations__}
exec(namespace.frame.f_code, namespace.frame.f_globals, new_locals)
return super().__new__(metacls, name, bases, new_locals, **kwargs)
class Foo(metaclass=AnnotationsMeta):
a: int
if annotations := locals().get('__annotations__'):
print(annotations)
This outputs:
{'a': <class 'int'>}
blhsing (Ben Hsing) March 3, 2026, 2:01am 16
Another way of getting the code object of the class body function is with the __build_class__ hook:
import builtins
from types import resolve_bases, _calculate_meta
class WithAnnotations:
pass
def build_class_with_annotations(func, name, *bases, metaclass=type, **kwargs):
cls = orig_build_class(func, name, *bases, metaclass, **kwargs)
if WithAnnotations not in bases:
return cls
metaclass = _calculate_meta(metaclass, bases)
namespace = metaclass.__prepare__(name, bases, **kwargs)
namespace['__annotations__'] = cls.__annotations__
if (resolved_bases := resolve_bases(bases)) is not bases:
namespace['__orig_bases__'] = bases
exec(func.__code__, func.__globals__, namespace)
return metaclass(name, resolved_bases, namespace, **kwargs)
orig_build_class = __build_class__
builtins.__build_class__ = build_class_with_annotations
class Foo(WithAnnotations):
a: int
if annotations := locals().get('__annotations__'):
print(annotations) # {'a': <class 'int'>}
blhsing (Ben Hsing) March 4, 2026, 2:00am 17
Ah, it did not ring a bell to me when I first read your post, but your suggestion is definitely by far the best workaround as long as the interpreter is CPython-compatible:
import sys
from types import CodeType, CellType, FunctionType
class X:
a: int
print(
FunctionType(
next(
const for const in sys._getframe().f_code.co_consts
if isinstance(const, CodeType) and
const.co_name == '__annotate__'
),
globals(),
closure=(CellType({}),)
)(1)
)
which outputs:
{'a': <class 'int'>}