bpo-36867: Make semaphore_tracker track other system resources. by pierreglaser · Pull Request #13222 · python/cpython (original) (raw)
Primary Purpose
This PR proposes to extend the semaphore_tracker range of action to other system resources.
Currently in python3.8, if a process that created a shared_memory object gets violently terminated (with kill for example), the memory segment will not be properly released until the next machine reboot. This will be problematic for long-running clusters used by many different users.
The primary reason of this PR is thus to extend the semaphore_tracker to track also shared_memory segments. The semaphore_tracker module gets consequently renamed resource_tracker.
Extension to a potentially public API
Additionally, the design of the resource_tracker is more generic, and possibly suggests a new public API to enable the tracking other named-resources. For now, the private _CLEANUP_FUNCS dict references all the types that can possibly be tracked.
To start tracking a new type, the resource_tracker only needs to know which cleanup routine to apply to each leaked instance of this type after the python session ends. The public API would thus consist of a function like resource_tracker.register_resource_type(resource_type, cleanup_func), where:
- resource_type is a
string cleanup_funcmust be a callable taking a single string as an argument.
Under the hood, register_resource_type would populate the CLEANUP_FUNCS with the new (type, cleanup_func) record.
Operating System Availability
To ease the review process of this PR. the resource_tracker is now POSIX only. I however have a functional windows implementation that I will provide upon demand :)
pinging @pitrou @ogrisel @tomMoral
@pablogsal this may interest you as well :)