xarray.backends.CachingFileManager (original) (raw)

class xarray.backends.CachingFileManager(opener, *args, mode=, kwargs=None, lock=None, cache=None, manager_id=None, ref_counts=None)[source]#

Wrapper for automatically opening and closing file objects.

Unlike files, CachingFileManager objects can be safely pickled and passed between processes. They should be explicitly closed to release resources, but a per-process least-recently-used cache for open files ensures that you can safely create arbitrarily large numbers of FileManager objects.

Don’t directly close files acquired from a FileManager. Instead, call FileManager.close(), which ensures that closed files are removed from the cache as well.

Example usage:

manager = FileManager(open, "example.txt", mode="w") f = manager.acquire() f.write(...) manager.close() # ensures file is closed

Note that as long as previous files are still cached, acquiring a file multiple times from the same FileManager is essentially free:

f1 = manager.acquire() f2 = manager.acquire() assert f1 is f2

__init__(opener, *args, mode=, kwargs=None, lock=None, cache=None, manager_id=None, ref_counts=None)[source]#

Initialize a CachingFileManager.

The cache, manager_id and ref_counts arguments exist solely to facilitate dependency injection, and should only be set for tests.

Parameters:

• opener (callable()) – Function that when called like opener(*args, **kwargs) returns an open file object. The file object must implement a close()method.
• *args – Positional arguments for opener. A mode argument should be provided as a keyword argument (see below). All arguments must be hashable.
• mode (optional) – If provided, passed as a keyword argument to opener along with**kwargs. mode='w' `` has special treatment: after the first call it is replaced by ``mode='a' in all subsequent function to avoid overriding the newly created file.
• kwargs (dict, optional) – Keyword arguments for opener, excluding mode. All values must be hashable.
• lock (duck-compatible threading.Lock, optional) – Lock to use when modifying the cache inside acquire() and close(). By default, uses a new threading.Lock() object. If set, this object should be pickleable.
• cache (MutableMapping, optional) – Mapping to use as a cache for open files. By default, uses xarray’s global LRU file cache. Because cache typically points to a global variable and contains non-picklable file objects, an unpickled FileManager objects will be restored with the default cache.
• manager_id (hashable, optional) – Identifier for this CachingFileManager.
• ref_counts (dict, optional) – Optional dict to use for keeping track the number of references to the same file.

Methods