LLVM: llvm::cas::ObjectStore Class Reference (original) (raw)

Content-addressable storage for objects. More...

#include "[llvm/CAS/ObjectStore.h](ObjectStore%5F8h%5Fsource.html)"

Content-addressable storage for objects.

Conceptually, objects are stored in a "unique set".

• Objects are immutable ("value objects") that are defined by their content. They are implicitly deduplicated by content.
• Each object has a unique identifier (UID) that's derived from its content, called a CASID.
  • This UID is a fixed-size (strong) hash of the transitive content of a CAS object.
  • It's comparable between any two CAS instances that have the same CASIDContext::getHashSchemaIdentifier().
  • The UID can be printed (e.g., CASID::toString()) and it can parsed by the same or a different CAS instance with ObjectStore::parseID().
• An object can be looked up by content or by UID.
  • store() is "get-or-create" methods, writing an object if it doesn't exist yet, and return a ref to it in any case.
  • loadObject(const CASID&) looks up an object by its UID.
• Objects can reference other objects, forming an arbitrary DAG.

The ObjectStore interface has a few ways of referencing objects:

• ObjectRef encapsulates a reference to something in the CAS. It is an opaque type that references an object inside a specific CAS. It is implementation defined if the underlying object exists or not for an ObjectRef, and it can used to speed up CAS lookup as an implementation detail. However, you don't know anything about the underlying objects. "Loading" the object is a separate step that may not have happened yet, and which can fail (e.g. due to filesystem corruption) or introduce latency (if downloading from a remote store).
• ObjectHandle encapulates a loaded object in the CAS. You need one of these to inspect the content of an object: to look at its stored data and references. This is internal to CAS implementation and not availble from CAS public APIs.
• CASID: the UID for an object in the CAS, obtained through ObjectStore::getID() or ObjectStore::parseID(). This is a valid CAS identifier, but may reference an object that is unknown to this CAS instance.
• ObjectProxy pairs an ObjectHandle (subclass) with a ObjectStore, and wraps access APIs to avoid having to pass extra parameters. It is the object used for accessing underlying data and refs by CAS users.

Both ObjectRef and ObjectHandle are lightweight, wrapping a uint64_t and are only valid with the associated ObjectStore instance.

There are a few options for accessing content of objects, with different lifetime tradeoffs:

• getData() accesses data without exposing lifetime at all.
• getMemoryBuffer() returns a MemoryBuffer whose lifetime is independent of the CAS (it can live longer).
• getDataString() return StringRef with lifetime is guaranteed to last as long as ObjectStore.
• readRef() and forEachRef() iterate through the references in an object. There is no lifetime assumption.

Definition at line 90 of file ObjectStore.h.

◆ ObjectStore()

◆ createProxy()

◆ createUnknownObjectError()

◆ dump()

◆ forEachRef()

Methods for handling objects.

CAS implementations need to override to provide functions to access stored CAS objects and references.

Referenced by readRefs(), and validateTree().

◆ getContext()

◆ getData()

◆ getDataSize()

◆ getDataString()

◆ getID()

◆ getMemoryBuffer()

◆ getNumRefs()

◆ getProxy() [1/2]

◆ getProxy() [2/2]

◆ getProxyIfExists()

◆ getReference()

◆ getStorageSize()

Returns

the storage size of the on-disk CAS data.

Implementations that don't have an implementation for this should return std::nullopt.

Definition at line 239 of file ObjectStore.h.

◆ importObject()

Import object from another CAS.

This will import the full tree from the other CAS.

Keeps track of the state of visitation for current node and all of its parents. Upstream Cursor holds information only from upstream CAS.

PrimaryNodeStack holds the ObjectRef of the current CAS, with nodes either just stored in the CAS or nodes already exists in the current CAS.

A map from upstream ObjectRef to current ObjectRef.

Definition at line 159 of file ObjectStore.cpp.

References llvm::ArrayRef(), assert(), llvm::SmallVectorTemplateCommon< T, typename >::back(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::end(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::find(), llvm::SmallVectorTemplateCommon< T, typename >::front(), getData(), getNumRefs(), I, LLVM_UNLIKELY, load(), ObjectStore(), llvm::Other, llvm::SmallVectorTemplateBase< T, bool >::pop_back(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), readRef(), llvm::Ref, llvm::SmallVectorTemplateCommon< T, typename >::size(), store(), llvm::SmallVectorImpl< T >::truncate(), and llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::try_emplace().

◆ isMaterialized()

Returns

true if the object is directly available from the local CAS, for implementations that have this kind of distinction.

References llvm::Ref.

◆ load()

◆ loadIfExists()

Load the object referenced by Ref.

Errors if the object cannot be loaded.

Returns

std::nullopt if the object is missing from the CAS.

References llvm::Ref.

Referenced by getProxyIfExists(), and load().

◆ parseID()

Get a [CASID](classllvm%5F1%5F1cas%5F1%5F1CASID.html "Unique identifier for a CAS object.") from a [ID](namespacellvm%5F1%5F1CallingConv.html#abdf8cf606905c10634e831390981b0ed "LLVM IR allows to use arbitrary numbers as calling convention identifiers."), which should have been generated by CASID::print().

This succeeds as long as validateID() would pass. The object may be unknown to this CAS instance.

TODO: Remove, and update callers to use validateID() or extractHashFromID().

Implemented in llvm::cas::builtin::BuiltinCAS.

◆ print()

◆ pruneStorageData()

Prune local storage to reduce its size according to the desired size limit.

Pruning can happen concurrently with other operations.

Implementations may leave this unimplemented.

Definition at line 247 of file ObjectStore.h.

References llvm::Error::success().

◆ readData()

◆ readRef()

◆ readRefs()

◆ setSizeLimit()

Set the size for limiting growth of on-disk storage.

This has an effect for when the instance is closed.

Implementations may leave this unimplemented.

Definition at line 231 of file ObjectStore.h.

References SizeLimit, and llvm::Error::success().

◆ store()

◆ storeFromOpenFile()

Default implementation reads FD and calls storeNode().

Does not take ownership of FD; the caller is responsible for closing it.

If [Status](structStatus.html) is sent in it is to be treated as a hint. Implementations must protect against the file size potentially growing after the status was taken (i.e., they cannot assume that an mmap will be null-terminated where [Status](structStatus.html) implies).

Returns the CASID and the size of the file.

Definition at line 201 of file ObjectStore.h.

References storeFromOpenFileImpl().

◆ storeFromOpenFileImpl()

◆ storeFromString()

◆ validate()

◆ validateObject()

◆ validateTree()

Validate the whole node tree.

Definition at line 135 of file ObjectStore.cpp.

References llvm::SmallVectorTemplateCommon< T, typename >::empty(), forEachRef(), getID(), I, llvm::detail::DenseSetImpl< ValueT, MapTy, ValueInfoT >::insert(), load, llvm::SmallVectorImpl< T >::pop_back_val(), llvm::SmallVectorTemplateBase< T, bool >::push_back(), llvm::Ref, llvm::Error::success(), and validateObject().

◆ ObjectProxy

The documentation for this class was generated from the following files:

• include/llvm/CAS/ObjectStore.h
• lib/CAS/ObjectStore.cpp