LLVM: llvm::cas::ondisk::UnifiedOnDiskCache Class Reference (original) (raw)

A unified CAS nodes and key-value database, using on-disk storage for both. More...

#include "[llvm/CAS/UnifiedOnDiskCache.h](UnifiedOnDiskCache%5F8h%5Fsource.html)"

Public Member Functions
OnDiskGraphDB &	getGraphDB ()
	The OnDiskGraphDB instance for the open directory.
OnDiskKeyValueDB &	getKeyValueDB ()
	The OnDiskGraphDB instance for the open directory.
LLVM_ABI_FOR_TEST Error	close (bool CheckSizeLimit=true)
	This is called implicitly at destruction time, so it is not required for a client to call this.
LLVM_ABI_FOR_TEST void	setSizeLimit (std::optional< uint64_t > SizeLimit)
	Set the size for limiting growth.
LLVM_ABI_FOR_TEST uint64_t	getStorageSize () const
LLVM_ABI_FOR_TEST bool	hasExceededSizeLimit () const
bool	needsGarbageCollection () const
Error	collectGarbage ()
	Remove unused data from the current UnifiedOnDiskCache.
LLVM_ABI_FOR_TEST	~UnifiedOnDiskCache ()

Static Public Member Functions
static LLVM_ABI_FOR_TEST Expected< std::unique_ptr< UnifiedOnDiskCache > >	open (StringRef Path, std::optional< uint64_t > SizeLimit, StringRef HashName, unsigned HashByteSize, OnDiskGraphDB::FaultInPolicy FaultInPolicy=OnDiskGraphDB::FaultInPolicy::FullTree)
	Open a UnifiedOnDiskCache instance for a directory.
static Expected< ValidationResult >	validateIfNeeded (StringRef Path, StringRef HashName, unsigned HashByteSize, bool CheckHash, bool AllowRecovery, bool ForceValidation, std::optional< StringRef > LLVMCasBinary)
	Validate the data in Path, if needed to ensure correctness.
static LLVM_ABI_FOR_TEST Error	collectGarbage (StringRef Path)
	Remove any unused data from the directory at Path.
static LLVM_ABI_FOR_TEST ObjectID	getObjectIDFromValue (ArrayRef< char > Value)
	Helper function to convert the value stored in KeyValueDB and ObjectID.
static LLVM_ABI_FOR_TEST ValueBytes	getValueFromObjectID (ObjectID ID)

Friends
class	OnDiskGraphDB
class	OnDiskKeyValueDB

A unified CAS nodes and key-value database, using on-disk storage for both.

It manages storage growth and provides APIs for garbage collection.

High-level properties:

While [UnifiedOnDiskCache](classllvm%5F1%5F1cas%5F1%5F1ondisk%5F1%5F1UnifiedOnDiskCache.html "A unified CAS nodes and key-value database, using on-disk storage for both.") is open on a directory, by any process, the storage size in that directory will keep growing unrestricted. For data to become eligible for garbage-collection there should be no open instances of [UnifiedOnDiskCache](classllvm%5F1%5F1cas%5F1%5F1ondisk%5F1%5F1UnifiedOnDiskCache.html "A unified CAS nodes and key-value database, using on-disk storage for both.") for that directory, by any process.
Garbage-collection needs to be triggered explicitly by the client. It can be triggered on a directory concurrently, at any time and by any process, without affecting any active readers/writers, in the same process or other processes.

Usage patterns should be that an instance of [UnifiedOnDiskCache](classllvm%5F1%5F1cas%5F1%5F1ondisk%5F1%5F1UnifiedOnDiskCache.html "A unified CAS nodes and key-value database, using on-disk storage for both.") is open for a limited period of time, e.g. for the duration of a build operation. For long-living processes that need periodic access to a [UnifiedOnDiskCache](classllvm%5F1%5F1cas%5F1%5F1ondisk%5F1%5F1UnifiedOnDiskCache.html "A unified CAS nodes and key-value database, using on-disk storage for both."), the client should devise a scheme where access is performed within some defined period. For example, if a service is designed to continuously wait for requests that access a [UnifiedOnDiskCache](classllvm%5F1%5F1cas%5F1%5F1ondisk%5F1%5F1UnifiedOnDiskCache.html "A unified CAS nodes and key-value database, using on-disk storage for both."), it could keep the instance alive while new requests are coming in but close it after a time period in which there are no new requests.

Definition at line 41 of file UnifiedOnDiskCache.h.

◆ ValueBytes

UnifiedOnDiskCache::~UnifiedOnDiskCache	(	)

◆ close()

Error UnifiedOnDiskCache::close	(	bool	CheckSizeLimit = true	)

This is called implicitly at destruction time, so it is not required for a client to call this.

After calling close the only method that is valid to call is needsGarbageCollection.

Parameters

CheckSizeLimit	if true it will check whether the primary store has exceeded its intended size limit. If false the check is skipped even if a SizeLimit was passed to the open call.

Definition at line 514 of file UnifiedOnDiskCache.cpp.

References assert(), llvm::sys::fs::closeFile(), llvm::sys::fs::convertFDToNativeFile(), llvm::sys::fs::create_directory(), llvm::createFileError(), llvm::sys::fs::Exclusive, llvm::sys::path::get_separator(), getNextDBDirName(), hasExceededSizeLimit(), llvm::make_scope_exit(), llvm::no_lock_available, llvm::Error::success(), llvm::cas::ondisk::tryLockFileThreadSafe(), and llvm::cas::ondisk::unlockFileThreadSafe().

Referenced by ~UnifiedOnDiskCache().

◆ collectGarbage() [1/2]

Error UnifiedOnDiskCache::collectGarbage	(	)

◆ collectGarbage() [2/2]

◆ getGraphDB()

OnDiskGraphDB & llvm::cas::ondisk::UnifiedOnDiskCache::getGraphDB ( )	inline

◆ getKeyValueDB()

OnDiskKeyValueDB & llvm::cas::ondisk::UnifiedOnDiskCache::getKeyValueDB ( )	inline

◆ getObjectIDFromValue()

◆ getStorageSize()

uint64_t UnifiedOnDiskCache::getStorageSize	(	)	const

◆ getValueFromObjectID()

UnifiedOnDiskCache::ValueBytes UnifiedOnDiskCache::getValueFromObjectID ( ObjectID ID)	static

◆ hasExceededSizeLimit()

bool UnifiedOnDiskCache::hasExceededSizeLimit	(	)	const

Returns

whether the primary store has exceeded the intended size limit. This can return false even if the overall size of the opened directory is over the SizeLimit passed to open. To know whether garbage collection needs to be triggered or not, call needsGarbaseCollection.

Definition at line 489 of file UnifiedOnDiskCache.cpp.

Referenced by close().

◆ needsGarbageCollection()

bool llvm::cas::ondisk::UnifiedOnDiskCache::needsGarbageCollection ( ) const	inline

Returns

whether there are unused data that can be deleted using a collectGarbage call.

Definition at line 119 of file UnifiedOnDiskCache.h.

◆ open()

Open a [UnifiedOnDiskCache](classllvm%5F1%5F1cas%5F1%5F1ondisk%5F1%5F1UnifiedOnDiskCache.html "A unified CAS nodes and key-value database, using on-disk storage for both.") instance for a directory.

Parameters

Path	directory for the on-disk database. The directory will be created if it doesn't exist.
SizeLimit	Optional size for limiting growth. This has an effect for when the instance is closed.
HashName	Identifier name for the hashing algorithm that is going to be used.
HashByteSize	Size for the object digest hash bytes.
FaultInPolicy	Controls how nodes are copied to primary store. This is recorded at creation time and subsequent opens need to pass the same policy otherwise the open will fail.

If there is only one directory open databases on it. If there are 2 or more directories, get the most recent directories and chain them, with the most recent being the primary one. The remaining directories are unused data than can be garbage-collected.

Definition at line 390 of file UnifiedOnDiskCache.cpp.

References llvm::sys::path::append(), assert(), llvm::sys::fs::CD_OpenAlways, llvm::sys::fs::create_directories(), llvm::createFileError(), DBDirPrefix, getAllDBDirs(), llvm::cas::ondisk::lockFileThreadSafe(), llvm::sys::fs::OF_None, llvm::cas::ondisk::OnDiskGraphDB::open(), llvm::cas::ondisk::OnDiskKeyValueDB::open(), llvm::sys::fs::openFileForReadWrite(), and llvm::sys::fs::Shared.

Referenced by llvm::cas::builtin::createBuiltinUnifiedOnDiskCache(), and validateInProcess().

◆ setSizeLimit()

void UnifiedOnDiskCache::setSizeLimit	(	std::optional< uint64_t >	SizeLimit	)

Set the size for limiting growth.

This has an effect for when the instance is closed.

Definition at line 472 of file UnifiedOnDiskCache.cpp.

◆ validateIfNeeded()

Validate the data in Path, if needed to ensure correctness.

Note: if invalid data is detected and AllowRecovery is true, then recovery requires exclusive access to the CAS and it is an error to attempt recovery if there is concurrent use of the CAS.

Parameters

Path	directory for the on-disk database.
HashName	Identifier name for the hashing algorithm that is going to be used.
HashByteSize	Size for the object digest hash bytes.
CheckHash	Whether to validate hashes match the data.
AllowRecovery	Whether to automatically recover from invalid data by marking the files for garbage collection.
ForceValidation	Whether to force validation to occur even if it should not be necessary.
LLVMCasBinary	If provided, validation is performed out-of-process using the given llvm-cas executable which protects against crashes during validation. Otherwise validation is performed in-process.

Returns

Valid if the data is already valid, Recovered if data was invalid but has been cleared, Skipped if validation is not needed, or an [Error](classllvm%5F1%5F1Error.html "Lightweight error class with error context and mandatory checking.") if validation cannot be performed or if the data is left in an invalid state because AllowRecovery is false.

Definition at line 270 of file UnifiedOnDiskCache.cpp.

References llvm::sys::path::append(), assert(), llvm::SmallString< InternalLen >::assign(), llvm::sys::fs::CD_OpenAlways, llvm::sys::fs::closeFile(), llvm::consumeError(), llvm::sys::fs::convertFDToNativeFile(), CorruptPrefix, llvm::sys::fs::create_directories(), llvm::createFileError(), llvm::createStringError(), llvm::directory_not_empty, llvm::raw_fd_ostream::error(), llvm::sys::fs::Exclusive, llvm::file_exists, getAllDBDirs(), llvm::cas::ondisk::getBootTime(), llvm::raw_fd_ostream::has_error(), llvm::illegal_byte_sequence, llvm::cas::ondisk::lockFileThreadSafe(), llvm::make_scope_exit(), llvm::sys::fs::OF_None, llvm::sys::fs::openFileForReadWrite(), llvm::sys::fs::readNativeFileToEOF(), llvm::cas::Recovered, llvm::sys::path::remove_filename(), llvm::sys::fs::rename(), llvm::sys::fs::resize_file(), llvm::raw_fd_ostream::seek(), llvm::cas::Skipped, llvm::cas::ondisk::tryLockFileThreadSafe(), llvm::cas::ondisk::unlockFileThreadSafe(), llvm::cas::Valid, validateInProcess(), validateOutOfProcess(), and ValidationFilename.

Referenced by llvm::cas::validateOnDiskUnifiedCASDatabasesIfNeeded().

◆ OnDiskGraphDB

◆ OnDiskKeyValueDB

The documentation for this class was generated from the following files:

include/llvm/CAS/UnifiedOnDiskCache.h
lib/CAS/UnifiedOnDiskCache.cpp