ValueStore - Data storage shared by MATLAB clients and workers - MATLAB (original) (raw)

Data storage shared by MATLAB clients and workers

Since R2022a

Description

ValueStore is an object that stores data owned by a specific job. Each entry of the object consists of a value and its corresponding key. When the owning job is deleted, the ValueStore object is deleted as well. UseValueStore to store data from MATLAB® workers that can be retrieved by MATLAB clients during the execution of a job (even while the job is still running).

• Any MATLAB process client or worker can write an entry to theValueStore at any time. Any MATLAB process client or worker can then read this entry from theValueStore at any time. However, the ordering of operations executed by different processes is not guaranteed.
• ValueStore is not held in system memory, so it can be used to store large results.

Creation

The ValueStore object is automatically created when you create:

• A job on a cluster, which is a parallel.Job object. To create a job, use the batch, createJob, or createCommunicatingJob function.
• A parallel pool of process workers on the local machine, which is a ProcessPool object. To create a process pool, use the parpool function.
• A parallel pool of thread workers on the local machine, which is a ThreadPool object. To create a thread pool, use the parpool function. (since R2023b)
• A parallel pool of workers on a cluster of machines, which is a ClusterPool object. To create a cluster pool, use the parpool function.

You can access the ValueStore object on a worker by using the getCurrentValueStore function. You can then retrieve theValueStore object on a client by using the ValueStore property that is associated with the job or the parallel pool. For example, see Run Batch Job and Retrieve Data from Workers.

Properties

expand all

KeyUpdatedFcn — Callback executed when entry is added or replaced

function handle

Callback executed when an entry is added or replaced, specified as a function handle. The function handle must accept two input arguments that represent the ValueStore object and its key when an entry is added or replaced.

KeyRemovedFcn — Callback executed when entry is removed

function handle

Callback executed when an entry is removed, specified as a function handle. The function handle must accept two input arguments that represent the ValueStore object and its key when an entry is removed.

Object Functions

Examples

collapse all

Run Batch Job and Retrieve Data from Workers

Run a simulation on workers and retrieve the data storage of the job on a client. The data storage is a ValueStore object with key-value entries.

The following simulation finds the singular values of random matrices and stores the results in the ValueStore object.

function workerSvdCode(models) % Get the ValueStore of the current job store = getCurrentValueStore; for i = 1:numel(models) % Store simulation results in the ValueStore object pause(1) key = strcat("result_",num2str(i)); store(key) = svd(rand(models(i))); store("progress") = i/numel(models); end end

The following callback function is executed when an entry is added to the ValueStore object.

function handleNewEntry(store,key) if strcmp(key,"progress") fprintf("Progress update: %.2f %%\n",store(key)*100); else fprintf("Result %s added\n",key); end end

Run a batch job on workers using the default cluster profile.

models = [8,16,32,20]; c = parcluster; job = batch(c,@workerSvdCode,0,{models});

Retrieve the ValueStore object on the client while the job is still running. Show the progress of the job.

store = job.ValueStore; store.KeyUpdatedFcn = @handleNewEntry; wait(job);

Result result_1 added Progress update: 25.00 % Result result_2 added Progress update: 50.00 % Result result_3 added Progress update: 75.00 % Result result_4 added Progress update: 100.00 %

Get the entry value as specified by the key "result_1" from the object.

val1 =

4.3318
1.2988
1.1040
0.8813
0.5711
0.3991
0.2092
0.1048

Run Simulation on Parallel Pool of Process Workers and Retrieve Data

Run simulation on a parallel pool of process workers and retrieve the data storage on a client.

The following simulation finds the inverse of random matrices and stores the results in the ValueStore object.

function workerInvCode(models) % Get the ValueStore of the current job store = getCurrentValueStore; for i = 1:numel(models) % Store simulation results in the ValueStore object pause(1); key = strcat("result_",num2str(i)); store(key) = inv(rand(models(i))); end end

The following callback function is executed when an entry is added to the ValueStore object.

function handleUpdatedEntry(store,key) fprintf("Result %s added\n",key); end

Start a parallel pool of process workers.

pool = parpool('Processes');

Starting parallel pool (parpool) using the 'Processes' profile ... Connected to parallel pool with 6 workers.

Get the ValueStore for this pool and assign the callback function to be executed when an entry is added.

store = pool.ValueStore; store.KeyUpdatedFcn = @handleUpdatedEntry;

Run the simulation on the pool.

models = [4,8,32,20]; future = parfeval(@workerInvCode,0,models); wait(future);

Result result_1 added Result result_2 added Result result_3 added Result result_4 added

Run Independent Tasks and Retrieve Data and Files from All Tasks

Run a job of independent tasks. Then, retrieve the data and file storage of the job on a client.

The following simulation finds the permutations and combinations of a vector, and stores the results in the ValueStore and FileStore objects.

function taskFunction(dataset,keyname) % Get the ValueStore and FileStore of the current job valueStore = getCurrentValueStore; fileStore = getCurrentFileStore; % Run the simulation to find permutation and combination [result,logFile] = runSimulation(dataset); % Store results in ValueStore to release system memory valueStore(keyname) = result; % Copy file to FileStore to retrieve the file from non-shared file system copyFileToStore(fileStore,logFile,keyname); end

function [result,logFile] = runSimulation(dataset) permutations = perms(dataset{1}); combinations = nchoosek(dataset{1},dataset{2}); result.N_perm = length(permutations); result.N_comb = length(combinations); logFile = strcat(tempname("C:\myLogFolder"),".mat"); save(logFile,"permutations","combinations") end

Create a job using the default cluster profile.

c = parcluster; job = createJob(c);

Create independent tasks for the job. Each task runs the simulation with the given input.

set_1 = {[12,34,54],2}; set_2 = {[45,33],1}; set_3 = {[12,12,12,13,14],3}; tasks = createTask(job,@taskFunction,0,{{set_1,"sim_1"},{set_2,"sim_2"},{set_3,"sim_3"}});

Run the job and wait for it to finish.

Retrieve the data and file storage of the job.

valueStore = job.ValueStore; fileStore = job.FileStore;

Show the result of the third task that is stored in the ValueStore object.

result_3 = valueStore("sim_3")

result_3 = struct with fields: N_perm: 120 N_comb: 10

Copy files from the file storage as specified by the corresponding keys "sim_1" and "sim_2" to the local files "analysis_1.mat" and "analysis_2.mat".

copyFileFromStore(fileStore,["sim_1" "sim_2"],["analysis_1.mat" "analysis_2.mat"]);

Display all the information on the variables stored in the local files.

whos -file 'analysis_1.mat'

Name Size Bytes Class Attributes

combinations 3x2 48 double
permutations 6x3 144 double

whos -file 'analysis_2.mat'

Name Size Bytes Class Attributes

combinations 2x1 16 double
permutations 2x2 32 double

Limitations

• When using parallel.cluster.Generic clusters with'HasSharedFileSystem' set to false, the visibility of modifications made to ValueStore while a job is running depends on your specific implementation. Without additional synchronization between the MATLAB client and worker JobStorageLocation, changes might only be visible once the job has completed.

Version History

Introduced in R2022a

expand all

R2023b: Use ValueStore on thread-based parallel pools

You can now use ValueStore on ThreadPool objects.

ValueStore is not supported on MATLABbackgroundPool.