Scale Up Parallel Code to Large Clusters - MATLAB & Simulink (original) (raw)
Main Content
You can scale up your parallel code to several thousand workers using different approaches. Parallel Computing Toolbox™ supports up to 2000 workers in a single parallel pool. However, when your computational tasks demand more than the supported number of workers, you can use alternative approaches to scale your code beyond parallel pool limits.
| When to Use | Recommended Approach | Description |
|---|---|---|
| You want to use an interactive pool to scale up code containing parfor orparfeval to thousands of workers. | Use parfor orparfeval in an interactive parallel pool. | Directly scale your code without modifications. Use DataQueue, poolValueStore, andFileStore objects in your code.Supports up to 2000 workers in one pool.Example: Analyze Wind Data with Large Compute Cluster |
| You want to offload and run code containingparfor or parfeval with thousands of workers on a cluster.You want to use a parallel pool on a cluster. | Run your code containing parfor orparfeval as a batch job with a parallel pool. | Offload computations to a batch job with a parallel pool and close your MATLAB® client. You can retrieve the results later.Access the batch job's ValueStore and FileStore objects.Supports up to 2000 workers in one batch pool. |
| You want to scale up parfor code to thousands of workers, but your cluster does not support parallel pools.You want to scale up parfor code to more than 2000 workers. | Run parfor directly on the cluster with parforOptions. | parfor uses workers dynamically as needed.Use this approach when your cluster does not allow communicating jobs such as parallel pools.This approach does not supportDataQueue orConstant objects and can lead to significant overheads on third-party scheduler clusters because each parfor subrange starts a MATLAB worker process.This approach allows you to scale up to 10,000 workers on MATLAB Job Scheduler clusters and supports unlimited scaling on third-party clusters.Example: Run parfor-Loops Without a Parallel Pool |
| Modify your parfor code to use a jobs and tasks workflow. | Refactor the parfor iterations into independent tasks for a job.Access the job's ValueStore andFileStore objects.This approach allows you to scale up to 10,000 workers on MATLAB Job Scheduler clusters and supports unlimited scaling on third-party clusters.Example: Scale Up with Parallel Jobs and Tasks |
See Also
parpool | parfeval | parfor | batch