partition - Partition minibatchqueue - MATLAB (original) (raw)
Main Content
Partition minibatchqueue
Since R2020b
Syntax
Description
[submbq](#mw%5Fc7da613b-7ba4-440d-88fa-84524e74e2ff) = partition([mbq](#mw%5F3d7ade8d-0d0f-490b-a1fb-fac0c10a5cab%5Fsep%5Fmw%5F1052debc-abad-4547-9b2c-423b954fdb8a),[numParts](#mw%5F51ecd613-7adf-462a-8852-df8315464e02),[indx](#mw%5F6791060f-bb70-4e2b-95c0-1a847ef5202a)) partitions the minibatchqueue objectmbq into numParts parts and returns the partition corresponding to the index indx. The properties ofsubmbq are the same as the properties of mbq.
The output minibatchqueue object has access only to the partition of data it is given when it is created. Using reset withsubmbq resets the minibatchqueue object to the start of the data partition. Using shuffle withsubmbq shuffles only the partitioned data. If you want to shuffle the data across multiple partitions, you must shuffle the originalminibatchqueue object and then re-partition.
Examples
Partition minibatchqueue
Use the partition function to divide aminibatchqueue object into three parts.
Create a minibatchqueue object from a datastore.
ds = digitDatastore; mbq = minibatchqueue(ds)
mbq = minibatchqueue with 1 output and properties:
Mini-batch creation: MiniBatchSize: 128 PartialMiniBatch: 'return' MiniBatchFcn: 'collate' PreprocessingEnvironment: 'serial'
Outputs: OutputCast: {'single'} OutputAsDlarray: 1 MiniBatchFormat: {''} OutputEnvironment: {'auto'}
Partition the minibatchqueue object into three parts and return the first partition.
sub1 = partition(mbq,3,1)
sub1 = minibatchqueue with 1 output and properties:
Mini-batch creation: MiniBatchSize: 128 PartialMiniBatch: 'return' MiniBatchFcn: 'collate' PreprocessingEnvironment: 'serial'
Outputs: OutputCast: {'single'} OutputAsDlarray: 1 MiniBatchFormat: {''} OutputEnvironment: {'auto'}
sub1 contains approximately the first third of the data inmbq.
Partition minibatchqueue in Parallel
Use the partition function to divide aminibatchqueue object into three parts.
Create a minibatchqueue object from a datastore.
ds = digitDatastore; mbq = minibatchqueue(ds)
mbq = minibatchqueue with 1 output and properties:
Mini-batch creation: MiniBatchSize: 128 PartialMiniBatch: 'return' MiniBatchFcn: 'collate' PreprocessingEnvironment: 'serial'
Outputs: OutputCast: {'single'} OutputAsDlarray: 1 MiniBatchFormat: {''} OutputEnvironment: {'auto'}
Partition the minibatchqueue object into three parts on three workers in a parallel pool. Iterate over the data on each worker.
numWorkers = 3; p = parpool("Processes",numWorkers); parfor i=1:3 submbq = partition(mbq,3,i); while hasdata(submbq) data = next(submbq); end end
Each worker has access to a subset of the data in the originalminibatchqueue object.
Input Arguments
mbq — Mini-batch queue
minibatchqueue object
Mini-batch queue, specified as a minibatchqueue object.
numParts — Number of partitions
numeric scalar
Number of partitions, specified as a numeric scalar.
indx — Partition index
numeric scalar
Partition index, specified as a numeric scalar.
Output Arguments
submbq — Output minibatchqueue
minibatchqueue
Output minibatchqueue, specified as aminibatchqueue object. submbq contains a subset of the data in mbq. The properties of submbq are the same as the properties of mbq.
Version History
Introduced in R2020b