Select Datastore for File Format or Application - MATLAB & Simulink (original) (raw)
A datastore is a repository for collections of data that are too large to fit in memory. Each file format and application uses a different type of datastore, which contains properties pertinent to the type of data or application that it supports. MATLAB® provides datastores for standard file formats such as Excel® files and datastores for specific applications such as Deep Learning. In addition to the existing datastores, if your data is in a proprietary format, then you can develop a customized datastore using the custom datastore framework.
Datastores for Standard File Formats
For a collection of data in standard file format use one of these options.
Transform or combine existing datastores.
Datastores to integrate with MapReduce and tall arrays.
| Datastore | Description |
|---|---|
| KeyValueDatastore | Key-value pair data that are inputs to or outputs ofmapreduce |
| TallDatastore | Datastore for checkpointing tall arrays |
Datastores for Specific Applications
Based on your application use one of these datastores.
| Application | Datastore | Description | Toolbox Required |
|---|---|---|---|
| Simulink Model Data | SimulationDatastore (Simulink) | Datastore for simulation input and output data that you use with a Simulink® model | Simulink |
| Simulation Ensemble and Predictive Maintenance Data | simulationEnsembleDatastore (Predictive Maintenance Toolbox) | Datastore to manage simulation ensemble data | Predictive Maintenance Toolbox™ |
| fileEnsembleDatastore (Predictive Maintenance Toolbox) | Datastore to manage ensemble data in custom file format | Predictive Maintenance Toolbox | |
| Measurement Data Format (MDF) Files | mdfDatastore (Vehicle Network Toolbox) | Datastore for collection of MDF files | Vehicle Network Toolbox™ |
| mdfDatastore (Powertrain Blockset) | Datastore for collection of MDF files | Powertrain Blockset™ | |
| Deep LearningDatastores for preprocessing image or sequence data | pixelLabelDatastore (Computer Vision Toolbox) | Datastore for pixel label data | Computer Vision Toolbox™ and Deep Learning Toolbox™ |
| boxLabelDatastore (Computer Vision Toolbox) | Datastore for bounding box label data | Computer Vision Toolbox and Deep Learning Toolbox | |
| signalDatastore (Signal Processing Toolbox) | Datastore for collection of signal files | Signal Processing Toolbox™ and Deep Learning Toolbox | |
| randomPatchExtractionDatastore (Image Processing Toolbox) | Datastore for extracting random patches from images or pixel label imagesDatastore is nondeterministic | Image Processing Toolbox™ and Deep Learning Toolbox | |
| denoisingImageDatastore (Image Processing Toolbox) | Datastore to train an image denoising deep neural networkDatastore is nondeterministic | Image Processing Toolbox and Deep Learning Toolbox | |
| augmentedImageDatastore (Deep Learning Toolbox) | Datastore for resizing and augmenting training imagesDatastore is nondeterministic | Deep Learning Toolbox | |
| Audio Data | audioDatastore (Audio Toolbox) | Datastore for collection of audio files | Audio Toolbox™ |
| Out-of-Memory Image Data | blockedImageDatastore (Image Processing Toolbox) | Datastore to manage blocks of a single image that is too large to fit in memory | Image Processing Toolbox |
| Database Data | databaseDatastore (Database Toolbox) | Datastore for collections of data in a relational database | Database Toolbox™ |
Custom File Formats
For a collection of data in a custom file format, if each individual file fits in memory, use FileDatastore along with your custom file reading function. Otherwise, develop your own fully customized datastore for custom or proprietary data using the matlab.io.Datastore class. See Develop Custom Datastore.
Nondeterministic Datastores
Datastores that do not return the exact same data for a call to the read function after a call to the reset function are nondeterministic datastores. Do not use nondeterministic datastores with tall arrays, mapreduce, or any other code that requires reading the data more than once.
Some applications require data that is randomly augmented or transformed. For example, the augmentedImageDatastore (Deep Learning Toolbox) datastore, from the deep learning application augments training image data with randomized preprocessing operations to help prevent the network from overfitting and memorizing the exact details of the training images. The output of this datastore is different every time you perform a read operation after a call to reset.
See Also
TabularTextDatastore | SpreadsheetDatastore | ImageDatastore | FileDatastore | TallDatastore | tall