datastore - Create datastore for large collections of data - MATLAB (original) (raw)

Create datastore for large collections of data

Syntax

Description

[ds](#budis9p-1-ds) = datastore([location](#budis9p-1-location)) creates a datastore from the collection of data specified bylocation. A datastore is a repository for collections of data that are too large to fit in memory. After creating ds, you can read and process the data.

example

[ds](#budis9p-1-ds) = datastore([location](#budis9p-1-location),[Name,Value](#namevaluepairarguments)) specifies additional parameters for ds using one or more name-value pair arguments. For example, you can create a datastore for image files by specifying 'Type','image'.

example

Examples

collapse all

Create Datastore for Text Data

Create a datastore associated with the sample fileairlinesmall.csv. This file contains airline data from the years 1987 through 2008.

To manage the import of missing data in numeric columns, use the"TreatAsMissing" and"MissingValue" name-value arguments. Replace every instance of "NA" with a 0 in the imported data by specifying the value of "TreatAsMissing" as"NA" and the value of"MissingValue" as 0.

ds = datastore("airlinesmall.csv","TreatAsMissing","NA",... "MissingValue",0)

ds = TabularTextDatastore with properties:

                  Files: {
                         'B:\matlab\toolbox\matlab\demos\airlinesmall.csv'
                         }
                Folders: {
                         'B:\matlab\toolbox\matlab\demos'
                         }
           FileEncoding: 'UTF-8'

AlternateFileSystemRoots: {} VariableNamingRule: 'modify' ReadVariableNames: true VariableNames: {'Year', 'Month', 'DayofMonth' ... and 26 more} DatetimeLocale: en_US

Text Format Properties: NumHeaderLines: 0 Delimiter: ',' RowDelimiter: '\r\n' TreatAsMissing: 'NA' MissingValue: 0

Advanced Text Format Properties: TextscanFormats: {'%f', '%f', '%f' ... and 26 more} TextType: 'char' ExponentCharacters: 'eEdD' CommentStyle: '' Whitespace: ' \b\t' MultipleDelimitersAsOne: false

Properties that control the table returned by preview, read, readall: SelectedVariableNames: {'Year', 'Month', 'DayofMonth' ... and 26 more} SelectedFormats: {'%f', '%f', '%f' ... and 26 more} ReadSize: 20000 rows OutputType: 'table' RowTimes: []

Write-specific Properties: SupportedOutputFormats: ["txt" "csv" "xlsx" "xls" "parquet" "parq"] DefaultOutputFormat: "txt"

datastore creates aTabularTextDatastore.

Create Datastore for Image Data

Create a datastore containing all .tif files in the MATLAB® path and its subfolders.

ds = datastore(fullfile(matlabroot,"toolbox","matlab"),... "IncludeSubfolders",true,"FileExtensions",".tif","Type","image")

ds = ImageDatastore with properties: Files: { 'H:\matlab\toolbox\matlab\demos\example.tif'; 'H:\matlab\toolbox\matlab\imagesci\corn.tif' } Folders: { 'H:\matlab\toolbox\matlab' } AlternateFileSystemRoots: {} ReadSize: 1 Labels: {} SupportedOutputFormats: ["png" "jpg" "jpeg" "tif" "tiff"] DefaultOutputFormat: "png" ReadFcn: @readDatastoreImage

Input Arguments

collapse all

location — Files or folders to include in datastore

FileSet | DsFileSet object | string array | character vector | cell array of character vectors

Files or folders to include in the datastore, specified as one of these values:

• FileSet object — Specifying the location as aFileSet object leads to a faster construction time for datastores compared to specifying a path or DsFileSet object. For more information, see matlab.io.datastore.FileSet.
• DsFileSet object — For more information, see matlab.io.datastore.DsFileSet.
• File path — You can specify a single file path as a string scalar or character vector. You can specify multiple file paths as a string array or cell array of character vectors.

Files or folders can be local or remote:

• Local files or folders — If the files are not in the current folder, then specify full or relative paths. Files within subfolders of a specified folder are not automatically included in the datastore. You can use the wildcard character (*) when specifying the local path. This character specifies that the datastore include all matching files or all files in the matching folders.
• Remote files or folders — Specify full paths to remote files or folders as a uniform resource locator (URL) of the formhdfs:///_`pathtofile`_. Internet URLs must include the protocol type "http://" or"https://". For more information, see Work with Remote Data.

When you specify a folder, the datastore includes only files with supported file formats and ignores files with any other format. To specify a custom list of file extensions to include in your datastore, see the FileExtensions name-value argument.

For KeyValueDatastore, the location argument must be MAT files or Sequence files generated by themapreduce function. MAT files must be in a local file system or in a network file system. Sequence files can be in a local, network, or HDFS™ file system. For a datastore of typeDatabaseDatastore, the location argument need not be files. For more information, see DatabaseDatastore (Database Toolbox).

Example: "file1.csv"

Example: "../dir/data/file1.jpg"

Example: ["C:\dir\data\file1.xls","C:\dir\data\file2.xlsx"]

Example: "C:\dir\data\*.mat"

Example: "hdfs:///data/file1.txt"

Name-Value Arguments

Specify optional pairs of arguments asName1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: datastore("myfolder","FileExtensions"=[".jpg",".tif"]) includes all extensions with a .jpg or .tif extension for an ImageDatastore object.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: datastore("myfolder","FileExtensions",[".jpg",".tif"]) includes all extensions with a .jpg or .tif extension for an ImageDatastore object.

Type — Type of datastore

'tabulartext' | 'image' | 'spreadsheet' | 'keyvalue' | 'file' | 'tall' | ...

Type of datastore, specified as the comma-separated pair consisting of'Type' and one of the following:

• If there are multiple types that support the format of the files, then use the 'Type' argument to specify a datastore type.
• If you do not specify a value for 'Type', then datastore automatically determines the appropriate type of datastore to create based on the extensions of the files.

Data Types: char | string

IncludeSubfolders — Include subfolders within folder

true or false | 0 or 1

Include subfolders within a folder, specified as the comma-separated pair consisting of 'IncludeSubfolders' andtrue (1) or false (0). Specifytrue to include all files and subfolders within each folder or false to include only the files within each folder.

When you do not specify 'IncludeSubfolders', then the default value is false.

The 'IncludeSubfolders' name-value pair is only valid when creating these objects:

• TabularTextDatastore
• ImageDatastore
• SpreadsheetDatastore
• FileDatastore
• KeyValueDatastore
• ParquetDatastore

Example: 'IncludeSubfolders',true

Data Types: logical | double

FileExtensions — Extensions of files

character vector | cell array of character vectors | string scalar | string array

Extensions of files, specified as the comma-separated pair consisting of 'FileExtensions' and a character vector, cell array of character vectors, string scalar, or string array. When specifying 'FileExtensions', also specify'Type'. You can use the empty quotes'' to represent files without extensions.

If 'FileExtensions' is not specified, thendatastore automatically includes all supported file extensions depending on the datastore type. If you want to include unsupported extensions, then specify each extension you want to include individually.

• For TabularTextDatastore objects, supported extensions include .txt,.csv, .dat,.dlm, .asc,.text, and no extension.
• For ImageDatastore objects, supported extensions include all imformats extensions.
• For SpreadsheetDatastore objects, supported extensions include .xls,.xlsx, .xlsm,.xltx, and.xltm.
• For TallDatastore objects, supported extensions include .mat and.seq.
• For ParquetDatastore objects, supported extensions include .parquet and.parq.

The 'FileExtensions' name-value pair is only valid when creating these objects:

• TabularTextDatastore
• ImageDatastore
• SpreadsheetDatastore
• FileDatastore
• KeyValueDatastore
• ParquetDatastore

Example: 'FileExtensions','.jpg'

Example: 'FileExtensions',{'.txt','.text'}

Data Types: char | cell | string

AlternateFileSystemRoots — Alternate file system root paths

string vector | cell array

Alternate file system root paths, specified as the name-value argument consisting of"AlternateFileSystemRoots" and a string vector or a cell array. Use"AlternateFileSystemRoots" when you create a datastore on a local machine, but need to access and process the data on another machine (possibly of a different operating system). Also, when processing data using the Parallel Computing Toolbox™ and the MATLAB® Parallel Server™, and the data is stored on your local machines with a copy of the data available on different platform cloud or cluster machines, you must use"AlternateFileSystemRoots" to associate the root paths.

• To associate a set of root paths that are equivalent to one another, specify"AlternateFileSystemRoots" as a string vector. For example,
  ["Z:\datasets","/mynetwork/datasets"]
• To associate multiple sets of root paths that are equivalent for the datastore, specify "AlternateFileSystemRoots" as a cell array containing multiple rows where each row represents a set of equivalent root paths. Specify each row in the cell array as either a string vector or a cell array of character vectors. For example:
  • Specify "AlternateFileSystemRoots" as a cell array of string vectors.
    {["Z:\datasets", "/mynetwork/datasets"];...
    ["Y:\datasets", "/mynetwork2/datasets","S:\datasets"]}
  • Alternatively, specify "AlternateFileSystemRoots" as a cell array of cell array of character vectors.
    {{'Z:\datasets','/mynetwork/datasets'};...
    {'Y:\datasets', '/mynetwork2/datasets','S:\datasets'}}

The value of "AlternateFileSystemRoots" must satisfy these conditions:

• Contains one or more rows, where each row specifies a set of equivalent root paths.
• Each row specifies multiple root paths and each root path must contain at least two characters.
• Root paths are unique and are not subfolders of one another.
• Contains at least one root path entry that points to the location of the files.

For more information, see Set Up Datastore for Processing on Different Machines or Clusters.

Example: ["Z:\datasets","/mynetwork/datasets"]

Data Types: string | cell

TextType — Output data type of text variables

'char' (default) | 'string'

Output data type of text variables, specified as the comma-separated pair consisting of 'TextType' and either'char' or 'string'. If the output table from the read,readall, or preview functions contains text variables, then 'TextType' specifies the data type of those variables forTabularTextDatastore andSpreadsheetDatastore objects only. If'TextType' is 'char', then the output is a cell array of character vectors. If'TextType' is 'string', then the output has type string.

Data Types: char | string

DatetimeType — Type for imported date and time data

'datetime' (default) | 'text'

Type for imported date and time data, specified as the comma-separated pair consisting of 'DatetimeType' and one of these values: 'datetime' or 'text'. The'DatetimeType' argument only applies when creating a TabularTextDatastore object.

Example: 'DatetimeType','datetime'

Data Types: char | string

DurationType — Output data type of duration data

'duration' (default) | 'text'

Output data type of duration data from text files, specified as the comma-separated pair consisting of 'DurationType' and either 'duration' or 'text'.

Data Types: char | string | datetime

VariableNamingRule — Flag to preserve variable names

"modify" (default) | "preserve"

Flag to preserve variable names, specified as either "modify" or"preserve".

• "modify" — Convert invalid variable names (as determined by the isvarname function) to valid MATLAB identifiers.
• "preserve" — Preserve variable names that are not valid MATLAB identifiers such as variable names that include spaces and non-ASCII characters.

Starting in R2019b, variable names and row names can include any characters, including spaces and non-ASCII characters. Also, they can start with any characters, not just letters. Variable and row names do not have to be valid MATLAB identifiers (as determined by the isvarname function). To preserve these variable names and row names, set the value of VariableNamingRule to "preserve". Variable names are not refreshed when the value of VariableNamingRule is changed from "modify" to "preserve".

Data Types: char | string

In addition to these name-value pairs, you also can specify any of the properties of the following objects as name-value pairs, except for the Files property:

• TabularTextDatastore
• ImageDatastore
• SpreadsheetDatastore
• KeyValueDatastore
• TallDatastore
• DatabaseDatastore (Database Toolbox)

Output Arguments

collapse all

ds — Datastore for collection of data

TabularTextDatastore | ImageDatastore | SpreadsheetDatastore | KeyValueDatastore | FileDatastore | TallDatastore | ...

Datastore for a collection of data, returned as one of these objects:TabularTextDatastore,ImageDatastore,SpreadsheetDatastore,KeyValueDatastore, FileDatastore,TallDatastore, ParquetDatastore, or DatabaseDatastore. The type of the datastore depends on the type of files or the location argument. For more information, click the datastore name in the following table:

For each of these datastore types, the Files property is a cell array of character vectors. Each character vector is an absolute path to a file resolved by the location argument.

Limitations

• MATLAB does not support internet URLs that require authentication.
• MATLAB Online™ supports internet URLs associated with Microsoft® OneDrive™ files and folders, while the installed version of MATLAB supports only local OneDrive files.

Version History

Introduced in R2014b

expand all

R2024b: Read data over HTTP and HTTPS using datastore functions

You can read data from primary online sources by performing datastore operations over an internet URL.