FileDatastore - Datastore with custom file reader - MATLAB (original) (raw)

Datastore with custom file reader

Description

Use a FileDatastore object to manage large collections of custom format files where the collection does not necessarily fit in memory or when a large custom format file does not fit in memory. You can create aFileDatastore object using the fileDatastore function, specify its properties, and then import and process the data using object functions.

Creation

Syntax

Description

`fds` = fileDatastore([location](#d126e337526),"ReadFcn",[@fcn](#mw%5F8d4186c6-c0c4-4e67-a84c-82c9d5ece3be)) creates a datastore from the collection of files specified bylocation and uses the function fcn to read the data from the files.

`fds` = fileDatastore([location](#d126e337526),"ReadFcn",[@fcn](#mw%5F8d4186c6-c0c4-4e67-a84c-82c9d5ece3be),[Name,Value](#namevaluepairarguments)) specifies additional parameters and properties for fds using one or more name-value pair arguments. For example, you can specify which files to include in the datastore depending on their extensions withfileDatastore(location,"ReadFcn",@customreader,"FileExtensions",[".exts",".extx"]).

example

Input Arguments

expand all

location — Files or folders to include in datastore

FileSet | DsFileSet object | string array | character vector | cell array of character vectors

Files or folders to include in the datastore, specified as one of these values:

• FileSet object — Specifying the location as aFileSet object leads to a faster construction time for datastores compared to specifying a path or DsFileSet object. For more information, see matlab.io.datastore.FileSet.
• DsFileSet object — For more information, see matlab.io.datastore.DsFileSet.
• File path — You can specify a single file path as a string scalar or character vector. You can specify multiple file paths as a string array or cell array of character vectors.

Files or folders can be local or remote:

• Local files or folders — If the files are not in the current folder, then specify full or relative paths. Files within subfolders of a specified folder are not automatically included in the datastore. You can use the wildcard character (*) when specifying the local path. This character specifies that the datastore include all matching files or all files in the matching folders.
• Remote files or folders — Specify full paths to remote files or folders as a uniform resource locator (URL) of the formhdfs:///_`pathtofile`_. Internet URLs must include the protocol type "http://" or"https://". For more information, see Work with Remote Data.

When you specify a folder, the datastore includes only files with supported file formats and ignores files with any other format. To specify a custom list of file extensions to include in your datastore, see the FileExtensions name-value argument.

Example: "file1.ext"

Example: "../dir/data/file1.ext"

Example: ["C:\dir\data\file1.exts","C:\dir\data\file2.extx"]

Example: "C:\dir\data\*.ext"

@fcn — Function that reads file data

function handle

Function that reads the file data, specified as a function handle.

The signature of the function represented by the function handle@fcn depends on the value of the specifiedReadMode. The function that reads the file data must confirm to one of these signatures.

The value specified in @fcn, sets the value of theReadFcn property.

Example: @customreader

Data Types: function_handle

Name-Value Arguments

Specify optional pairs of arguments asName1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: fds = fileDatastore("C:\dir\data",FileExtensions=[".exts",".extx"])

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: fds = fileDatastore("C:\dir\data","FileExtensions",[".exts",".extx"])

IncludeSubfolders — Subfolder inclusion flag

true or false | 0 or 1

Subfolder inclusion flag, specified as the comma-separated pair consisting of "IncludeSubfolders" andtrue, false, 0, or 1. Specify true to include all files and subfolders within each folder or false to include only the files within each folder.

If you do not specify "IncludeSubfolders", then the default value is false.

Example: "IncludeSubfolders",true

Data Types: logical | double

FileExtensions — Custom format file extensions

character vector | cell array of character vectors | string scalar | string array

Custom format file extensions, specified as the comma-separated pair consisting of "FileExtensions" and a character vector, cell array of character vectors, string scalar, or string array.

When you specify a file extension, thefileDatastore function creates a datastore object only for files with the specified extension. You also can create a datastore for files without any extensions by specifying"FileExtensions" as an empty character vector, ''. If you do not specify"FileExtensions", thenfileDatastore automatically includes all files within a folder.

Example: "FileExtensions",''

Example: "FileExtensions",".ext"

Example: "FileExtensions",[".exts",".extx"]

Data Types: char | cell | string

PreviewFcn — Function to preview input data

@ReadFcn (default) | function handle

Function to preview the input data, specified as a function handle.

If you do not specify a preview function,FileDatastore uses the value specified in@ReadFcn as the default preview function. Alternatively, you can specify your own custom preview function for your data.

• @ReadFcn (default) — UseReadFcn to sampleFileDatastore data. This option can lead to slower performance fortall construction.
• Function handle — Use your custom preview function for FileDatastore and tall construction to sample the input data. Use PreviewFcn to provide a function that reads only the minimum needed part of input data for preview and tall construction.

The function specified by PreviewFcn must return values with the same data types that theReadFcn returns.

Data Types: function_handle

ReadMode — Portion of the file to read

"file" (default) | "partialfile" | "bytes"

Portion of the file to read, specified as"file", "partialfile", or"bytes".

To use the subset andshuffle functions on aFileDatastore object, you must set"ReadMode" to"file".

Data Types: char | string

BlockSize — Number of bytes to read

positive integer

Number of bytes to read with every read operation, specified as a positive integer.

To ensure that you can distribute multiple blocks of a file across multiple parallel MATLAB® workers, specify BlockSize as a positive integer greater than 131072 bytes (128 kilobytes).

To specify or to change the value of BlockSize, you must first set ReadMode to"bytes". FileDatastore sets the default value of BlockSize based on the value specified in ReadMode.

• If ReadMode is"file" or"partialfile", thenFileDatastore sets the defaultBlockSize toinf.
• If ReadMode is"bytes", thenFileDatastore sets the defaultBlockSize to128 megabytes.

AlternateFileSystemRoots — Alternate file system root paths

string vector | cell array

Alternate file system root paths, specified as the name-value argument consisting of"AlternateFileSystemRoots" and a string vector or a cell array. Use"AlternateFileSystemRoots" when you create a datastore on a local machine, but need to access and process the data on another machine (possibly of a different operating system). Also, when processing data using the Parallel Computing Toolbox™ and the MATLAB Parallel Server™, and the data is stored on your local machines with a copy of the data available on different platform cloud or cluster machines, you must use"AlternateFileSystemRoots" to associate the root paths.

• To associate a set of root paths that are equivalent to one another, specify"AlternateFileSystemRoots" as a string vector. For example,
  ["Z:\datasets","/mynetwork/datasets"]
• To associate multiple sets of root paths that are equivalent for the datastore, specify "AlternateFileSystemRoots" as a cell array containing multiple rows where each row represents a set of equivalent root paths. Specify each row in the cell array as either a string vector or a cell array of character vectors. For example:
  • Specify "AlternateFileSystemRoots" as a cell array of string vectors.
    {["Z:\datasets", "/mynetwork/datasets"];...
    ["Y:\datasets", "/mynetwork2/datasets","S:\datasets"]}
  • Alternatively, specify "AlternateFileSystemRoots" as a cell array of cell array of character vectors.
    {{'Z:\datasets','/mynetwork/datasets'};...
    {'Y:\datasets', '/mynetwork2/datasets','S:\datasets'}}

The value of "AlternateFileSystemRoots" must satisfy these conditions:

• Contains one or more rows, where each row specifies a set of equivalent root paths.
• Each row specifies multiple root paths and each root path must contain at least two characters.
• Root paths are unique and are not subfolders of one another.
• Contains at least one root path entry that points to the location of the files.

For more information, see Set Up Datastore for Processing on Different Machines or Clusters.

Example: ["Z:\datasets","/mynetwork/datasets"]

Data Types: string | cell

Properties

expand all

FileDatastore properties describe the files associated with aFileDatastore object. Except for the Files property, you can specify the value of FileDatastore properties using name-value pair arguments. To view or modify a property after creating the object, use the dot notation.

Files — Files included in datastore

character vector | cell array of character vectors | string scalar | string array

Files included in the datastore, resolved as a character vector, cell array of character vectors, string scalar, or string array, where each character vector or string is a full path to a file. Thelocation argument in thefileDatastore and datastore functions defines Files when the datastore is created.

Example: {"C:\dir\data\file1.ext";"C:\dir\data\file2.ext"}

Example: "hdfs:///data/*.mat"

Data Types: char | cell | string

Folders — Folders used to construct datastore

cell array of character vectors

This property is read-only.

Folders used to construct datastore, returned as a cell array of character vectors. The cell array is oriented as a column vector. Each character vector is a path to a folder that contains data files. Thelocation argument in thefileDatastore and datastore functions defines Folders when the datastore is created.

The Folders property is reset when you modify theFiles property of a FileDatastore object.

Data Types: cell

ReadFcn — Function that reads file data

function handle

Function that reads the file data, specified as a function handle.

The value specified by @fcn, sets the value of the ReadFcn property.

Example: @MyCustomFileReader

Data Types: function_handle

UniformRead — Vertically concatenateable flag

false (default) | true

This property is read-only.

Vertically concatenateable flag, specified as a logicaltrue or false. Specify the value of this property when you first create the FileDatastore object.

Example: fds = fileDatastore(location,"ReadFcn",@load,"UniformRead",true)

Data Types: logical | double

SupportedOutputFormats — Formats supported for writing

string row vector

This property is read-only.

Formats supported for writing, returned as a row vector of strings. This property specifies the possible output formats when using writeall to write output files from the datastore.

Object Functions

Examples

collapse all

Create FileDatastore Object

Create a fileDatastore object using either a FileSet object or file paths.

Create a FileSet object. Create a fileDatastore object.

fs = matlab.io.datastore.FileSet("airlinesmall.parquet"); fds = fileDatastore(fs,"ReadFcn",@load)

fds = FileDatastore with properties: Files: { ' ...\matlab\toolbox\matlab\demos\airlinesmall.parquet' } Folders: { '...\matlab\toolbox\matlab\demos' } UniformRead: 0 ReadMode: 'file' BlockSize: Inf PreviewFcn: @load SupportedOutputFormats: ["txt" "csv" "xlsx" "xls" "parquet" "parq" "png" "jpg" "jpeg" "tif" "tiff" "wav" "flac" "ogg" "mp4" "m4a"] ReadFcn: @load AlternateFileSystemRoots: {}

Alternatively, you can use file paths to create your fileDatastore object.

fds = fileDatastore("airlinesmall.parquet","ReadFcn",@load);

Read Datastore of MAT-Files

Create a datastore containing all the .mat files within the MATLAB® demos folder, specifying theload function to read the file data.

fds = fileDatastore(fullfile(matlabroot,"toolbox","matlab","demos"),"ReadFcn",@load,"FileExtensions",".mat")

fds = FileDatastore with properties: Files: { '...\matlab\toolbox\matlab\demos\accidents.mat'; '...\matlab\toolbox\matlab\demos\airfoil.mat'; ' ...\matlab\toolbox\matlab\demos\airlineResults.mat' ... and 38 more } Folders: { '...\matlab\toolbox\matlab\demos' } UniformRead: 0 ReadMode: 'file' BlockSize: Inf PreviewFcn: @load SupportedOutputFormats: ["txt" "csv" "xlsx" "xls" "parquet" "parq" "png" "jpg" "jpeg" "tif" "tiff" "wav" "flac" "ogg" "mp4" "m4a"] ReadFcn: @load AlternateFileSystemRoots: {}

Read the first file in the datastore, and then read the second file.

data1 = read(fds);
data2 = read(fds);

Read all files in the datastore simultaneously.

Initialize a cell array to hold the data and counteri.

dataarray = cell(numel(fds.Files), 1); i = 1;

Reset the datastore to the first file and read the files one at a time until there is no data left. Assign the data to the arraydataarray.

reset(fds);
while hasdata(fds)
dataarray{i} = read(fds); i = i+1; end

Read One Array at a Time From Large MAT-File

You can create a datastore to read from a large MAT-file that does not necessarily fit in memory. Assuming that each array in the large MAT-file fits in the available memory, create a datastore to read and process the data in three steps:

1. Write a custom reading function that reads one array at a time from a MAT-file.
2. Set up the parameters of the datastore function to perform partial reads.
3. Read one array at a time from the MAT-file.

Write a custom function that reads one array at time from MAT-file. The function must have a signature as described in the @ReadFcn argument of FileDatastore. Save this file in your working folder or in a folder that is on the MATLAB path. For this example, a custom function load_variable is included here.

function [data,variables,done] = load_variable(filename,variables)

% If variable list is empty, 
% create list of variables from the file
if isempty(variables) 
    variables = who('-file', filename);
end

% Load a variable from the list of variables
data = load(filename, variables{1});

% Remove the newly-read variable from the list
variables(1) = []; 

% Move on to the next file if this file is done reading.
done = isempty(variables); 

end

Create and setup a FileDatastore containing accidents.mat. Specify the datastore parameters to use "partialfile" as the read mode and load_variable as the custom reading function.

fds = fileDatastore("accidents.mat","ReadMode","partialfile","ReadFcn",@load_variable);

Read the first three variables from the file using the datastore. The file accidents.mat contains nine variables and every call to read returns one variable. Therefore, to get the first three variables, call the read function three times.

data = struct with fields: datasources: {3x1 cell}

data = struct with fields: hwycols: 17

data = struct with fields: hwydata: [51x17 double]

Note that the sample file accidents.mat is small and fits in memory, but you can expect similar results for large MAT-files that do not fit in memory.

Limitations

• MATLAB does not support internet URLs that require authentication.
• MATLAB Online™ supports internet URLs associated with Microsoft® OneDrive™ files and folders, while the installed version of MATLAB supports only local OneDrive files.

Tips

• To use the subset and shuffle functions on a FileDatastore object, you must set"ReadMode" to "file".

Version History

Introduced in R2016a

expand all

R2024b: Read data over HTTP and HTTPS using datastore functions

You can read data from primary online sources by performing datastore operations over an internet URL.