Run Standalone MATLAB MapReduce Application - MATLAB & Simulink (original) (raw)

Main Content

Supported Platform: Linux® only.

This example shows you how to create a standalone MATLAB® MapReduce application using the mcc command and run it against a Hadoop® cluster.

Goal: Calculate the maximum arrival delay of an airline from the given dataset.

Prerequisites

1. Start this example by creating a new work folder that is visible to the MATLAB search path.
2. Before starting MATLAB, at a terminal, set the environment variable HADOOP_PREFIX to point to the Hadoop installation folder. For example:

   Shell	Command
   csh / tcsh	% setenv HADOOP_PREFIX /usr/lib/hadoop
   bash	$ export HADOOP_PREFIX=/usr/lib/hadoop
   Note
   This example uses /usr/lib/hadoop as directory where Hadoop is installed. Your Hadoop installation directory maybe different.
   If you forget setting the HADOOP_PREFIX environment variable prior to starting MATLAB, set it up using the MATLAB function setenv at the MATLAB command prompt as soon as you start MATLAB. For example:
   setenv('HADOOP_PREFIX','/usr/lib/hadoop')

3. Install the MATLAB Runtime in a folder that is accessible by every worker node in the Hadoop cluster. This example uses/usr/local/MATLAB/MATLAB_Runtime/R2025a as the location of the MATLAB Runtime folder.
   If you don’t have the MATLAB Runtime, you can download it from the website at: https://www.mathworks.com/products/compiler/mcr.
   Note
   For information about MATLAB Runtime version numbers corresponding MATLAB releases, see this list.
4. Copy the map function maxArrivalDelayMapper.m from /usr/local/MATLAB/R2025a/toolbox/matlab/demos folder to the work folder.
   maxArrivalDelayMapper.m
   function maxArrivalDelayMapper (data, info, intermKVStore)
   partMax = max(data.ArrDelay);
   add(intermKVStore,'PartialMaxArrivalDelay',partMax);
   For more information, see Write a Map Function.
5. Copy the reduce function maxArrivalDelayReducer.m from _`matlabroot`_/toolbox/matlab/demos folder to the work folder.
   maxArrivalDelayReducer.m
   function maxArrivalDelayReducer(intermKey, intermValIter, outKVStore)
   maxVal = -inf;
   while hasnext(intermValIter)
   maxVal = max(getnext(intermValIter), maxVal);
   end
   add(outKVStore,'MaxArrivalDelay',maxVal);
   For more information, see Write a Reduce Function.
6. Create the directory/user/_`<username>`_/datasets on HDFS™ and copy the fileairlinesmall.csv to that directory. Here_`<username>`_ refers to your user name in HDFS.

$ ./hadoop fs -copyFromLocal airlinesmall.csv hdfs://host:54310/user/<username>/datasets  

Procedure

1. Start MATLAB and verify that the HADOOP_PREFIX environment variable has been set. At the command prompt, type:

   getenv('HADOOP_PREFIX')
   If ans is empty, review the Prerequisites section above to see how you can set theHADOOP_PREFIX environment variable.

2. Create a new MATLAB script with the namedepMapRedStandAlone.m. You will add the code listed in the steps listed below to this script file.
3. Create a datastore that points to the airline data in Hadoop Distributed File System (HDFS).
   ds = datastore('hdfs:///user/username/datasets/airlinesmall.csv',...
   'TreatAsMissing','NA',...
   'SelectedVariableNames',{'UniqueCarrier','ArrDelay'});
   For more information, see Work with Remote Data.
4. Configure the application for deployment against Hadoop with default settings.
   config = matlab.mapreduce.DeployHadoopMapReducer;
   The class matlab.mapreduce.DeployHadoopMapReducer can be used to configure a standalone application based on the Hadoop environment where it is going to be deployed.
   For example, if you want to specify the location of the MATLAB Runtime on each of the worker nodes on the cluster, include a line of code similar to this:
   config = matlab.mapreduce.DeployHadoopMapReducer('MCRRoot','/opt/MATLAB/MATLAB_Runtime/R2025a');
   In this scenario, we assume that the MATLAB Runtime is installed in a non-default location such as/opt/MATLAB/MATLAB_Runtime on the worker nodes.
   For information on specifying additional cluster specific properties, see matlab.mapreduce.DeployHadoopMapReducer.
   Note
   Specifying a MATLAB Runtime location as part of the classmatlab.mapreduce.DeployHadoopMapReducer will override any MATLAB Runtime location specified during the execution of the standalone application.
5. Define the execution environment using the mapreducer.
6. Apply the mapreduce function.
   result = mapreduce(...
   ds,...
   @maxArrivalDelayMapper,@maxArrivalDelayReducer,...
   mr,...
   'OutputType','Binary', ...
   'OutputFolder','hdfs:///user//results/myresults');
   Note
   An HDFS directory such as .../myresults can be written to only once. If you plan on running your standalone application multiple times against the Hadoop cluster, make sure you delete the.../myresults directory on HDFS prior to each execution. Another option is to change the name of the .../myresults directory in the MATLAB code and recompile the application.
7. Read the result from the resulting datastore.
   myAppResult = readall(result)
8. Use the mcc command with the -m flag to create a standalone application.
   mcc -m depMapRedStandAlone.m
   The -m flag creates a standard executable that can be run from a command line. However, the mcc command cannot package the results in an installer.
9. Run the standalone application from a Linux shell using the following command:

$ ./run_depMapRedStandAlone.sh /usr/local/MATLAB/MATLAB_Runtime/R2025a  

/usr/local/MATLAB/MATLAB_Runtime/R2025a is an argument indicating the location of the MATLAB Runtime.
Prior to executing the above command, verify that theHADOOP_PREFIX environment variable is set in the Terminal by typing:
Ifecho comes up empty, see the Prerequisites section above to see how you can set the HADOOP_PREFIX environment variable.
Your application will fail to execute if theHADOOP_PREFIX environment variable is not set. 10. You will see the following output:
myAppResult =
Key Value
_________________ ______
'MaxArrivalDelay' [1014]

To learn more about using the map andreduce functions, see Getting Started with MapReduce.

Complete code for the standalone applicationdepMapRedStandAlone can be found here:

depMapRedStandAlone.m

%% Create datastore ds = datastore(... 'hdfs:///user/username/datasets/airlinesmall.csv',... 'TreatAsMissing','NA',... 'SelectedVariableNames',{'UniqueCarrier','ArrDelay'});

%% Configure application for deployment against Hadoop with default settings config = matlab.mapreduce.DeployHadoopMapReducer;

%% Define the execution environment mr = mapreducer(config);

%% Apply the mapreduce function result = mapreduce(... ds,... @maxArrivalDelayMapper,@maxArrivalDelayReducer,... mr,... 'OutputType','Binary', ... 'OutputFolder','hdfs:///user/username/results/myresults');

%% Read the result from the resulting datastore myAppResult = readall(result)

See Also

datastore | TabularTextDatastore | KeyValueDatastore | matlab.mapreduce.DeployHadoopMapReducer | mcc

Topics

• Create Standalone Application from MATLAB
• Pass Parallel Computing Toolbox Profile at Run Time