Configure Metrics for MATLAB Job Scheduler - MATLAB & Simulink (original) (raw)

Configure MATLAB® Job Scheduler to export cluster monitoring metrics such as cluster status, worker utilization, and licenses in use from the job manager. These metrics help you:

Monitor the health of the MATLAB Job Scheduler cluster
Diagnose cluster issues
Optimize cluster performance

You can gather the exported metrics with a cluster monitoring system such as Prometheus® and visualize the metrics in a preconfigured Grafana® dashboard with usage analysis and alerting tools.

Prerequisites

You must install and configure MATLAB Job Scheduler if this is the first time you are integrating MATLAB Job Scheduler with your cluster. For the most common configuration options, seeInstall for MATLAB Job Scheduler with Network License Manager.

You must know the location of your installed MATLAB Parallel Server™ software. You use this location when configuring MATLAB Job Scheduler and generating the certificate and key files.

Edit MATLAB Job Scheduler Parameter File

To set up your MATLAB Job Scheduler to export metrics, you must edit the mjs_def file on your head node before installing the mjs service and starting MATLAB Job Scheduler. You can find this file in these locations. Substitutematlabroot with the path to the location of your MATLAB Parallel Server installation.

matlabroot\toolbox\parallel\bin\mjs_def.bat on Windows® operating systems
matlabroot/toolbox/parallel/bin/mjs_def.sh on Linux® operating systems

To learn more about the parameters in the mjs_def file, see Define MATLAB Job Scheduler Startup Parameters.

Use these parameters to configure the MATLAB Job Scheduler job manager to export metrics. Edit the parameters in themjs_def file with the required values.

Parameter	Description	Required Values
EXPORT_METRICS	Option to export cluster monitoring metrics from the job manager.	true
METRICS_PORT	Port for exporting metrics.The job manager uses an HTTP or HTTPS server at the port you specify to export metrics.	By default, the job manager uses port number8001.
USE_SECURE_METRICS	Option to use encrypted communication to export metrics.	By default, USE_SECURE_METRICS is set totrue and the job manager exports metrics on an encrypted HTTPS server.To disable encryption and export metrics on an HTTP server, setUSE_SECURE_METRICS to false.
METRICS_CA_FILE	Job manager's trusted certificate authority (CA) for encrypted metrics.If USE_SECURE_METRICS is true, the job manager only forms encrypted connections to clients that present certificates signed by this CA. You must set this parameter if USE_SECURE_METRICS istrue.	Specify a path to a CA certificate, for example:METRICS_CA_FILE=<~/mjs-metrics>/ca.crtYou can use the mjssetup tool to generate the job manager's CA certificate. To learn more, see Generate Certificate and Key Files.
METRICS_CERT_FILE	Job manager's certificate for encrypted metrics.The job manager presents this certificate to clients that want to form encrypted connections for metrics. You must set this parameter if USE_SECURE_METRICS istrue.	Specify a path to the job manager's certificate, for example:METRICS_CERT_FILE=<~/mjs-metrics>/jobmanager.crtYou can use the mjssetup tool to generate the job manager's certificate. To learn more, see Generate Certificate and Key Files.
METRICS_KEY_FILE	Job manager's private key for encrypted metrics.You must set this parameter if USE_SECURE_METRICS istrue.	Specify a path to the job manager's private key. You must associate the private key with the certificate you specify for METRICS_CERT_FILE. For example:METRICS_KEY_FILE=<~/mjs-metrics>/jobmanager.keyYou can use the mjssetup tool to generate the job manager's private key. To learn more, see Generate Certificate and Key Files.

Generate Certificate and Key Files

You can use the mjssetup tool to generate the job manager's certificate and key files you need to export metrics using encrypted communication. Themjssetup tool ships with MATLAB. You can find the executable for the mjssetup tool in these folders. Substitute matlabroot with the path to the location of yourMATLAB Parallel Server installation.

matlabroot/toolbox/parallel/bin/glnxa64 on Linux operating systems.
matlabroot\toolbox\parallel\bin\win64 on Windows operating systems.

To generate the job manager's certificate and key files, specify thegenerate-metrics-certificates-and-keys command, the hostname of the job manager, and the output folder for the certificate and key files. For example, on a Linux operating system, generate certificate and key files for a job manager with the hostname MJSJobManager in a folder with the namemjs-metrics.

cd matlabroot/toolbox/parallel/bin/glnxa64
mjssetup generate-metrics-certificates-and-keys -jobmanagerhost <MJSJobManager> -outdir <mjs-metrics>

The command also generates the prometheus.crt public certificate and theprometheus.key private key that you use to set up a Prometheus data source. To learn more, see Monitor Cluster Metrics with Prometheus and Grafana.

Access Server Metrics

When you configure MATLAB Job Scheduler to export metrics, the job manager starts an HTTP or HTTPS server on the head node. The server opens up a port at the number specified usingMETRICS_PORT and exposes a page with the cluster metrics. The cluster metrics are in a human-readable Prometheus text-based format that you can easily integrate with a server performance monitoring platform.

You can use the nodestatus command to find the URL of the page with the cluster metrics. To learn how to use the metrics URL to set up a Prometheus data source, see Monitor Cluster Metrics with Prometheus and Grafana.

Job manager lookup process: Status Running

Job manager: Name MJSJobManager Running on host host1.dhcp.mycompany.com Number of workers 4 Status running Supported releases R2024b Metrics URL https://host1.dhcp.mycompany.com:8001

Monitor Cluster Metrics with Prometheus and Grafana

After you set up your MATLAB Job Scheduler cluster to export metrics, you can set up a Prometheus data source to gather the cluster metrics and a Grafana dashboard to visualize and analyze the metrics.

To configure the Prometheus data source and access the preconfigured Grafana dashboard, follow the instructions provided in the GitHub® repository Cluster Monitoring Integration for MATLAB Job Scheduler.