Datasets — NVIDIA NeMo Microservices (original) (raw)

Use datasets in your customization and evautions jobs for a model.

Prerequisites#

Before creating and managing datasets, make sure that you have:

• The NeMo Entity Store and NeMo Data Store microservices are running on your cluster.
• The base URLs for the NeMo Entity Store and NeMo Data Store microservices. This depends on how your cluster administrator configures the ingress setup for the NeMo microservices in your cluster. For more information, see Demo Cluster Setup on Minikube and Ingress Setup for Production Environment.
• Hugging Face CLI or SDK installed.

Dataset Registry Workflow#

1. Prepare datasets.
2. Create a repo_id using {namespace}/{dataset_name}.
3. Register the dataset in NeMo Entity Store.
4. Upload dataset files to NeMo Data Store using the Hugging Face CLI or SDK with the repo_id.

Task Guides#

List datasets available for use in customization jobs.

List Datasets