GitHub - google/ml-metadata: For recording and retrieving metadata associated with ML developer and data scientist workflows. (original) (raw)

ML Metadata

Python PyPI

ML Metadata (MLMD) is a library for recording and retrieving metadata associated with ML developer and data scientist workflows.

NOTE: ML Metadata may be backwards incompatible before version 1.0.

Getting Started

For more background on MLMD and instructions on using it, see thegetting started guide

Installing from PyPI

The recommended way to install ML Metadata is to use thePyPI package:

Then import the relevant packages:

from ml_metadata import metadata_store from ml_metadata.proto import metadata_store_pb2

Nightly Packages

ML Metadata (MLMD) also hosts nightly packages athttps://pypi-nightly.tensorflow.org on Google Cloud. To install the latest nightly package, please use the following command:

pip install --extra-index-url https://pypi-nightly.tensorflow.org/simple ml-metadata

Installing with Docker

This is the recommended way to build ML Metadata under Linux, and is continuously tested at Google.

Please first install docker and docker-compose by following the directions:docker;docker-compose.

Then, run the following at the project root:

DOCKER_SERVICE=manylinux-python${PY_VERSION} sudo docker compose build ${DOCKER_SERVICE} sudo docker compose run ${DOCKER_SERVICE}

where PY_VERSION is one of {39, 310, 311}.

A wheel will be produced under dist/, and installed as follows:

Installing from source

1. Prerequisites

To compile and use ML Metadata, you need to set up some prerequisites.

Install Bazel

If Bazel is not installed on your system, install it now by following these directions.

Install cmake

If cmake is not installed on your system, install it now by following these directions.

2. Clone ML Metadata repository

git clone https://github.com/google/ml-metadata cd ml-metadata

Note that these instructions will install the latest master branch of ML Metadata. If you want to install a specific branch (such as a release branch), pass -b <branchname> to the git clone command.

3. Build the pip package

ML Metadata uses Bazel to build the pip package from source:

python setup.py bdist_wheel

You can find the generated .whl file in the dist subdirectory.

4. Install the pip package

5.(Optional) Build the grpc server

ML Metadata uses Bazel to build the c++ binary from source:

bazel build -c opt --define grpc_no_ares=true //ml_metadata/metadata_store:metadata_store_server

Supported platforms

MLMD is built and tested on the following 64-bit operating systems: