tf.raw_ops.AutoShardDataset

Stay organized with collections Save and categorize content based on your preferences.

Creates a dataset that shards the input dataset.

View aliases

Compat aliases for migration

tf.compat.v1.raw_ops.AutoShardDataset

tf.raw_ops.AutoShardDataset(
    input_dataset,
    num_workers,
    index,
    output_types,
    output_shapes,
    auto_shard_policy=0,
    num_replicas=0,
    name=None
)

Creates a dataset that shards the input dataset by num_workers, returning a sharded dataset for the index-th worker. This attempts to automatically shard a dataset by examining the Dataset graph and inserting a shard op before the inputs to a reader Dataset (e.g. CSVDataset, TFRecordDataset).

This dataset will throw a NotFound error if we cannot shard the dataset automatically.

Args
input_dataset	A Tensor of type variant. A variant tensor representing the input dataset.
num_workers	A Tensor of type int64. A scalar representing the number of workers to distribute this dataset across.
index	A Tensor of type int64. A scalar representing the index of the current worker out of num_workers.
output_types	A list of tf.DTypes that has length >= 1.
output_shapes	A list of shapes (each a tf.TensorShape or list of ints) that has length >= 1.
auto_shard_policy	An optional int. Defaults to 0.
num_replicas	An optional int. Defaults to 0.
name	A name for the operation (optional).

Returns
A Tensor of type variant.

tf.raw_ops.AutoShardDataset | TensorFlow v2.16.1 (original) (raw)

View aliases