tft.experimental.document_frequency  |  TFX  |  TensorFlow (original) (raw)

tft.experimental.document_frequency

Stay organized with collections Save and categorize content based on your preferences.

Maps the terms in x to their document frequency in the same order.

tft.experimental.document_frequency(
    x: tf.SparseTensor, vocab_size: int, name: Optional[str] = None
) -> tf.SparseTensor

The document frequency of a term is the number of documents that contain the term in the entire dataset. Each unique vocab term has a unique document frequency.

Example usage:

def preprocessing_fn(inputs): integerized = tft.compute_and_apply_vocabulary(inputs['x']) vocab_size = tft.get_num_buckets_for_transformed_feature(integerized) return { 'df': tft.experimental.document_frequency(integerized, vocab_size), 'integerized': integerized, } raw_data = [dict(x=["I", "like", "pie", "pie", "pie"]), dict(x=["yum", "yum", "pie"])] feature_spec = dict(x=tf.io.VarLenFeature(tf.string)) raw_data_metadata = tft.DatasetMetadata.from_feature_spec(feature_spec) with tft_beam.Context(temp_dir=tempfile.mkdtemp()): transformed_dataset, transform_fn = ( (raw_data, raw_data_metadata) | tft_beam.AnalyzeAndTransformDataset(preprocessing_fn)) transformed_data, transformed_metadata = transformed_dataset transformed_data [{'df': array([1, 1, 2, 2, 2]), 'integerized': array([3, 2, 0, 0, 0])}, {'df': array([1, 1, 2]), 'integerized': array([1, 1, 0])}]

  example strings: [["I", "like", "pie", "pie", "pie"], ["yum", "yum", "pie]]
  in: SparseTensor(indices=[[0, 0], [0, 1], [0, 2], [0, 3], [0, 4],
                            [1, 0], [1, 1], [1, 2]],
                   values=[1, 2, 0, 0, 0, 3, 3, 0])
  out: SparseTensor(indices=[[0, 0], [0, 1], [0, 2], [0, 3], [0, 4],
                            [1, 0], [1, 1], [1, 2]],
                   values=[1, 1, 2, 2, 2, 1, 1, 2])