encode - Encode documents as matrix of word or n-gram counts - MATLAB (original) (raw)

Encode documents as matrix of word or n-gram counts

Syntax

Description

Use encode to encode an array of tokenized documents as a matrix of word or n-gram counts according to a bag-of-words or bag-of-n-grams model. To encode documents as vectors or word indices, use a wordEncoding object.

[counts](#d126e20930) = encode([bag](#d126e20752),[documents](#mw%5Fa4b07ec7-c38c-4596-b21f-5d715df8cb79)) returns a matrix of frequency counts for documents based on the bag-of-words or bag-of-n-grams model bag.

example

[counts](#d126e20930) = encode([bag](#d126e20752),[words](#d126e20817)) returns a matrix of frequency counts for a list of words.

example

[counts](#d126e20930) = encode(___,[Name,Value](#namevaluepairarguments)) specifies additional options using one or more name-value pair arguments.

example

Examples

collapse all

Encode an array of documents as a matrix of word counts.

documents = tokenizedDocument([ "an example of a short sentence" "a second short sentence"]); bag = bagOfWords(documents)

bag = bagOfWords with properties:

    NumWords: 7
      Counts: [2×7 double]
  Vocabulary: ["an"    "example"    "of"    "a"    "short"    "sentence"    "second"]
NumDocuments: 2

documents = tokenizedDocument([ "a new sentence" "a second new sentence"])

documents = 2×1 tokenizedDocument:

3 tokens: a new sentence
4 tokens: a second new sentence

View the documents encoded as a matrix of word counts. The word "new" does not appear in bag, so it is not counted.

counts = encode(bag,documents); full(counts)

ans = 2×7

 0     0     0     1     0     1     0
 0     0     0     1     0     1     1

The columns correspond to the vocabulary of the bag-of-words model.

ans = 1×7 string "an" "example" "of" "a" "short" "sentence" "second"

Encode an array of words as a vector of word counts.

documents = tokenizedDocument([ "an example of a short sentence" "a second short sentence"]); bag = bagOfWords(documents)

bag = bagOfWords with properties:

    NumWords: 7
      Counts: [2×7 double]
  Vocabulary: ["an"    "example"    "of"    "a"    "short"    "sentence"    "second"]
NumDocuments: 2

words = ["another" "example" "of" "a" "short" "example" "sentence"]; counts = encode(bag,words)

counts = 1×7 sparse double row vector (5 nonzeros) (1,2) 2 (1,3) 1 (1,4) 1 (1,5) 1 (1,6) 1

Encode an array of documents as a matrix of word counts with documents in columns.

documents = tokenizedDocument([ "an example of a short sentence" "a second short sentence"]); bag = bagOfWords(documents)

bag = bagOfWords with properties:

    NumWords: 7
      Counts: [2×7 double]
  Vocabulary: ["an"    "example"    "of"    "a"    "short"    "sentence"    "second"]
NumDocuments: 2

documents = tokenizedDocument([ "a new sentence" "a second new sentence"])

documents = 2×1 tokenizedDocument:

3 tokens: a new sentence
4 tokens: a second new sentence

View the documents encoded as a matrix of word counts with documents in columns. The word "new" does not appear in bag, so it is not counted.

counts = encode(bag,documents,'DocumentsIn','columns'); full(counts)

ans = 7×2

Input Arguments

collapse all

Input documents, specified as a tokenizedDocument array, a string array of words, or a cell array of character vectors. If documents is a string array or a cell array of character vectors, then it must be a row vector representing a single document, where each element is a word.

Tip

To ensure that the documents are encoded correctly, you must preprocess the input documents using the same steps as the documents used to create the input model. For an example showing how to create a function to preprocess text data, see Prepare Text Data for Analysis.

Input words, specified as a string vector, character vector, or cell array of character vectors. If you specify words as a character vector, then the function treats the argument as a single word.

Data Types: string | char | cell

Name-Value Arguments

collapse all

Specify optional pairs of arguments asName1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: 'DocumentsIn','rows' specifies the orientation of the output documents as rows.

Orientation of output documents in the frequency count matrix, specified as the comma-separated pair consisting of 'DocumentsIn' and one of the following:

'rows' – Return a matrix of frequency counts with rows corresponding to documents.
'columns' – Return a transposed matrix of frequency counts with columns corresponding to documents.

Data Types: char

Indicator for forcing output to be returned as cell array, specified as the comma separated pair consisting of 'ForceCellOutput' and true or false.

Data Types: logical

Output Arguments

collapse all

Word or n-gram counts, returned as a sparse matrix of nonnegative integers or a cell array of sparse matrices.

If bag is a non-scalar array or'ForceCellOutput' is true, then the function returns the outputs as a cell array of sparse matrices. Each element in the cell array is matrix of word or n-gram counts of the corresponding element of bag.

Version History

Introduced in R2017b