Data formats · spaCy API Documentation (original) (raw)

Data formats

Details on spaCy's input and output data formats

This section documents input and output formats of data used by spaCy, including the training config, training data and lexical vocabulary data. For an overview of label schemes used by the models, see themodels directory. Each trained pipeline documents the label schemes used in its components, depending on the data it was trained on.

Training config v3.0

Config files define the training process and pipeline and can be passed tospacy train. They useThinc’s configuration system under the hood. For details on how to use training configs, see theusage documentation. To get started with the recommended settings for your use case, check out thequickstart widget or run theinit config command.

nlp section

Defines the nlp object, its tokenizer andprocessing pipeline component names.

components section

This section includes definitions of thepipeline components and their models, if available. Components in this section can be referenced in the pipeline of the[nlp] block. Component blocks need to specify either a factory (named function to use to create component) or a source (name of path of trained pipeline to copy components from). See the docs ondefining pipeline components for details.

paths, system variables

These sections define variables that can be referenced across the other sections as variables. For example ${paths.train} uses the value of train defined in the block [paths]. If your config includes custom registered functions that need paths, you can define them here. All config values can also beoverwritten on the CLI when you runspacy train, which is especially relevant for data paths that you don’t want to hard-code in your config file.

corpora section

This section defines a dictionary mapping of string keys to functions. Each function takes an nlp object and yields Example objects. By default, the two keys train and dev are specified and each refer to aCorpus. When pretraining, an additional pretrainsection is added that defaults to a JsonlCorpus. You can also register custom functions that return a callable.

Alternatively, the [corpora] block can refer to one function that returns a dictionary keyed by the corpus names. This can be useful if you want to load a single corpus once and then divide it up into train and dev partitions.

training section

This section defines settings and controls for the training and evaluation process that are used when you run spacy train.

pretraining sectionoptional

This section is optional and defines settings and controls forlanguage model pretraining. It’s used when you run spacy pretrain.

initialize section

This config block lets you define resources for initializing the pipeline. It’s used by Language.initialize and typically called right before training (but not at runtime). The section allows you to specify local file paths or custom functions to load data resources from, without requiring them at runtime when you load the trained pipeline back in. Also see the usage guides on theconfig lifecycle andcustom initialization.

Training data

Binary training format v3.0

The main data format used in spaCy v3.0 is a binary format created by serializing a DocBin, which represents a collection of Docobjects. This means that you can train spaCy pipelines using the same format it outputs: annotated Doc objects. The binary format is extremely efficient in storage, especially when packing multiple documents together.

Typically, the extension for these binary files is .spacy, and they are used as input format for specifying a training corpus and for spaCy’s CLI train command. The built-inconvert command helps you convert spaCy’s previousJSON format to the new binary format. It also supports conversion of the .conllu format used by theUniversal Dependencies corpora.

Note that while this is the format used to save training data, you do not have to understand the internal details to use it or create training data. See the section on preparing training data.

JSON training format deprecated

Example structure

``

Here’s an example of dependencies, part-of-speech tags and named entities, taken from the English Wall Street Journal portion of the Penn Treebank:

Annotation format for creating training examples

An Example object holds the information for one training instance. It stores two Doc objects: one for holding the gold-standard reference data, and one for holding the predictions of the pipeline. Examples can be created using theExample.from_dict method with a reference Doc and a dictionary of gold-standard annotations.

Examples

``

Lexical data for vocabulary

This data file can be provided via the vocab_data setting in the[initialize] block of the training config to pre-define the lexical data to initialize the nlp object’s vocabulary with. The file should contain one lexical entry per line. The first line defines the language and vocabulary settings. All other lines are expected to be JSON objects describing an individual lexeme. The lexical attributes will be then set as attributes on spaCy’s Lexeme object.

First line

``

Entry structure

``

Here’s an example of the 20 most frequent lexemes in the English training data:

The pipeline meta is available as the file meta.json and exported automatically when you save an nlp object to disk. Its contents are available as nlp.meta.