An Investigation of the Interactions Between Pre-Trained Word Embeddings, Character Models and POS Tags in Dependency Parsing (original) (raw)
Related papers
ArXiv, 2019
We present an extensive evaluation of three recently proposed methods for contextualized embeddings on 89 corpora in 54 languages of the Universal Dependencies 2.3 in three tasks: POS tagging, lemmatization, and dependency parsing. Employing the BERT, Flair and ELMo as pretrained embedding inputs in a strong baseline of UDPipe 2.0, one of the best-performing systems of the CoNLL 2018 Shared Task and an overall winner of the EPE 2018, we present a one-to-one comparison of the three contextualized word embedding methods, as well as a comparison with word2vec-like pretrained embeddings and with end-to-end character-level word embeddings. We report state-of-the-art results in all three tasks as compared to results on UD 2.2 in the CoNLL 2018 Shared Task.
The Impact of Word Embeddings on Neural Dependency Parsing
2021
Using neural models to parse natural language into dependency structures has improved the state of the art considerably. These models heavily rely on word embeddings as input rep resentations, which raises the question whether the observed improvement is contributed by the learning abilities of the network itself or by the lexical information captured by means of the word embeddings they use. To answer this question, we conducted a series of experiments on German data from three different genres using artificial embeddings intentionally made uninformative in different ways. We found that without the context information provided by the embeddings, parser performance drops to that of conventional parsers, but not below. Ex periments with domainspecific embeddings, however, did not yield additional improve ments in comparison to largescale general purpose embeddings.
An improved joint model: POS tagging and dependency parsing
Journal of Artificial Intelligence and Data Mining, 2016
Dependency parsing is a way of syntactic parsing and a natural language that automatically analyzes the dependency structure of sentences, and the input for each sentence creates a dependency graph. Part-Of-Speech (POS) tagging is a prerequisite for dependency parsing. Generally, dependency parsers do the POS tagging task along with dependency parsing in a pipeline mode. Unfortunately, in pipeline models, a tagging error propagates, but the model is not able to apply useful syntactic information. The goal of joint models simultaneously reduce errors of POS tagging and dependency parsing tasks. In this research, we attempted to utilize the joint model on the Persian and English language using Corbit software. We optimized the model's features and improved its accuracy concurrently. Corbit software is an implementation of a transition-based approach for word segmentation, POS tagging and dependency parsing. In this research, the joint accuracy of POS tagging and dependency parsing over the test data on Persian, reached 85.59% for coarse-grained and 84.24% for fine-grained POS. Also, we attained 76.01% for coarse-grained and 74.34% for fine-grained POS on English.
A high-throughput dependency parser
2017
Dependency parsing is an important task in NLP, and it is used in many downstream tasks for analyzing the semantic structure of sentences. Analyzing very large corpora in a reasonable amount of time, however, requires a fast parser. In this thesis we develop a transitionbased dependency parser with a neural-network decision function which outperforms spaCy, Stanford CoreNLP, and MALTParser in terms of speed while having a comparable, and in some cases better, accuracy. We also develop several variations of our model to investigate the trade-off between accuracy and speed. This leads to a model with a greatly reduced feature set which is much faster but less accurate, as well as a more complex model involving a BiLSTM simultaneously trained to produce POS tags which is more accurate, but much slower. We compare the accuracy and speed of our different parser models against the three mentioned parsers on the Penn Treebank, Universal Dependencies English, and Ontonotes datasets using tw...
2018 10th International Conference on Knowledge and Systems Engineering (KSE)
In Vietnamese dependency parsing, several methods have been proposed. Dependency parser which uses deep neural network model has been reported that achieved state-of-theart results. In this paper, we proposed a new method which applies LSTM easy-first dependency parsing with pre-trained word embeddings and character-level word embeddings. Our method achieves an accuracy of 80.91% of unlabeled attachment score and 72.98% of labeled attachment score on the Vietnamese Dependency Treebank (VnDT).
A Framework for Understanding the Role of Morphology in Universal Dependency Parsing
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
This paper presents a simple framework for characterizing morphological complexity and how it encodes syntactic information. In particular, we propose a new measure of morphosyntactic complexity in terms of governordependent preferential attachment that explains parsing performance. Through experiments on dependency parsing with data from Universal Dependencies (UD), we show that representations derived from morphological attributes deliver important parsing performance improvements over standard word form embeddings when trained on the same datasets. We also show that the new morphosyntactic complexity measure is predictive of the gains provided by using morphological attributes over plain forms on parsing scores, making it a tool to distinguish languages using morphology as a syntactic marker from others.
Tailoring Continuous Word Representations for Dependency Parsing
Word representations have proven useful for many NLP tasks, e.g., Brown clusters as features in dependency parsing (Koo et al., 2008). In this paper, we investigate the use of continuous word representations as features for dependency parsing. We compare several popular embeddings to Brown clusters, via multiple types of features, in both news and web domains. We find that all embeddings yield significant parsing gains, including some recent ones that can be trained in a fraction of the time of others. Explicitly tailoring the representations for the task leads to further improvements. Moreover, an ensemble of all representations achieves the best results, suggesting their complementarity.
Informed ways of improving data-driven dependency parsing for German
We investigate a series of targeted modifications to a data-driven dependency parser of German and show that these can be highly effective even for a relatively well studied language like German if they are made on a (linguistically and methodologically) informed basis and with a parser implementation that allows for fast and robust training and application. Making relatively small changes to a range of very different system components, we were able to increase labeled accuracy on a standard test set (from the CoNLL 2009 shared task), ignoring gold standard partof-speech tags, from 87.64% to 89.40%. The study was conducted in less than five weeks and as a secondary project of all four authors. Effective modifications include the quality and combination of autoassigned morphosyntactic features entering machine learning, the internal feature handling as well as the inclusion of global constraints and a combination of different parsing strategies.
N-best Rescoring for Parsing Based on Dependency-Based Word Embeddings
2016
Rescoring approaches for parsing aims to re-rank and change the order of parse trees produced by a general parser for a given sentence. The re-ranking performance depends on whether or not the rescoring function is able to precisely estimate the quality of parse trees by using more complex features from the whole parse tree. However it is a challenge to design an appropriate rescoring function since complex features usually face the severe problem of data sparseness. And it is also difficult to obtain sufficient information requisite in re-estimatation of tree structures because existing annotated Treebanks are generally small-sized. To address the issue, in this paper, we utilize a large amount of auto-parsed trees to learn the syntactic and sememtic information. And we propose a simple but effective score function in order to integrate the scores provided by the baseline parser and dependency association scores based on dependency-based word embeddings, learned from auto-parsed trees. The dependency association scores can relieve the problem of data sparseness, since they can be still calculated by word embeddings even without occurrence of a dependency word pair in a corpus. Moreover, semantic role labels are also considered to distinct semantic relation of word pairs. Experimental results show that our proposed model improves the base Chinese parser significantly.
Joint Ensemble Model for POS Tagging and Dependency Parsing
2014
In this paper we present several approaches towards constructing joint ensemble models for morphosyntactic tagging and dependency parsing for a morphologically rich language-Bulgarian. In our experiments we use state-of-the-art taggers and dependency parsers to obtain an extended version of the treebank for Bulgarian, BulTreeBank, which, in addition to the standard CoNLL fields, contains predicted morphosyntactic tags and dependency arcs for each word. In order to select the most suitable tag and arc from the proposed ones, we use several ensemble techniques, the result of which is a valid dependency tree. Most of these approaches show improvement over the results achieved individually by the tools for tagging and parsing.