Using Domain Knowledge for Low Resource Named Entity Recognition (original) (raw)

DOMAIN-TRANSFERABLE METHOD FOR NAMED ENTITY RECOGNITION TASK

Named Entity Recognition (NER) is a fundamental task in the fields of natural language processing and information extraction. NER has been widely used as a standalone tool or an essential component in a variety of applications such as question answering, dialogue assistants and knowledge graphs development. However, training reliable NER models requires a large amount of labelled data which is expensive to obtain, particularly in specialized domains. This paper describes a method to learn a domain-specific NER model for an arbitrary set of named entities when domain-specific supervision is not available. We assume that the supervision can be obtained with no human effort, and neural models can learn from each other. The code, data and models are publicly available.

Improving Low Resource Named Entity Recognition using Cross-lingual Knowledge Transfer

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, 2018

Neural networks have been widely used for high resource language (e.g. English) named entity recognition (NER) and have shown state-of-the-art results. However, for low resource languages, such as Dutch and Spanish, due to the limitation of resources and lack of annotated data, NER models tend to have lower performances. To narrow this gap, we investigate cross-lingual knowledge to enrich the semantic representations of low resource languages. We first develop neural networks to improve low resource word representations via knowledge transfer from high resource language using bilingual lexicons. Further, a lexicon extension strategy is designed to address out-of lexicon problem by automatically learning semantic projections. Finally, we regard word-level entity type distribution features as an external languageindependent knowledge and incorporate them into our neural architecture. Experiments on two low resource languages (Dutch and Spanish) demonstrate the effectiveness of these additional semantic representations (average 4.8% improvement). Moreover, on Chinese OntoNotes 4.0 dataset, our approach achieves an F-score of 83.07% with 2.91% absolute gain compared to the state-of-the-art systems.

Transfer Learning and Domain Adaptation for Named-Entity Recognition

2020

Transfer learning is a technique that is widely used in machine learning tasks when the availability of data for a task is limited and not enough to train a model. Transfer learning consists of primarily two major phases: pre-training and fine-tuning. During pre-training, a model is trained on a different but similar corpus which has a large amount of data in it. This model is further fine-tuned on the target task using the limited data available. Selecting the appropriate pre-training corpus is vital to avoid negative transfer learning that can harm a target task and impede the transfer of knowledge. Transfer learning is being used widely in industrial applications where the amount of labelled data is limited. This paper applies transfer learning to the task of named-entity recognition and investigates the effectiveness of transfer learning in a stacked bi-LSTM architecture with fastText word embedding.

Zero-Resource Cross-Domain Named Entity Recognition

Proceedings of the 5th Workshop on Representation Learning for NLP, 2020

Existing models for cross-domain named entity recognition (NER) rely on numerous unlabeled corpus or labeled NER training data in target domains. However, collecting data for low-resource target domains is not only expensive but also time-consuming. Hence, we propose a cross-domain NER model that does not use any external resources. We first introduce a Multi-Task Learning (MTL) by adding a new objective function to detect whether tokens are named entities or not. We then introduce a framework called Mixture of Entity Experts (MoEE) to improve the robustness for zero-resource domain adaptation. Finally, experimental results show that our model outperforms strong unsupervised cross-domain sequence labeling models, and the performance of our model is close to that of the state-of-theart model which leverages extensive resources.

Domain Specific Entity Recognition with Semantic-based Deep Learning Approach

IEEE Access, 2021

In digital agriculture, agronomists are required to make timely, profitable and more actionable precise decisions based on knowledge and experience. The input can be cultivated and related agricultural data, and one of them is text data, including news articles, business news, policy documents, or farming notes. To process this kind of data, identifying agricultural entities in the text is necessary to update news with agricultural orientation. This task is called Agriculture Entity Recognition (AGER-a kind of Named Entity Recognition task, NER, in the agriculture domain). However, there are very few approaches on AGER because of a lack of the consistent tagset and resources. In this study, we developed a new tagset for AGER to cover popular concepts in agriculture and we also propose a process for this task that consists of two stages: in the first stage, we use semantic-based approaches for detecting agricultural entities and semiautomatically build an annotated corpus of agricultural entities, while in the second stage, we identify the agricultural entities from the plain text using a deep learning approach, train on the annotated corpus. For the evaluation and validation, we build an annotated agriculture corpus and demonstrated the efficiency and robustness of our approach.

Entity Extraction in Low Resource Domains with Selective Pre-training of Large Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

Transformer-based language models trained on large natural language corpora have been very useful in downstream entity extraction tasks. However, they often result in poor performances when applied to domains that are different from those they are pretrained on. Continued pretraining using unlabeled data from target domains can help improve the performances of these language models on the downstream tasks. However, using all of the available unlabeled data for pretraining can be time-intensive; also, it can be detrimental to the performance of the downstream tasks, if the unlabeled data is not aligned with the data distribution for the target tasks. Previous works employed external supervision in the form of ontologies for selecting appropriate data samples for pretraining, but external supervision can be quite hard to obtain in low-resource domains. In this paper, we introduce effective ways to select data from unlabeled corpora of target domains for language model pretraining to improve the performances in target entity extraction tasks. Our data selection strategies do not require any external supervision. We conduct extensive experiments for the task of named entity recognition (NER) on seven different domains and show that language models pretrained on target domain unlabeled data obtained using our data selection strategies achieve better performances compared to those using data selection strategies in previous works that use external supervision. We also show that these pretrained language models using our data selection strategies outperform those pretrained on all of the available unlabeled target domain data.

CrossNER: Evaluating Cross-Domain Named Entity Recognition

2021

Cross-domain named entity recognition (NER) models are able to cope with the scarcity issue of NER samples in target domains. However, most of the existing NER benchmarks lack domain-specialized entity types or do not focus on a certain domain, leading to a less effective cross-domain evaluation. To address these obstacles, we introduce a cross-domain NER dataset (CrossNER), a fully-labeled collection of NER data spanning over five diverse domains with specialized entity categories for different domains. Additionally, we also provide a domain-related corpus since using it to continue pre-training language models (domain-adaptive pre-training) is effective for the domain adaptation. We then conduct comprehensive experiments to explore the effectiveness of leveraging different levels of the domain corpus and pre-training strategies to do domain-adaptive pre-training for the cross-domain task. Results show that focusing on the fractional corpus containing domain-specialized entities an...

A Survey on Recent Advances in Named Entity Recognition from Deep Learning models

ArXiv, 2018

Named Entity Recognition (NER) is a key component in NLP systems for question answering, information retrieval, relation extraction, etc. NER systems have been studied and developed widely for decades, but accurate systems using deep neural networks (NN) have only been introduced in the last few years. We present a comprehensive survey of deep neural network architectures for NER, and contrast them with previous approaches to NER based on feature engineering and other supervised or semi-supervised learning algorithms. Our results highlight the improvements achieved by neural networks, and show how incorporating some of the lessons learned from past work on feature-based NER systems can yield further improvements.

Deep Learning Transformer Architecture for Named Entity Recognition on Low Resourced Languages: State of the art results

Computer Science and Information Systems (FedCSIS), 2019 Federated Conference on, 2022

paper reports on the evaluation of Deep Learning (DL) transformer architecture models for Named-Entity Recognition (NER) on ten low-resourced South African (SA) languages. In addition, these DL transformer models were compared to other Neural Network and Machine Learning (ML) NER models. The findings show that transformer models substantially improve performance when applying discrete finetuning parameters per language. Furthermore, fine-tuned transformer models outperform other neural network and machine learning models on NER with the low-resourced SA languages. For example, the transformer models obtained the highest F-scores for six of the ten SA languages and the highest average F-score surpassing the Conditional Random Fields ML model. Practical implications include developing high-performance NER capability with less effort and resource costs, potentially improving downstream NLP tasks such as Machine Translation (MT). Therefore, the application of DL transformer architecture models for NLP NER sequence tagging tasks on low-resourced SA languages is viable. Additional research could evaluate the more recent transformer architecture models on other Natural Language Processing tasks and applications, such as Phrase chunking, MT, and Part-of-Speech tagging.

Named Entity Recognition for Novel Types by Transfer Learning

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016

In named entity recognition, we often don't have a large in-domain training corpus or a knowledge base with adequate coverage to train a model directly. In this paper, we propose a method where, given training data in a related domain with similar (but not identical) named entity (NE) types and a small amount of in-domain training data, we use transfer learning to learn a domain-specific NE model. That is, the novelty in the task setup is that we assume not just domain mismatch, but also label mismatch.