Berry de Bruijn - Academia.edu (original) (raw)

Papers by Berry de Bruijn

Journal of Clinical Epidemiology, 2016

To maximize the proportion of relevant studies identified for inclusion in systematic reviews (re... more To maximize the proportion of relevant studies identified for inclusion in systematic reviews (recall), complex time-consuming Boolean searches across multiple databases are common. Although MEDLINE provides excellent coverage of health science evidence, it has proved challenging to achieve high levels of recall through Boolean searches alone. Recall of one Boolean search method, the clinical query (CQ), combined with a ranking method, support vector machine (SVM), or PubMed-related articles, was tested against a gold standard of studies added to 6 updated Cochrane reviews and 10 Agency for Healthcare Research and Quality (AHRQ) evidence reviews. For the AHRQ sample, precision and temporal stability were examined for each method. Recall of new studies was 0.69 for the CQ, 0.66 for related articles, 0.50 for SVM, 0.91 for the combination of CQ and related articles, and 0.89 for the combination of CQ and SVM. Precision was 0.11 for CQ and related articles combined, and 0.11 for CQ and SVM combined. Related articles showed least stability over time. The complementary combination of a Boolean search strategy and a ranking strategy appears to provide a robust method for identifying relevant studies in MEDLINE.

Systematic reviews, Jan 22, 2016

Data abstraction, a critical systematic review step, is time-consuming and prone to errors. Curre... more Data abstraction, a critical systematic review step, is time-consuming and prone to errors. Current standards for approaches to data abstraction rest on a weak evidence base. We developed the Data Abstraction Assistant (DAA), a novel software application designed to facilitate the abstraction process by allowing users to (1) view study article PDFs juxtaposed to electronic data abstraction forms linked to a data abstraction system, (2) highlight (or "pin") the location of the text in the PDF, and (3) copy relevant text from the PDF into the form. We describe the design of a randomized controlled trial (RCT) that compares the relative effectiveness of (A) DAA-facilitated single abstraction plus verification by a second person, (B) traditional (non-DAA-facilitated) single abstraction plus verification by a second person, and (C) traditional independent dual abstraction plus adjudication to ascertain the accuracy and efficiency of abstraction. This is an online, randomized, t...

Permission is granted to quote short excerpts and to reproduce figures and tables from this repor... more Permission is granted to quote short excerpts and to reproduce figures and tables from this report, provided that the source of such material is fully acknowledged.

Studies in Health Technology and Informatics, 2011

The Receiver-Operating Characteristic curve or ROC has been a long standing and well appreciated ... more The Receiver-Operating Characteristic curve or ROC has been a long standing and well appreciated tool to assess performance of classifiers or diagnostic tests. Likewise, the Area Under the ROC (AUC) has been a metric to summarize the power of a test or ability of a classifier in one measurement. This article aims to revisit the AUC, and ties it to key characteristics of the noncentral hypergeometric distribution. It is demonstrated that this statistical distribution can be used in modeling the behaviour of classifiers, which is of value for comparing classifiers.

Information Retrieval experiments rarely examine more than a small number of user or system chara... more Information Retrieval experiments rarely examine more than a small number of user or system characteristics because of the limited availability of human subjects. In this article we present an interaction model, and based on that an experimental method. This automated method helps identify which user and system variables are relevant, which are independent of other variables, and which ranges of the variables are most important.

International Journal of Healthcare Technology and Management, 1999

Methods of Information in Medicine

A method is presented for assigning classification codes to pathology reports by searching simila... more A method is presented for assigning classification codes to pathology reports by searching similar reports from an archive collection. The key for searching is textual similarity, which estimates the true, semantic similarity. This method does not require explicit modeling, and can be applied to any language or any application domain that uses natural language reporting. A number of simulation experiments was run to assess the accuracy of the method and to indicate the role of size of the archive and the transfer of document collections across laboratories. In at least 63% of the simulation trials, the most similar archive text offered a suitable classification on organ, origin and diagnosis. In 85 to 90% of the trials, the archive's best solution was found within the first five similar reports. The results indicate that the method is suitable for its purpose: suggesting potentially correct classifications to the reporting diagnostician.

A method is presented that enables automatic classification of medical diagnoses through their re... more A method is presented that enables automatic classification of medical diagnoses through their reports in natural language. The method is based on the nearest neighbor rule: classification of a new case by assigning to it the same codes as the most similar case in the archive. A simulation experiment showed that in 84% of the trials a suitable classification was contained within the first five alternatives.

a c Abstract. A method is presented that enables automatic classification of medical diagnoses th... more a c Abstract. A method is presented that enables automatic classification of medical diagnoses through their reports in natural language. The method is based on the nearest neighbor rule: classification of a new case by assigning to it the same codes as the most similar case in the archive. A simulation experiment showed that in 84% of the trials a suitable classification was contained within the first five alternatives.

Studies in health technology and informatics, 2011

The Receiver-Operating Characteristic curve or ROC has been a long standing and well appreciated ... more The Receiver-Operating Characteristic curve or ROC has been a long standing and well appreciated tool to assess performance of classifiers or diagnostic tests. Likewise, the Area Under the ROC (AUC) has been a metric to summarize the power of a test or ability of a classifier in one measurement. This article aims to revisit the AUC, and ties it to key characteristics of the noncentral hypergeometric distribution. It is demonstrated that this statistical distribution can be used in modeling the behaviour of classifiers, which is of value for comparing classifiers.

Methods of information in medicine, 2001

The paper focuses on the problem of adequately coding pathology reports using SNOMED. Both the ag... more The paper focuses on the problem of adequately coding pathology reports using SNOMED. Both the agreement between pathologists in coding and the quality of a system that supports pathologists in coding pathology reports were evaluated. Six sets of three pathologists each received a different set of 40 pathology reports. Five different SNOMED code lines accompanied each pathology report. Three pathologists evaluated the correctness of each of these code lines. Kappa values and values for the reliability coefficients were determined to gain insight in the variance observed when coding pathology reports. The system that is evaluated compares a newly entered report, represented as a multi-dimensional word vector, with reports in a library, represented in the same way. The reports in the library are already coded. The system presents the code lines belonging to the five library reports most similar to the newly entered one to the pathologist in this way supporting the pathologist in deter...

Journal of the American Medical Informatics Association, 2011

Objective As clinical text mining continues to mature, its potential as an enabling technology fo... more Objective As clinical text mining continues to mature, its potential as an enabling technology for innovations in patient care and clinical research is becoming a reality. A critical part of that process is rigid benchmark testing of natural language processing methods on realistic clinical narrative. In this paper, the authors describe the design and performance of three state-of-the-art text-mining applications from the National Research Council of Canada on evaluations within the 2010 i2b2 challenge. Design The three systems perform three key steps in clinical information extraction: (1) extraction of medical problems, tests, and treatments, from discharge summaries and progress notes; (2) classification of assertions made on the medical problems; (3) classification of relations between medical concepts. Machine learning systems performed these tasks using large-dimensional bags of features, as derived from both the text itself and from external sources: UMLS, cTAKES, and Medline. Measurements Performance was measured per subtask, using micro-averaged F-scores, as calculated by comparing system annotations with ground-truth annotations on a test set. Results The systems ranked high among all submitted systems in the competition, with the following F-scores: concept extraction 0.8523 (ranked first); assertion detection 0.9362 (ranked first); relationship detection 0.7313 (ranked second). Conclusion For all tasks, we found that the introduction of a wide range of features was crucial to success. Importantly, our choice of machine learning algorithms allowed us to be versatile in our feature design, and to introduce a large number of features without overfitting and without encountering computing-resource bottlenecks.

Journal of the American Medical Informatics Association, 2013

Objective An analysis of the timing of events is critical for a deeper understanding of the cours... more Objective An analysis of the timing of events is critical for a deeper understanding of the course of events within a patient record. The 2012 i2b2 NLP challenge focused on the extraction of temporal relationships between concepts within textual hospital discharge summaries. Materials and methods The team from the National Research Council Canada (NRC) submitted three system runs to the second track of the challenge: typifying the time-relationship between pre-annotated entities. The NRC system was designed around four specialist modules containing statistical machine learning classifiers. Each specialist targeted distinct sets of relationships: local relationships, 'sectime'-type relationships, non-local overlap-type relationships, and non-local causal relationships. Results The best NRC submission achieved a precision of 0.7499, a recall of 0.6431, and an F1 score of 0.6924, resulting in a statistical tie for first place. Post hoc improvements led to a precision of 0.7537, a recall of 0.6455, and an F1 score of 0.6954, giving the highest scores reported on this task to date. Discussion and conclusions Methods for general relation extraction extended well to temporal relations, and gave top-ranked state-of-the-art results. Careful ordering of predictions within result sets proved critical to this success.

Journal of Biomedical Informatics, 2013

This paper addresses an information-extraction problem that aims to identify semantic relations a... more This paper addresses an information-extraction problem that aims to identify semantic relations among medical concepts (problems, tests, and treatments) in clinical text. The objectives of the paper are twofold. First, we extend an earlier one-page description (appearing as a part of [5]) of a top-ranked model in the 2010 I2B2 NLP Challenge to a necessary level of details, with the belief that feature design is the most crucial factor to the success of our system and hence deserves a more detailed discussion. We present a precise quantification of the contributions of a wide variety of knowledge sources. In addition, we show the end-to-end results obtained on the noisy output of a top-ranked concept detector, which could help construct a more complete view of the state of the art in the real-world scenario. As the second major objective, we reformulate our models into a composite-kernel framework and present the best result, according to our knowledge, on the same dataset.

International Journal of Healthcare Technology and Management, 1999

Computer Methods and Programs in Biomedicine, 2000

In this contribution two methods from the domain of information retrieval are compared. The goal ... more In this contribution two methods from the domain of information retrieval are compared. The goal of the retrieval is to select from a library of pathology reports those ones that are most similar to a given report. The SNOMED codes that accompany these reports are presented to the pathologist who has to code the given report with the aim to improve the quality of coding. The reports were represented either as a vector of words or as a vector of N-grams. Both 4-, 5-and 6-grams were used. The similarity of the reports was determined by comparing the SNOMED terms that were added to the reports. It could be concluded that the word-based method was consistently better than the N-gram method.

Computer Methods and Programs in Biomedicine, 1995

Automatic speech recognition has since long been seen as an ideal method for innovation in diagno... more Automatic speech recognition has since long been seen as an ideal method for innovation in diagnosis reporting. Speech technology now seems on the verge of introducing (commercially) attractive systems. The selection of a good speech recogniser is only one consideration in system design. Interface aspects, error handling, reporting method and implementation in the daily working routine are interwoven with the selection of an appropriate speech recognition technique, and should therefore be determined first.

Computer Methods and Programs in Biomedicine, 1997

This paper presents a method of automatic classification of clinical narrative through text compa... more This paper presents a method of automatic classification of clinical narrative through text comparison. A diagnosis report can be classified by searching archive texts that show a high textual similarity, and the 'nearest neighbor classifies the case. This paper describes the method's theoretical background and gives implementation details. Large scale simulation experiments were run with a wide range of histology reports. Results showed that for 80-84% of the trials, relevant classification lines were included among the first five alternatives. In 5% of the cases, retrieval was unsuccessful due to the absence of relevant archive reports. From the results it is concluded that the method is a versatile approach for finding potentially good classifications.

Behaviour & Information Technology, 1998

ABSTRACT

Journal of Clinical Epidemiology, 2016

To maximize the proportion of relevant studies identified for inclusion in systematic reviews (re... more To maximize the proportion of relevant studies identified for inclusion in systematic reviews (recall), complex time-consuming Boolean searches across multiple databases are common. Although MEDLINE provides excellent coverage of health science evidence, it has proved challenging to achieve high levels of recall through Boolean searches alone. Recall of one Boolean search method, the clinical query (CQ), combined with a ranking method, support vector machine (SVM), or PubMed-related articles, was tested against a gold standard of studies added to 6 updated Cochrane reviews and 10 Agency for Healthcare Research and Quality (AHRQ) evidence reviews. For the AHRQ sample, precision and temporal stability were examined for each method. Recall of new studies was 0.69 for the CQ, 0.66 for related articles, 0.50 for SVM, 0.91 for the combination of CQ and related articles, and 0.89 for the combination of CQ and SVM. Precision was 0.11 for CQ and related articles combined, and 0.11 for CQ and SVM combined. Related articles showed least stability over time. The complementary combination of a Boolean search strategy and a ranking strategy appears to provide a robust method for identifying relevant studies in MEDLINE.

Systematic reviews, Jan 22, 2016

Data abstraction, a critical systematic review step, is time-consuming and prone to errors. Curre... more Data abstraction, a critical systematic review step, is time-consuming and prone to errors. Current standards for approaches to data abstraction rest on a weak evidence base. We developed the Data Abstraction Assistant (DAA), a novel software application designed to facilitate the abstraction process by allowing users to (1) view study article PDFs juxtaposed to electronic data abstraction forms linked to a data abstraction system, (2) highlight (or "pin") the location of the text in the PDF, and (3) copy relevant text from the PDF into the form. We describe the design of a randomized controlled trial (RCT) that compares the relative effectiveness of (A) DAA-facilitated single abstraction plus verification by a second person, (B) traditional (non-DAA-facilitated) single abstraction plus verification by a second person, and (C) traditional independent dual abstraction plus adjudication to ascertain the accuracy and efficiency of abstraction. This is an online, randomized, t...

Permission is granted to quote short excerpts and to reproduce figures and tables from this repor... more Permission is granted to quote short excerpts and to reproduce figures and tables from this report, provided that the source of such material is fully acknowledged.

Studies in Health Technology and Informatics, 2011

The Receiver-Operating Characteristic curve or ROC has been a long standing and well appreciated ... more The Receiver-Operating Characteristic curve or ROC has been a long standing and well appreciated tool to assess performance of classifiers or diagnostic tests. Likewise, the Area Under the ROC (AUC) has been a metric to summarize the power of a test or ability of a classifier in one measurement. This article aims to revisit the AUC, and ties it to key characteristics of the noncentral hypergeometric distribution. It is demonstrated that this statistical distribution can be used in modeling the behaviour of classifiers, which is of value for comparing classifiers.

Information Retrieval experiments rarely examine more than a small number of user or system chara... more Information Retrieval experiments rarely examine more than a small number of user or system characteristics because of the limited availability of human subjects. In this article we present an interaction model, and based on that an experimental method. This automated method helps identify which user and system variables are relevant, which are independent of other variables, and which ranges of the variables are most important.

International Journal of Healthcare Technology and Management, 1999

Methods of Information in Medicine

A method is presented for assigning classification codes to pathology reports by searching simila... more A method is presented for assigning classification codes to pathology reports by searching similar reports from an archive collection. The key for searching is textual similarity, which estimates the true, semantic similarity. This method does not require explicit modeling, and can be applied to any language or any application domain that uses natural language reporting. A number of simulation experiments was run to assess the accuracy of the method and to indicate the role of size of the archive and the transfer of document collections across laboratories. In at least 63% of the simulation trials, the most similar archive text offered a suitable classification on organ, origin and diagnosis. In 85 to 90% of the trials, the archive's best solution was found within the first five similar reports. The results indicate that the method is suitable for its purpose: suggesting potentially correct classifications to the reporting diagnostician.

A method is presented that enables automatic classification of medical diagnoses through their re... more A method is presented that enables automatic classification of medical diagnoses through their reports in natural language. The method is based on the nearest neighbor rule: classification of a new case by assigning to it the same codes as the most similar case in the archive. A simulation experiment showed that in 84% of the trials a suitable classification was contained within the first five alternatives.

a c Abstract. A method is presented that enables automatic classification of medical diagnoses th... more a c Abstract. A method is presented that enables automatic classification of medical diagnoses through their reports in natural language. The method is based on the nearest neighbor rule: classification of a new case by assigning to it the same codes as the most similar case in the archive. A simulation experiment showed that in 84% of the trials a suitable classification was contained within the first five alternatives.

Studies in health technology and informatics, 2011

The Receiver-Operating Characteristic curve or ROC has been a long standing and well appreciated ... more The Receiver-Operating Characteristic curve or ROC has been a long standing and well appreciated tool to assess performance of classifiers or diagnostic tests. Likewise, the Area Under the ROC (AUC) has been a metric to summarize the power of a test or ability of a classifier in one measurement. This article aims to revisit the AUC, and ties it to key characteristics of the noncentral hypergeometric distribution. It is demonstrated that this statistical distribution can be used in modeling the behaviour of classifiers, which is of value for comparing classifiers.

Methods of information in medicine, 2001

The paper focuses on the problem of adequately coding pathology reports using SNOMED. Both the ag... more The paper focuses on the problem of adequately coding pathology reports using SNOMED. Both the agreement between pathologists in coding and the quality of a system that supports pathologists in coding pathology reports were evaluated. Six sets of three pathologists each received a different set of 40 pathology reports. Five different SNOMED code lines accompanied each pathology report. Three pathologists evaluated the correctness of each of these code lines. Kappa values and values for the reliability coefficients were determined to gain insight in the variance observed when coding pathology reports. The system that is evaluated compares a newly entered report, represented as a multi-dimensional word vector, with reports in a library, represented in the same way. The reports in the library are already coded. The system presents the code lines belonging to the five library reports most similar to the newly entered one to the pathologist in this way supporting the pathologist in deter...

Journal of the American Medical Informatics Association, 2011

Objective As clinical text mining continues to mature, its potential as an enabling technology fo... more Objective As clinical text mining continues to mature, its potential as an enabling technology for innovations in patient care and clinical research is becoming a reality. A critical part of that process is rigid benchmark testing of natural language processing methods on realistic clinical narrative. In this paper, the authors describe the design and performance of three state-of-the-art text-mining applications from the National Research Council of Canada on evaluations within the 2010 i2b2 challenge. Design The three systems perform three key steps in clinical information extraction: (1) extraction of medical problems, tests, and treatments, from discharge summaries and progress notes; (2) classification of assertions made on the medical problems; (3) classification of relations between medical concepts. Machine learning systems performed these tasks using large-dimensional bags of features, as derived from both the text itself and from external sources: UMLS, cTAKES, and Medline. Measurements Performance was measured per subtask, using micro-averaged F-scores, as calculated by comparing system annotations with ground-truth annotations on a test set. Results The systems ranked high among all submitted systems in the competition, with the following F-scores: concept extraction 0.8523 (ranked first); assertion detection 0.9362 (ranked first); relationship detection 0.7313 (ranked second). Conclusion For all tasks, we found that the introduction of a wide range of features was crucial to success. Importantly, our choice of machine learning algorithms allowed us to be versatile in our feature design, and to introduce a large number of features without overfitting and without encountering computing-resource bottlenecks.

Journal of the American Medical Informatics Association, 2013

Objective An analysis of the timing of events is critical for a deeper understanding of the cours... more Objective An analysis of the timing of events is critical for a deeper understanding of the course of events within a patient record. The 2012 i2b2 NLP challenge focused on the extraction of temporal relationships between concepts within textual hospital discharge summaries. Materials and methods The team from the National Research Council Canada (NRC) submitted three system runs to the second track of the challenge: typifying the time-relationship between pre-annotated entities. The NRC system was designed around four specialist modules containing statistical machine learning classifiers. Each specialist targeted distinct sets of relationships: local relationships, 'sectime'-type relationships, non-local overlap-type relationships, and non-local causal relationships. Results The best NRC submission achieved a precision of 0.7499, a recall of 0.6431, and an F1 score of 0.6924, resulting in a statistical tie for first place. Post hoc improvements led to a precision of 0.7537, a recall of 0.6455, and an F1 score of 0.6954, giving the highest scores reported on this task to date. Discussion and conclusions Methods for general relation extraction extended well to temporal relations, and gave top-ranked state-of-the-art results. Careful ordering of predictions within result sets proved critical to this success.

Journal of Biomedical Informatics, 2013

This paper addresses an information-extraction problem that aims to identify semantic relations a... more This paper addresses an information-extraction problem that aims to identify semantic relations among medical concepts (problems, tests, and treatments) in clinical text. The objectives of the paper are twofold. First, we extend an earlier one-page description (appearing as a part of [5]) of a top-ranked model in the 2010 I2B2 NLP Challenge to a necessary level of details, with the belief that feature design is the most crucial factor to the success of our system and hence deserves a more detailed discussion. We present a precise quantification of the contributions of a wide variety of knowledge sources. In addition, we show the end-to-end results obtained on the noisy output of a top-ranked concept detector, which could help construct a more complete view of the state of the art in the real-world scenario. As the second major objective, we reformulate our models into a composite-kernel framework and present the best result, according to our knowledge, on the same dataset.

International Journal of Healthcare Technology and Management, 1999

Computer Methods and Programs in Biomedicine, 2000

In this contribution two methods from the domain of information retrieval are compared. The goal ... more In this contribution two methods from the domain of information retrieval are compared. The goal of the retrieval is to select from a library of pathology reports those ones that are most similar to a given report. The SNOMED codes that accompany these reports are presented to the pathologist who has to code the given report with the aim to improve the quality of coding. The reports were represented either as a vector of words or as a vector of N-grams. Both 4-, 5-and 6-grams were used. The similarity of the reports was determined by comparing the SNOMED terms that were added to the reports. It could be concluded that the word-based method was consistently better than the N-gram method.

Computer Methods and Programs in Biomedicine, 1995

Automatic speech recognition has since long been seen as an ideal method for innovation in diagno... more Automatic speech recognition has since long been seen as an ideal method for innovation in diagnosis reporting. Speech technology now seems on the verge of introducing (commercially) attractive systems. The selection of a good speech recogniser is only one consideration in system design. Interface aspects, error handling, reporting method and implementation in the daily working routine are interwoven with the selection of an appropriate speech recognition technique, and should therefore be determined first.

Computer Methods and Programs in Biomedicine, 1997

This paper presents a method of automatic classification of clinical narrative through text compa... more This paper presents a method of automatic classification of clinical narrative through text comparison. A diagnosis report can be classified by searching archive texts that show a high textual similarity, and the 'nearest neighbor classifies the case. This paper describes the method's theoretical background and gives implementation details. Large scale simulation experiments were run with a wide range of histology reports. Results showed that for 80-84% of the trials, relevant classification lines were included among the first five alternatives. In 5% of the cases, retrieval was unsuccessful due to the absence of relevant archive reports. From the results it is concluded that the method is a versatile approach for finding potentially good classifications.

Behaviour & Information Technology, 1998

ABSTRACT