Usefulness as the criterion for evaluation of interactive information retrieval (original) (raw)
Related papers
The development of a method for the evaluation of interactive information retrieval systems
Journal of Documentation, 1997
The paper describes the ideas and assumptions underlying the development of a new method for the evaluation and testing of interactive information retrieval (IR) systems, and reports on the initial tests of the proposed method. The method is designed to collect different types of empirical data, i.e. cognitive data as well as traditional systems performance data. The method is based on the novel concept of a 'simulated work task situation' or scenario and the involvement of real end users. The method is also based on a mixture of simulated and real information needs, and involves a group of test persons as well as assessments made by individual panel members. The relevance assessments are made with reference to the concepts of topical as well as situational relevance. The method takes into account the dynamic nature of information needs which are assumed to develop over time for the same user, a variability which is presumed to be strongly connected to the processes of relevance assessment.
Experimental components for the evaluation of interactive information retrieval systems
Journal of Documentation, 2000
This paper presents a set of basic components which constitutes the experimental setting intended for the evaluation of interactive information retrieval (IIR) systems, the aim of which is to facilitate evaluation of IIR systems in a way which is as close as possible to realistic IR processes. The experimental setting consists of three components: (1) the involvement of potential users as test persons;
Proceedings of the American Society for Information Science and Technology, 2010
User satisfaction, though difficult to measure, is the main goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, user effectiveness also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new measures to evaluate IIR systems, the Normalized Task Completion Time (NT) and the Normalized User Effectiveness (NUE). The two measures overcome limitations of existing measures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two measures and the user satisfaction and effectiveness of a given IR system. The learning effects described by NT, NUE, and the task completion time were also studied and compared. The results show that NT is strongly correlated with user satisfaction, NUE is a better indicator of system effectiveness than task completion time, and both new measures are superior to task completion time in describing the learning effect of the given IR system.
The influence of relevance levels on the effectiveness of interactive information retrieval
Journal of the American Society for …, 2004
In this paper, we focus on the effect of graded relevance on the results of interactive information retrieval (IR) experiments based on assigned search tasks in a test collection. A group of 26 subjects searched for four Text REtrieval Conference (TREC) topics using automatic and interactive query expansion based on relevance feedback. The TREC-and user-suggested pools of relevant documents were reassessed on a four-level relevance scale. The results show that the users could identify nearly all highly relevant documents and about half of the marginal ones. Users also selected a fair number of irrelevant documents for query expansion. The findings suggest that the effectiveness of query expansion is closely related to the searchers' success in retrieving and identifying highly relevant documents for feedback. The implications of the results on interpreting the findings of past experiments with liberal relevance thresholds are also discussed.
Evaluating Interactive Information Retrieval Systems
2012
Neste artigo e proposta uma extensa metodologia para avaliar a recuperacao da informacao interativa. A proposta baseia-se em principios fundamentais apresentados na literatura de avaliacao para definir os objetivos do sistema ou ferramenta a ser avaliado, e inferir as medidas e os criterios de sucesso na consecucao dos objetivos. Propoe-se que, ao avaliar uma ferramenta de pesquisa, seja analisado em que medida ela beneficia os utilizadores, aumentando a sua capacidade de pesquisa e, consequentemente, contribuindo para a qualidade da lista de resultados. Alem da qualidade da lista de resultados, e importante avaliar ate que ponto o processo de busca e as ferramentas que o suportam atingem os seus objetivos
Rethinking Interest in Studies of Interactive Information Retrieval
Proceedings of the 2021 Conference on Human Information Interaction and Retrieval
Interest characterizes a cognitive-emotional relationship between people and information and is a key construct in human information interaction. As a motivational variable, interest has been widely studied in psychology and education, but has received less consistent and theory-driven attention in the field of interactive information retrieval (IIR). In this perspective paper we examine the role of interest and review how it has been studied and operationalized in IIR research. We draw upon a survey of 58 research studies that have manipulated, controlled or measured searcher interest in some way. The intent of the paper is to raise the profile of interest as a user-centred variable in IIR and to advocate for more conceptual and methodological consistency in future studies to better evaluate the impact of interest in information search. CCS CONCEPTS •Information systems~Information retrieval~Users and interactive retrieval •Human-centered computing~Human computer interaction (HCI)~HCI design and evaluation methods~User studies
A Framework to Evaluate Interface Suitability for a Given Scenario of Textual Information Retrieval
J. Univers. Comput. Sci., 2011
Visualization of search results is an essential step in the textual Information Retrieval (IR) process. Indeed, Information Retrieval Interfaces (IRIs) are used as a link between users and IR systems, a simple example being the ranked list proposed by common search engines. Due to the importance that takes visualization of search results, many interfaces have been proposed in the last decade (which can be textual, 2D or 3D IRIs). Two kinds of evaluation methods have been developed: (1) various evaluation methods of these interfaces were proposed aiming at validating ergonomic and cognitive aspects; (2) various evaluation methods were applied on information retrieval systems (IRS) aiming at measuring their effectiveness. However, as far as we know, these two kinds of evaluation methods are disjoint. Indeed, considering a given IRI associated to a given IRS, what happens if we associate this IRI to another IRS not having the same effectiveness. In this context, we propose an IRI evalu...
Towards new measures of information retrieval evaluation
1995
All of the methods cumently used to evaluate information retrieval (Ill) systems have limitations in their ability to measure how well users are able to acquire information. We utilized on approach to assessing information obtained based on the user's ability to answer qtrcstions from a shortanswer test. Senior medical students took the ten-question test and then searched one of (WO IR systems on the five questions for which they were least ccrtuin of their answer. Our results showed thtit pre-searching scores on the test were low but that searching yielded a high proportion of answers with both systems. These methods are able to measure information obtained, and will be used in subsequent studies to assess differences among IR systems.