Representation Issues in Multimedia Case Retrieval (original) (raw)

Refining the universal indexing frame to support retrieval of tutorial stories

1994

A central challenge in multi-media retrieval is the development of indexing schemes that represent the important features of multi-media information. In particular, the retrieval of video clips for presentation in a case-based teaching environment requires an indexing representation that can be used to assess each clip's educational relevance. Surface features, which are relied on by traditional information retrieval techniques, do not adequately predict how a story will relate to a student's situation. We have developed a system that retrieves video stories using an indexing scheme that represents the points of interest, or anomalies, in a story. The representation is a refinement of the Universal Indexing Frame proposal .

Tailoring Retrieval to Support Case-Based Teaching

Aaai, 1994

This paper describes how a computer program can support learning by retrieving and presenting relevant stories drawn from a video case base. Although this is an information retrieval problem, it is not a problem that fits comfortably within the classical IR model because in the classical model the computer system is too passive. The standard model of IR assumes that the user will take the initiative to formulate retrieval requests, but a teaching system must be able to initiate retrieval and formulate retrieval requests automatically. We describe a system, called SPIEL, that performs this type of retrieval, and discuss theoretical challenges addressed in implementing such a system. These challenges include the development of a representation language for indexing the system's video library, and the development of set of retrieval strategies and recognition knowledge that allow the system to locate educationally relevant stories.

Retrieval strategies for tutorial stories

1993

Retrieving stories to present to students is a challenging application of case retrieval. This paper describes SPIEL, a system that retrieves tutorial stories, stored on video, for educational purposes. Although CBR methods are employed in SPIEL, its task requires a different emphasis than typically found in problem-solving CBR systems. One of the most significant of these is the centrality of multi-purpose retrieval in educational storytelling. SPIEL has a set of storytelling strategies, corresponding to different educational roles that stories can play, such as providing counter-examples or projecting possible results. To find stories that can fill these roles, the system uses a variety of comparisons including similarity, dissimilarity, and other relations. This paper describes three of these strategies in detail, showing how the strategies function in retrieval, what kinds of knowledge they use, and how they make use of SPIEL's indices.

Supporting Learning Through Active Retrieval of Video Stories

Expert Systems with Applications, 1995

This paper describes how a computer program can support learning by retrieving and presenting relevant stories drawn from a video case base. Although this is an information retrieval problem, it is not a problem that fits comfortably within the classical IR model .

Strategic retrieval of tutorial stories

This paper describes SPIEL, a system for retrieving and presenting tutorial stories for students who are using a social simulation to learn social skills. SPIEL's task is primarily retrieval, but it requires techniques from casebased reasoning to perform it. SPIEL's stories are stored in video form, which prevents the use of text-based processing or indexing. Instead of using a story's text, SPIEL uses complex structured indices intended to represent what the story is about.

A tale of two images: the quest to create a story-based image indexing system

Journal of Documentation, 2014

Purpose-This conceptual paper considers the possibility of designing a story-based image indexing system based on users' descriptions of images. It reports a pilot study which uses users' descriptions of two images. Design/methodology/approach-Eight interviews were undertaken to investigate storytelling in user interpretations of the images. Following this, storytelling was explored as an indexing input method. Twenty-six research subjects were asked to create stories about the images, which were then considered in relation to conventional story elements and in relation to Rafferty and Hidderley's 2005 image modality model. Findings-The results of the semi-structured interviews revealed that the majority of interpretations incorporated story elements related to setting, character, plot, literary devices, and themes. The fifty-two image stories included story elements identified in the first part of the project, and suggested that the image modality model is robust enough to deal with the 'writerly' images used in this study. In addition, using storytelling as an input method encourages the use of verbs and connotative level responses. Originality/value-User indexing is generally based on paradigmatic approaches to concept analysis and interpretation in the form of tagging; the novelty of the current study is its exploration of syntagmatic approaches to user indexing in the form of story-telling. It is a pilot, proof of concept study, but it is hoped that it might stimulate further interest in syntagmatic approaches to user indexing.

Browsing the Structure of Multimedia Stories

Stories may be analyzed as sequences of causally-related events and reactions to those events by the characters. We employ a notation of plot elements, similar to one developed by Lehnert,and we extend that by forming higher level ``story threads''Stories may be analyzed as sequences of causally-related events and reactions to those events by the characters. We employ a notation of plot elements, similar to one developed by Lehnert,and we extend that by forming higher level ``story threads''We apply the browser to Corduroy, a children's short feature which was analyzed in detail. We provide additional illustrations with analysis of Kiss of Death, a Film Noir classic. Effectively, the browser provides a framework for interactive summaries, video of the narrative

TREC 2003 Video Retrieval and Story Segmentation Task at NUS PRIS

2003

This paper describes the details of our systems for story segmentation task and search task of the TREC-2003 Video Track. In story segmentation task, we propose a two-level multi-modal framework. First we analyze the video at the shot level using a variety of low and high-level features, and classify the shots into pre-defined categories using a Decision Tree. Next we perform HMM analysis in order to identify news story boundaries. The two-level framework has been found to be effective in overcoming the data sparseness problem in machine learning. In the search task, we perform news video retrieval by integrating multiple intra-video features and external knowledge sources. The retrieval is performed in three stages. Stage 1 uses mainly question-ansering style text retrieval technology. It analyses the text query issued by the users and extracts relevant video stories based on ASR, and external resources like WordNet and related news articles on the web. The second stage acts as a concept filter, which eliminates the irrelevant video shots in the stories retrieved by text query system. The third stage re-ranks the retrieved shots using the image and video retrieval techniques with relevance feedback. Our system emphasizes the automated retrieve process. The experiments demonstrate the effectiveness of the story segmentation system and video retrieval system.

The Semantic Pathfinder: Using an Authoring Metaphor for Generic Multimedia Indexing

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000

This paper presents the semantic pathfinder architecture for generic indexing of multimedia archives. The semantic pathfinder extracts semantic concepts from video by exploring different paths through three consecutive analysis steps, which we derive from the observation that produced video is the result of an authoring-driven process. We exploit this authoring metaphor for machine-driven understanding. The pathfinder starts with the content analysis step. In this analysis step, we follow a data-driven approach of indexing semantics. The style analysis step is the second analysis step. Here, we tackle the indexing problem by viewing a video from the perspective of production. Finally, in the context analysis step, we view semantics in context. The virtue of the semantic pathfinder is its ability to learn the best path of analysis steps on a per-concept basis. To show the generality of this novel indexing approach, we develop detectors for a lexicon of 32 concepts and we evaluate the semantic pathfinder against the 2004 NIST TRECVID video retrieval benchmark, using a news archive of 64 hours. Top ranking performance in the semantic concept detection task indicates the merit of the semantic pathfinder for generic indexing of multimedia archives.