Subjective QoE assessment on video service: Laboratory controllable approach (original) (raw)
Related papers
Video Quality of Experience in the Presence of Accessibility and Retainability Failures
Accurate Quality of Experience measurement for streaming video has become more crucial with the increase in demand for online video viewing. Quantifying video Quality of Experience is a challenging task. Significant efforts to quantify video Quality of Experience have primarily focused on the measurement of Quality of Experience for videos with network and compression related impairments. These impairments, however, may not always be the only main factors affecting Quality of Experience in an entire video viewing session. In this paper, we evaluate Quality of Experience for entire video viewing sessions, from the beginning to the end. In doing so, we evaluate videos with temporary interruptions as well as those with permanent interruptions or failures. We consider two types of failures, namely Accessibility and Retainability failures, and present the results of two subjective studies. These results indicate: (a) Accessibility and Retainability failures are rated lower compared to temporary interruption impairments; (b) Accessibility failures are rated close to the lowest value on the rating scale; and (c) the traditionally used 5-point scale to measure video Quality of Experience is not sufficient in the presence of Accessibility and Retainability failures.
Qualitative Monitoring of Video Quality of Experience
2011 IEEE International Symposium on Multimedia, 2011
Real-time monitoring of multimedia Quality of Experience is a critical task for the providers of multimedia delivery services: from television broadcasters to IP content delivery networks or IPTV. For such scenarios, meaningful metrics are required which can generate useful information to the service providers that overcome the limitations of pure Quality of Service monitoring probes. However, most of objective multimedia quality estimators, aimed at modeling the Mean Opinion Score, are difficult to apply to massive quality monitoring. Thus we propose a lightweight and scalable monitoring architecture called Qualitative Experience Monitoring (QuEM), based on detecting identifiable impairment events such as the ones reported by the customers of those services. We also carried out a subjective assessment test to validate the approach and calibrate the metrics. Preliminary results of this test set support our approach.
Proc. of 21st ITC Specialist Seminar …, 2010
Interactive video services, like video telephony, social TV or on-line gaming, are about to become a significant part of the service portfolio for telecommunication service providers. As the future commercial success of these services will depend essentially on their end-to-end quality as perceived by the end user, appropriate Quality-of-Experience (QoE) measurement methods are of paramount importance. The aim of this paper is to provide an application-oriented re-evaluation of actual QoE metrics for these services and to design a test methodology for evaluating user perceived experience which goes beyond standard Mean Opinion Score (MOS) metrics. To this end, we describe a specific test scenario designed for assessing the currently recommended question sets from two ITU recommendations. We determine the goodness of fit of these question sets to the user perceived quality dimension. Altogether, the resulting reduced set of items (questions) provides a significant step towards a more realistic methodology for assessing audio-visual quality perception.
Quality-of-Experience (QoE) is a human centric notion that produces the blue print of human perception, feelings, needs and intentions while Quality-of-Service (QoS) is a technology centric metric used to assess the performance of a multimedia application and/or network. To ensure superior video QoE, it is important to understand the relationship between QoE and QoS. To achieve this goal, we conducted a pilot subjective user study simulating a video streaming service over a broadband network with varying distortion scenarios, namely packet losses (0, 0.5, 1, 3,7, and 15%), packet reorder (0, 1, 5, 10, 20, and 30%), and coding bit rates (100, 400, 600, and 800 Kbps). Users were asked to rate their experience using a subjective quantitative metric (termed Perceived Video Quality, PVQ) and qualitative indicators of “experience.” Simulation results suggest a) an exponential relationship between PVQ and packet loss and between PVQ and packet reorder, and b) a logarithmic relationship between PVQ and video bit rate. Similar trends were observed with the qualitative indicators. Exploratory analysis with two objective video quality metrics suggests that trends similar to those obtained with the subjective ratings were obtained, particularly with a full-reference metric.
In this paper we examine the reliability of subjective rating judgments along a single dimension, focusing on estimates of technical quality produced by integrity impairments and failures (non-accessibility, and non-retainability) associated with viewing video. There is often considerable variability, both within and between individuals, in subjective rating tasks. In the research reported here we consider different approaches to screening out unreliable participants. We review available alternatives, including a method developed by the ITU, a method based on screening outliers, a method based on strength of correlations with an assumed “natural” ordering of impairments, and a clustering technique that makes no assumptions about the data. We report on an experiment that assesses subjective quality of experience associated with impairments and failures of online video. We then assess the reliability of the results using a correlation method and a clustering method, both of which give similar results. Since the clustering method utilized here makes fewer assumptions about the data, it may be a useful supplement to existing techniques for assessing reliability of participants when making subjective evaluations of the technical quality of videos.
Impact of Human and Content Factors on Quality of Experience of Online Video Streaming
2020
Although expensive, but the most reliable measure of user perception is by direct human interaction by taking input from the user about a stimulus quality. In our previous studies, we have identified some subjects getting bored and losing focus by rating lots of video clips of small duration during subjective quality assessments. Moreover, the psychological effects, i.e. user delight, frequency of watching online videos (experience), mood, etc. must not influence the user Mean Opinion Score (MOS) for determining the quality of the shown stimuli. In this paper, we have investigated the impact of user delight, frequency of watching online video content (experience) and different mood levels on MOS for streamed video stimuli in various network conditions by subjective quality assessments. We have observed a slight tendency of better scores when the user likes the stimulus. However, our results show that if the subjective assessments are conducted by carefully following the guidelines, the users impartially rate the video stimuli solely based on the quality artifacts irrespective of their delight towards the shown content. Although, we have observed an effect of user mood on MOS ratings; for almost all the stimuli, but the results suggest the need of more detailed study; i.e. with a large and diverse set of subjects, to obtain significant statistical relevance.
Quality of Experience Assessment of Video Streaming
International Journal of Advanced Trends in Computer Science and Engineering, 2021
This study aims to determine the user's satisfaction level of online streaming by using different web browsers. At the client layer, the assessment of the user's QoE is conducted by evaluating the performance of three web browsers (Google Chrome, Mozilla Firefox, and Internet Explorer). We took the subjective test by conducting different experiments with the users and ask the users to assign ratings on the provided questionnaires, and from those ratings, we calculated results in the form of Mean Opinion Score.
Subjective Evaluation of Quality of Experience for Video Streaming Service
2014
This paper presents the results of the subjective evaluation of Quality of Experience (QoE) for video streaming service. The evaluation was conducted among 602 test subjects who rated the quality of 72 videos by watching them at their homes. The quality of each video was uniquely distorted (packet loss rate, number of packet loss occurrences during video streaming and total duration of those occurrences varied in each video). The results revealed that, in real-life environment, test subjects failed to notice significant number of video segments whose quality was degraded, they could not accurately quantify their total duration and they usually did not maintain negative attitude about perceived quality distortions.
Although adaptive video streaming solutions have become very popular over the last years, only a limited number of studies so far have investigated Quality of Experience (QoE) in the context of such services from a real user perspective. In this paper, we present results from a user study (N=32) on adaptive video streaming QoE. Content (i.e., music video clips) was streamed on portable terminal devices (namely iPads), using different bitrate profiles which represent realistic bandwidth conditions. All users conducted the test in two different usage scenarios, representing different social settings. QoE was evaluated in terms of traditional QoE-measures and complemented with a number of alternative, affectivestate related measures. The findings indicate differences between the considered bitrate profiles in terms of QoE (traditional and alternative measures) and indicate that a lower, constant bitrate seems to be preferred over the changing bitrate which is -on averagehigher. This is the case for both separate usage scenarios. Although we found no significant influence of the social setting when considering traditional QoE evaluation measures, some significant differences between both settings were detected when evaluating QoE in terms of delight and annoyance.