Stephen Doherty | The University of New South Wales (original) (raw)

Papers by Stephen Doherty

This study investigates the impact of same-language subtitles on the immersion into audiovisual n... more This study investigates the impact of same-language subtitles on the immersion into audiovisual narratives as a function of the viewer’s language (native or foreigner). Students from two universities in Australia and one in Spain were assigned randomly to one of two experimental groups, in which they saw a drama with the original English soundtrack either with same-language English subtitles (n=81) or without subtitles (n=92). The sample included an English native control group, and Mandarin Chinese, Korean, and Spanish groups with English as a foreign language. Participants used post-hoc Likert scales to self-report their presence, transportation to the narrative world, perceived realism, identification with the characters, and enjoyment. The main results showed that subtitles did not significantly reduce these measures of immersion. However, subtitles produced higher transportation, identification with the characters, and perceived realism scores, where the first language of viewers and their viewing habits accounted for most of this variance. Moreover, presence and enjoyment were unaffected by either condition or language. Finally, the main results also revealed that transportation to the narrative world appears to be the most revealing measure of immersion in that it shows the strongest and most consistent correlations, and is a significant predictor of enjoyment.

Japanese and other Asian TV producers have been deploying multi-colored, and highly visible, intr... more Japanese and other Asian TV producers have been deploying multi-colored, and highly visible, intra-lingual captions on TV programs to enhance their appeal and to influence their viewers’ interpretations. The practice of adding these captions is far from innocent and is prone to abuse and overuse due to the lack of official guidelines and an evidence base. We conducted a multimodal analysis within the framework of relevance theory to provide an empirically supported insight into the way in which these captions, known as “telop” in Japan, form part of a production’s deliberate and careful media design. Our findings suggest that telop are deployed in conjunction with other communicative resources that are deliberately used to influence viewers’ interpretations, to enhance and make affective values in TV programs more explicit. The increasing use of diegetically integrated captions elsewhere further justifies the need for critical TV and new media research on telop.

This chapter presents the findings of a study to investigate the impact of subtitles on the proce... more This chapter presents the findings of a study to investigate the impact of subtitles on the processing of audiovisual texts in terms of levels of self-reported engagement with the text. It also presents a methodology for investigating the neural processing of subtitles using electroencephalography (EEG) and psychometrics. By establishing the beta coherence between prefrontal and posterior regions, we can determine whether the effect of subtitles can be captured through EEG, and how this is related to psychological immersion. We therefore use EEG to investigate the cognitive processing of subtitled film in combination with subjective psychometrics. Initial validation of this methodology shows that adding same language subtitles results in statistically significantly higher levels of immersion and enjoyment, lending support to a view that subtitles facilitate (dis)embodied cognition. We propose these as the first steps towards a cognitive audiovisual translatology.

Technological advances have led to unprecedented changes in translation as a means of interlingua... more Technological advances have led to unprecedented changes in translation as a means of interlingual communication. This article discusses the impact of two major technological developments of contemporary translation: computer-assisted translation tools and machine translation. These technologies have increased productivity and quality in translation, supported international communication, and demonstrated the growing need for innovative technological solutions to the age-old problem of the language barrier. However, these tools also represent significant challenges and uncertainties for the translation profession and the industry. In highlighting the need for increased awareness and technological competencies, I propose that these challenges can be overcome and translation technologies will become even more integral in interlingual communication.

In many countries such as Japan, the use of intra-lingual Open Caption Telop (OCT) for multimodal... more In many countries such as Japan, the use of intra-lingual Open Caption Telop (OCT) for multimodal communications such as television and video is highly prevalent, especially in the context of entertainment and informative programmes. Unlike the conventional use of intra-lingual subtitles for the hard-of-hearing population, this particular adoption of OCT relates to the producers’ attempts at manipulating the viewers’ cognitive processing to achieve a directed and desired behaviour such as humour, enhanced comprehension and retention of content, or the reading of editorial opinions. In this paper, working within the frameworks of relevance theory and cognitive psychology, we examine the relations between linguistically represented stimulus and its effect on viewer experience, both during and after stimulus presentation. We demonstrate how OCT enables TV producers to achieve their intentions by hijacking the viewers’ interpretation processes by guiding viewers to a desired outcome. Us...

The following paper has been prepared by two independent researchers – Dr Stephen Doherty (UNSW) ... more The following paper has been prepared by two independent researchers – Dr Stephen Doherty (UNSW) and Dr Ignacio Garcia (UWS). It discusses options and alternatives for NAATI transitioning to Keyboarded Translator Assessment. NAATI commissioned this to support discussions with the broad group of stakeholders who engage with translator assessments – as the taker of tests, the deliverer of assessments or for other reasons. NAATI is seeking feedback and comment on the options put forward in the paper. After considering the information in the paper and any feedback received NAATI intends to publish a response including how it intends to move away from the current paper-based model for translator assessment. If you have feedback please email it to intproject@naati.com.au.

The Interpreter and Translator Trainer, 2014

In this paper we argue that the time is ripe for translator educators to engage with Statistical ... more In this paper we argue that the time is ripe for translator educators to engage with Statistical Machine Translation (SMT) in more profound ways than they have done to date. We explain the basic principles of SMT and reflect on the role of humans in SMT workflows. Against a background of diverging opinions on the latter, we argue for a holistic approach to the integration of SMT into translator training programmes, one that empowers rather than marginalises translators. We discuss potential barriers to the use of SMT by translators generally and in translator training in particular, and propose some solutions to problems thus identified. More specifically, cloud-based services are proposed as a means of overcoming some of the technical and ethical challenges posed by more advanced uses of SMT in the classroom. Ultimately the paper aims to pave the way for the design and implementation of a new translator-oriented SMT syllabus at our own University and elsewhere.

The Interpreter and Translator Trainer, 2014

Despite the acknowledged importance of translation technology in translation studies programmes a... more Despite the acknowledged importance of translation technology in translation studies programmes and the current ascendancy of Statistical Machine Translation (SMT), there has been little reflection to date on how SMT can or should be integrated into the translation studies curriculum. In a companion paper we set out a rationale for including a holistic SMT syllabus in the translation curriculum. In this paper, we show how the priorities and aspirations articulated in that source can be operationalised in the translation technology classroom and lab. We draw on our experience of designing and evaluating an SMT syllabus for a cohort of postgraduate student translators at Dublin City University in 2012. In particular, we report on data derived from a mixed-methods approach that aims to capture the students' view of the syllabus and their self-assessment of their own learning. Using the construct of self-efficacy, we show significant increases in students' knowledge of and confidence in using machine translation in general and SMT in particular, after completion of teaching units in SMT. We report on additional insights gleaned from student assignments, and conclude with ideas for future refinements of the syllabus.

Machine Translation, 2010

Eye tracking has been used successfully as a technique for measuring cognitive load in reading, p... more Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc. for some time now. Its application as a technique for measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semi-automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. Ten native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded non-invasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the “bad” sentences, while average fixation duration and pupil dilations were not found to be substantially different for output rated as good and output rated as bad. Comparisons between HTER scores and eye gaze data were also found to correlate well with gaze time and fixation count, but not with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as a semi-automatic MT evaluation technique, which does not require bi-lingual knowledge, and which can potentially tap into the end users’ experience of machine translation output.

The Journal of Specialised Translation, Jan 18, 2013

This article presents an on-going evaluation of translation technology lab sessions in a universi... more This article presents an on-going evaluation of translation technology lab sessions in a university setting. Lab sessions are practical supplements that allow student translators to develop upon what has already been learned in theory via traditional lectures. Both of these components develop the technical competencies required for professional work in translation, post-editing, and related areas. Data pertaining to the evaluation of the labs were collated and analysed using a mixed-methods approach, where the current paper focuses on the qualitative aspects. Upon examination of the data, several areas were identified as having the potential to be improved upon: attitudes, abilities, and resources. After making modifications to the labs, a follow-up study was carried out in the following academic year, the results of which were compared and contrasted with that of the first study to ascertain if the efforts to improve the labs were successful, and to guide further work.

National Accreditation Authority for Translators and Interpreters, May 25, 2015

The following paper has been prepared by two independent researchers – Dr Stephen Doherty (UNSW) ... more The following paper has been prepared by two independent researchers – Dr Stephen Doherty (UNSW) and Dr Ignacio Garcia (UWS). It discusses options and alternatives for NAATI transitioning to Keyboarded Translator Assessment.
NAATI commissioned this to support discussions with the broad group of stakeholders who engage with translator assessments – as the taker of tests, the deliverer of assessments or for other reasons.
NAATI is seeking feedback and comment on the options put forward in the paper.
After considering the information in the paper and any feedback received NAATI intends to publish a response including how it intends to move away from the current paper-based model for translator assessment.
If you have feedback please email it to intproject@naati.com.au.

The 9th Edition of the Language Resources and Evaluation Conference, At Reykjavik, Iceland, Jun 26, 2014

This paper aims to automatically identify which linguistic phenomena represent barriers to better... more This paper aims to automatically identify which linguistic phenomena represent barriers to better MT quality. We focus on the translation of news data for two bidirectional language pairs: EN↔ES and EN↔DE. Using the diagnostic MT evaluation toolkit DELiC4MT and a set of human reference translations, we relate translation quality barriers to a selection of 9 source-side PoS-based linguistic checkpoints. Using output from the winning SMT, RbMT, and hybrid systems of the WMT 2013 shared task, translation quality barriers are investigated (in relation to the selected linguistic checkpoints) according to two main variables: (i) the type of the MT approach, i.e. statistical, rule-based or hybrid, and (ii) the human evaluation of MT output, ranked into three quality groups corresponding to good, near miss and poor. We show that the combination of manual quality ranking and automatic diagnostic evaluation on a set of PoS-based linguistic checkpoints is able to identify the specific quality barriers of different MT system types across the four translation directions under consideration.

University of New South Wales Interpreting & Translation Seminar Series, Mar 25, 2014

Proceedings of Multimodality & Cyberpsychology II, Dec 31, 2013

Building upon the success of last year's inaugural conference, we are delighted to have been able... more Building upon the success of last year's inaugural conference, we are delighted to have been able to invite researchers and practitioners from all aspects of media, arts, and cyberpsychology to present their work, their experiences, and their enthusiasm for this new and exciting interdisciplinary field.

Gloablization & Localization Association, Dec 1, 2013

This short industry paper presents approaches to human evaluation, describes the advantages and s... more This short industry paper presents approaches to human evaluation, describes the advantages and shortcomings of each in relation to typical workflows, and highlights further materials and training resources.

This article reports on the results of a project that aimed to investigate the usability of raw m... more This article reports on the results of a project that aimed to investigate the usability of raw machine translated technical support documentation for a commercial online file storage service. Adopting a user-centered approach, the ISO/TR 16982 definition of usability-goal completion, satisfaction, effectiveness, and efficiency-is utilized and eye-tracking measures that are shown to be reliable indicators of cognitive effort are applied along with a posttask questionnaire. The study investigated these measures for the original user documentation written in English and in four target languages: Spanish, French, German, and Japanese, all of which were translated using a freely available online statistical machine translation engine. Using native speakers for each language, the study found several significant differences between the source and MT output, a finding that indicates a difference in usability between well-formed content and raw machine translated content. One target language in particular, Japanese, was found to have a considerably lower usability level when compared with the original English.

This study compares consistency in target texts produced using translation memory (TM) with that ... more This study compares consistency in target texts produced using translation memory (TM) with that of target texts produced using statistical machine translation (SMT), where the SMT engine is trained on the same texts as are reused in the TM workflow. These comparisons focus specifically on noun and verb inconsistencies, as such inconsistencies appear to be highly prevalent in TM data. The study substitutes inconsistent TM target text nouns and verbs for consistent nouns and verbs from the SMT output to test whether this results in improvements in overall TM consistency and whether an SMT engine trained on the ‘laundered’ TM data performs better than the baseline engine. Improvements were observed in both TM consistency and SMT performance, a finding that indicates the potential of this approach for improving TM/MT integration.

Machine Translation, Jan 1, 2010

Eye tracking has been used successfully as a technique for measuring cognitive load in reading, p... more Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc for some time now. Its application as a technique for automatically measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semiand/or automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. 10 native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded noninvasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the "bad" sentences, while average fixation duration and pupil dilations were not found to be substantially different between output rated as good or bad. Comparisons between BLEU scores and eye gaze data were also made and found to correlate well with gaze time and fixation count, and to a lesser extent with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as an automatic MT Evaluation technique.

Globalization and Localization Association, Jun 1, 2013

This study investigates the impact of same-language subtitles on the immersion into audiovisual n... more This study investigates the impact of same-language subtitles on the immersion into audiovisual narratives as a function of the viewer’s language (native or foreigner). Students from two universities in Australia and one in Spain were assigned randomly to one of two experimental groups, in which they saw a drama with the original English soundtrack either with same-language English subtitles (n=81) or without subtitles (n=92). The sample included an English native control group, and Mandarin Chinese, Korean, and Spanish groups with English as a foreign language. Participants used post-hoc Likert scales to self-report their presence, transportation to the narrative world, perceived realism, identification with the characters, and enjoyment. The main results showed that subtitles did not significantly reduce these measures of immersion. However, subtitles produced higher transportation, identification with the characters, and perceived realism scores, where the first language of viewers and their viewing habits accounted for most of this variance. Moreover, presence and enjoyment were unaffected by either condition or language. Finally, the main results also revealed that transportation to the narrative world appears to be the most revealing measure of immersion in that it shows the strongest and most consistent correlations, and is a significant predictor of enjoyment.

Japanese and other Asian TV producers have been deploying multi-colored, and highly visible, intr... more Japanese and other Asian TV producers have been deploying multi-colored, and highly visible, intra-lingual captions on TV programs to enhance their appeal and to influence their viewers’ interpretations. The practice of adding these captions is far from innocent and is prone to abuse and overuse due to the lack of official guidelines and an evidence base. We conducted a multimodal analysis within the framework of relevance theory to provide an empirically supported insight into the way in which these captions, known as “telop” in Japan, form part of a production’s deliberate and careful media design. Our findings suggest that telop are deployed in conjunction with other communicative resources that are deliberately used to influence viewers’ interpretations, to enhance and make affective values in TV programs more explicit. The increasing use of diegetically integrated captions elsewhere further justifies the need for critical TV and new media research on telop.

This chapter presents the findings of a study to investigate the impact of subtitles on the proce... more This chapter presents the findings of a study to investigate the impact of subtitles on the processing of audiovisual texts in terms of levels of self-reported engagement with the text. It also presents a methodology for investigating the neural processing of subtitles using electroencephalography (EEG) and psychometrics. By establishing the beta coherence between prefrontal and posterior regions, we can determine whether the effect of subtitles can be captured through EEG, and how this is related to psychological immersion. We therefore use EEG to investigate the cognitive processing of subtitled film in combination with subjective psychometrics. Initial validation of this methodology shows that adding same language subtitles results in statistically significantly higher levels of immersion and enjoyment, lending support to a view that subtitles facilitate (dis)embodied cognition. We propose these as the first steps towards a cognitive audiovisual translatology.

Technological advances have led to unprecedented changes in translation as a means of interlingua... more Technological advances have led to unprecedented changes in translation as a means of interlingual communication. This article discusses the impact of two major technological developments of contemporary translation: computer-assisted translation tools and machine translation. These technologies have increased productivity and quality in translation, supported international communication, and demonstrated the growing need for innovative technological solutions to the age-old problem of the language barrier. However, these tools also represent significant challenges and uncertainties for the translation profession and the industry. In highlighting the need for increased awareness and technological competencies, I propose that these challenges can be overcome and translation technologies will become even more integral in interlingual communication.

In many countries such as Japan, the use of intra-lingual Open Caption Telop (OCT) for multimodal... more In many countries such as Japan, the use of intra-lingual Open Caption Telop (OCT) for multimodal communications such as television and video is highly prevalent, especially in the context of entertainment and informative programmes. Unlike the conventional use of intra-lingual subtitles for the hard-of-hearing population, this particular adoption of OCT relates to the producers’ attempts at manipulating the viewers’ cognitive processing to achieve a directed and desired behaviour such as humour, enhanced comprehension and retention of content, or the reading of editorial opinions. In this paper, working within the frameworks of relevance theory and cognitive psychology, we examine the relations between linguistically represented stimulus and its effect on viewer experience, both during and after stimulus presentation. We demonstrate how OCT enables TV producers to achieve their intentions by hijacking the viewers’ interpretation processes by guiding viewers to a desired outcome. Us...

The following paper has been prepared by two independent researchers – Dr Stephen Doherty (UNSW) ... more The following paper has been prepared by two independent researchers – Dr Stephen Doherty (UNSW) and Dr Ignacio Garcia (UWS). It discusses options and alternatives for NAATI transitioning to Keyboarded Translator Assessment. NAATI commissioned this to support discussions with the broad group of stakeholders who engage with translator assessments – as the taker of tests, the deliverer of assessments or for other reasons. NAATI is seeking feedback and comment on the options put forward in the paper. After considering the information in the paper and any feedback received NAATI intends to publish a response including how it intends to move away from the current paper-based model for translator assessment. If you have feedback please email it to intproject@naati.com.au.

The Interpreter and Translator Trainer, 2014

In this paper we argue that the time is ripe for translator educators to engage with Statistical ... more In this paper we argue that the time is ripe for translator educators to engage with Statistical Machine Translation (SMT) in more profound ways than they have done to date. We explain the basic principles of SMT and reflect on the role of humans in SMT workflows. Against a background of diverging opinions on the latter, we argue for a holistic approach to the integration of SMT into translator training programmes, one that empowers rather than marginalises translators. We discuss potential barriers to the use of SMT by translators generally and in translator training in particular, and propose some solutions to problems thus identified. More specifically, cloud-based services are proposed as a means of overcoming some of the technical and ethical challenges posed by more advanced uses of SMT in the classroom. Ultimately the paper aims to pave the way for the design and implementation of a new translator-oriented SMT syllabus at our own University and elsewhere.

The Interpreter and Translator Trainer, 2014

Despite the acknowledged importance of translation technology in translation studies programmes a... more Despite the acknowledged importance of translation technology in translation studies programmes and the current ascendancy of Statistical Machine Translation (SMT), there has been little reflection to date on how SMT can or should be integrated into the translation studies curriculum. In a companion paper we set out a rationale for including a holistic SMT syllabus in the translation curriculum. In this paper, we show how the priorities and aspirations articulated in that source can be operationalised in the translation technology classroom and lab. We draw on our experience of designing and evaluating an SMT syllabus for a cohort of postgraduate student translators at Dublin City University in 2012. In particular, we report on data derived from a mixed-methods approach that aims to capture the students' view of the syllabus and their self-assessment of their own learning. Using the construct of self-efficacy, we show significant increases in students' knowledge of and confidence in using machine translation in general and SMT in particular, after completion of teaching units in SMT. We report on additional insights gleaned from student assignments, and conclude with ideas for future refinements of the syllabus.

Machine Translation, 2010

Eye tracking has been used successfully as a technique for measuring cognitive load in reading, p... more Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc. for some time now. Its application as a technique for measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semi-automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. Ten native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded non-invasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the “bad” sentences, while average fixation duration and pupil dilations were not found to be substantially different for output rated as good and output rated as bad. Comparisons between HTER scores and eye gaze data were also found to correlate well with gaze time and fixation count, but not with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as a semi-automatic MT evaluation technique, which does not require bi-lingual knowledge, and which can potentially tap into the end users’ experience of machine translation output.

The Journal of Specialised Translation, Jan 18, 2013

This article presents an on-going evaluation of translation technology lab sessions in a universi... more This article presents an on-going evaluation of translation technology lab sessions in a university setting. Lab sessions are practical supplements that allow student translators to develop upon what has already been learned in theory via traditional lectures. Both of these components develop the technical competencies required for professional work in translation, post-editing, and related areas. Data pertaining to the evaluation of the labs were collated and analysed using a mixed-methods approach, where the current paper focuses on the qualitative aspects. Upon examination of the data, several areas were identified as having the potential to be improved upon: attitudes, abilities, and resources. After making modifications to the labs, a follow-up study was carried out in the following academic year, the results of which were compared and contrasted with that of the first study to ascertain if the efforts to improve the labs were successful, and to guide further work.

National Accreditation Authority for Translators and Interpreters, May 25, 2015

The following paper has been prepared by two independent researchers – Dr Stephen Doherty (UNSW) ... more The following paper has been prepared by two independent researchers – Dr Stephen Doherty (UNSW) and Dr Ignacio Garcia (UWS). It discusses options and alternatives for NAATI transitioning to Keyboarded Translator Assessment.
NAATI commissioned this to support discussions with the broad group of stakeholders who engage with translator assessments – as the taker of tests, the deliverer of assessments or for other reasons.
NAATI is seeking feedback and comment on the options put forward in the paper.
After considering the information in the paper and any feedback received NAATI intends to publish a response including how it intends to move away from the current paper-based model for translator assessment.
If you have feedback please email it to intproject@naati.com.au.

The 9th Edition of the Language Resources and Evaluation Conference, At Reykjavik, Iceland, Jun 26, 2014

This paper aims to automatically identify which linguistic phenomena represent barriers to better... more This paper aims to automatically identify which linguistic phenomena represent barriers to better MT quality. We focus on the translation of news data for two bidirectional language pairs: EN↔ES and EN↔DE. Using the diagnostic MT evaluation toolkit DELiC4MT and a set of human reference translations, we relate translation quality barriers to a selection of 9 source-side PoS-based linguistic checkpoints. Using output from the winning SMT, RbMT, and hybrid systems of the WMT 2013 shared task, translation quality barriers are investigated (in relation to the selected linguistic checkpoints) according to two main variables: (i) the type of the MT approach, i.e. statistical, rule-based or hybrid, and (ii) the human evaluation of MT output, ranked into three quality groups corresponding to good, near miss and poor. We show that the combination of manual quality ranking and automatic diagnostic evaluation on a set of PoS-based linguistic checkpoints is able to identify the specific quality barriers of different MT system types across the four translation directions under consideration.

University of New South Wales Interpreting & Translation Seminar Series, Mar 25, 2014

Proceedings of Multimodality & Cyberpsychology II, Dec 31, 2013

Building upon the success of last year's inaugural conference, we are delighted to have been able... more Building upon the success of last year's inaugural conference, we are delighted to have been able to invite researchers and practitioners from all aspects of media, arts, and cyberpsychology to present their work, their experiences, and their enthusiasm for this new and exciting interdisciplinary field.

Gloablization & Localization Association, Dec 1, 2013

This short industry paper presents approaches to human evaluation, describes the advantages and s... more This short industry paper presents approaches to human evaluation, describes the advantages and shortcomings of each in relation to typical workflows, and highlights further materials and training resources.

This article reports on the results of a project that aimed to investigate the usability of raw m... more This article reports on the results of a project that aimed to investigate the usability of raw machine translated technical support documentation for a commercial online file storage service. Adopting a user-centered approach, the ISO/TR 16982 definition of usability-goal completion, satisfaction, effectiveness, and efficiency-is utilized and eye-tracking measures that are shown to be reliable indicators of cognitive effort are applied along with a posttask questionnaire. The study investigated these measures for the original user documentation written in English and in four target languages: Spanish, French, German, and Japanese, all of which were translated using a freely available online statistical machine translation engine. Using native speakers for each language, the study found several significant differences between the source and MT output, a finding that indicates a difference in usability between well-formed content and raw machine translated content. One target language in particular, Japanese, was found to have a considerably lower usability level when compared with the original English.

This study compares consistency in target texts produced using translation memory (TM) with that ... more This study compares consistency in target texts produced using translation memory (TM) with that of target texts produced using statistical machine translation (SMT), where the SMT engine is trained on the same texts as are reused in the TM workflow. These comparisons focus specifically on noun and verb inconsistencies, as such inconsistencies appear to be highly prevalent in TM data. The study substitutes inconsistent TM target text nouns and verbs for consistent nouns and verbs from the SMT output to test whether this results in improvements in overall TM consistency and whether an SMT engine trained on the ‘laundered’ TM data performs better than the baseline engine. Improvements were observed in both TM consistency and SMT performance, a finding that indicates the potential of this approach for improving TM/MT integration.

Machine Translation, Jan 1, 2010

Eye tracking has been used successfully as a technique for measuring cognitive load in reading, p... more Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc for some time now. Its application as a technique for automatically measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semiand/or automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. 10 native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded noninvasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the "bad" sentences, while average fixation duration and pupil dilations were not found to be substantially different between output rated as good or bad. Comparisons between BLEU scores and eye gaze data were also made and found to correlate well with gaze time and fixation count, and to a lesser extent with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as an automatic MT Evaluation technique.

Globalization and Localization Association, Jun 1, 2013

Se estudia el impacto de los subtítulos en el mismo idioma de la narrativa audiovisual según el i... more Se estudia el impacto de los subtítulos en el mismo idioma de la narrativa audiovisual según el idioma del receptor (nativo o extranjero). Estudiantes de dos universidades australianas y una española fueron asignados al azar a uno de dos grupos experimentales en los que se veía un drama con la banda sonora original en inglés con subtítulos en esa misma lengua (n=81) o sin subtítulos (n=92). La muestra incluía un grupo control de hablantes nativos de inglés, además de grupos de hablantes nativos de chino mandarín, coreano y español con inglés como lengua extranjera. Como medidas post-hoc, los participantes reportaron, mediante escalas Likert, su percepción de presencia, transporte, realismo percibido, identificación con los personajes y disfrute. Los resultados muestran que los subtítulos no reducen las medidas de inmersión. Además, que los subtítulos producen mayores puntuaciones de transporte, identificación con los personajes y percepción de realismo, cuya varianza se explica, esencialmente, por la primera lengua de los receptores y sus hábitos de visionado. Asimismo, los resultados señalan que ni a la presencia y ni al disfrute les afectan la condición experimental o el idioma del receptor. Finalmente, muestran que el transporte es la medida más reveladora de la inmersión porque produce las correlaciones más fuertes y consistentes, aparte de ser un predictor significativo del disfrute de los espectadores.