Session 3E

Previous studies have suggested that watching foreign language videos facilitates second language learners’ incidental vocabulary learning (e.g. Peters & Webb, 2018), and that its learning potential can be enhanced with the help of captions and subtitles (e.g. Montero Perez, Van Den Noortgate, & Desmet, 2013). Among the different types of subtitles, dual subtitles (i.e. simultaneous presentation of captions and subtitles at the bottom of the screen) are widely used in China. It has been suggested that dual subtitles could be more beneficial for viewers’ incidental vocabulary gains than other subtitle conditions (Li, 2016). However, studies have yielded conflicting results and its effectiveness for vocabulary learning is still controversial. One possible reason for these conflicting findings could be the way learners make use of the different sources of information in dual subtitles. However, little is known about how learners process this type of subtitling condition. Besides, it could be argued that the presence of the different input sources might also cause cognitive overload which may hinder the learning process (Lwo & Chia-Tzu Lin, 2012). Empirical evidence about the processing of dual subtitles, in comparison to other types of subtitles, is yet to be provided. The present study addresses these gaps by using a combination of online and offline measures, i.e. eye-movement recordings, pre- and post- tests. The main objectives of the present investigation are twofold: 1) to investigate the effectiveness of dual subtitles for incidental vocabulary learning and comprehension, compared to captions, subtitles, and no subtitles; 2) and to examine the processing of dual subtitles, as measured by readers’ eye movements. In this study, one-hundred-and-twelve Chinese speakers of English studying at a British university were randomly assigned to one of four conditions (captions, subtitles, dual subtitles, and no subtitles) to watch a 23-minute English documentary clip while their eye movements were recorded. Participants’ knowledge of a set of 24 target words appearing in the video was assessed by means of three vocabulary tests (form recognition, meaning recall, and meaning recognition) conducted before and after the viewing session. A multiple-choice comprehension test was also conducted after the viewing session. The percentage of total reading time to the subtitling area and fixation duration to the target words were examined across conditions. Results of a one-way ANOVA suggested that dual subtitles were more beneficial than captions in terms of meaning recognition and comprehension, but less effective for form recognition. Comprehension and vocabulary gains from dual subtitles were not significantly different from the subtitles condition. Regarding the analysis of eye-movements, results of one-way ANOVA revealed that the overall time spent on the subtitling area was similar in the dual subtitles and captions groups, which were both longer than the subtitles group. In terms of the reading of the target words, the eye-movement patterns supported the offline results. Participants in the dual condition spent longer time on Chinese translations of the target words than participants using subtitles only, but less time on the English words than the captions only group. Pedagogical implications of these findings will be discussed. References: Li, M. (2016). Investigation into the differential effects of subtitles (first language, second language, and bilingual) on second language vocabulary acquisition. (Unpublished doctoral dissertation). The University of Edinburgh, Edinburgh. Lwo, L., & Chia-Tzu Lin, M. (2012). The effects of captions in teenagers’ multimedia L2 learning. ReCALL, 24(02), 188-208. doi:10.1017/s0958344012000067 Montero Perez, M., Van Den Noortgate, W., & Desmet, P. (2013). Captioned video for L2 listening and vocabulary learning: A meta-analysis. System, 41(3), 720-739. Peters, E., & Webb, S. (2018). Incidental Vocabulary Acquisition through Viewing L2 Television and Factors That Affect Learning. Studies in Second Language Acquisition, 1-27.

Original version (OV) TV series supported by captions have been found to be a fruitful source of input for L2 vocabulary learning and content comprehension (Vanderplank, 2016). Up till now, very little is known about possible learning of L2 grammar constructions from extensive viewing of OV TV series, or the learner-related factors that may affect it. Previous studies on grammar learning from audio-visual input have mainly focused on the possible advantages of different captioning modes (standard captions and enhanced captions in Lee & Revesz, 2018; no captions and enhanced captions in Cintrón-Valentín, Garcia-Amaya, & Ellis, 2019). The researchers concluded that the enhanced condition was the most beneficial; however, the aforementioned studies did not compare all three captioning modes in the same experiment and had a short exposure to the audio-visual materials. This study extends this area of research and looks at extensive exposure to OV TV series with three captioning modes. Additionally, previous research suggests a mediating effect of language proficiency and WM capacity on language gains from audio-visual input (Vanderplank, 2016). Yet, no previous studies on L2 grammar learning from audio-visual input have taken these learner-related factors into account. In order to fill this gap, data were collected from 108 university non-Linguistics learners of English. The pre-/post-test design included extensive watching of 10 full-length episodes (218 minutes) of English OV TV series with three groups: Standard Captions, Textually Enhanced Captions, or Non-Captions. The participants also completed proficiency, WM capacity, and language aptitude tests. Preliminary findings suggest a significant role of learner related factors, and only a minor advantage of the textually enhanced captions condition for grammar learning from this type of input. In our presentation, we will discuss the significance of these results and their applicability to the language learning process both in and out of the classroom. Cintrón-Valentín, M., Garcia-Amaya, L., & Ellis, N. (2019). Captioning and grammar learning in the L2 Spanish classroom. The Language Learning Journal, 47(4), 1-21. Lee, M. & A. Revesz. (2018). Promoting grammatical development through textually enhanced captions: an eye-tracking study. The Modern Language Journal, 102(3), 557–77. Vanderplank, R. (2016). Captioned media in foreign language learning and teaching: Subtitles for the deaf and hard-of -hearing as tools for language learning. Oxford: Palgrave Macmillan.

Research on incidental vocabulary learning through reading, listening and, more recently, viewing has provided robust evidence that repeated encounters with unknown words in the input facilitate learning, although the number of occurrences needed for substantial learning remains unclear (Uchihara, Webb & Yanagisawa, 2019). A number of studies also support the idea that the imagery associated with videos can assist information processing, and that words occurring in close proximity to its visual representation are better learnt that the ones without image support (Peters, 2019; Rodgers, 2018). These recent studies in the area of vocabulary learning through audio-visual input suggest, then, that less encounters might be needed in this context compared to incidental reading or listening. The aim of this study is to explore the role frequency of occurrence and the role of imagery in the learning of unknown word forms and word meanings through the viewing of consecutive episodes of a TV series, and to investigate which variable better predicts word learning. Participants were 45 secondary school students (Grade 8) who took part in an eight-month classroom intervention, in which they watched 24 episodes of a TV series. Participants were distributed into two conditions according to the language of the on-screen text (L1 vs. L2). Prior knowledge of the target items selected from the episodes was tested by means of an aural recognition and recall test. Preliminary analysis revealed that both the presence of imagery and the frequency of encounters with unknown target words were positively correlated with learning, with a slightly stronger effect in meaning recall, independently of the language of the on-screen text and the learners’ proficiency level. Imagery, however, emerged as a better predictor than frequency, which would provide further evidence that fewer encounters are needed when words are presented together with its visual representation. Peters, E. (2019), The Effect of Imagery and On?Screen Text on Foreign Language Vocabulary Learning From Audiovisual Input. TESOL Q, 53, 1008-1032. Rodgers, M. (2018). The images in television programs and the potential for learning unknown words. Approaches to Learning, Testing, and Researching L2 Vocabulary ITL - International Journal of Applied Linguistics 169(1), 191-211. doi:10.1075/itl.00012.rod. Uchihara, T., Webb, S., & Yanagisawa, A. (2019). The Effects of Repetition on Incidental Vocabulary Learning: A Meta-Analysis of Correlational Studies. Language Learning, 69(3), 559-99.

One theoretical question that transcends much of second-language (L2) literature concerns the role of attention, namely whether the low perceptual salience of certain input features yields challenges for L2 acquisition (Ellis, 2017). Empirically, L2 researchers consider how to enhance learner attention to commonly unattended input features through form-focused instruction (FFI) (see Norris & Ortega, 2000). With the increased availability of multimedia language-learning materials, FFI research can more deeply scrutinize the role of multimodal input (i.e., aural, written, and visual) in facilitating L2 development. One promising multimodal technique is captioned video. The effects of captioned media on L2 vocabulary learning are well-studied (e.g., Muñoz, 2017), and researchers are now considering the role of captioned video on L2 grammar learning (e.g., Cintrón-Valentín et al., 2019). However, much work remains before we can fully understand the benefits of captioning on grammar learning. For instance, it is not clear whether captioned media is reliably effective for all grammar structures (e.g., Lee & Révész, 2018), or whether any positive effects of captioning are sustained over time (e.g., Cintrón-Valentín et al., 2019). We developed a randomized control design to investigate the role of captioned video in grammar learning. The study was integrated into a one-semester university L2-Spanish grammar course (n=409 students). Through six data-collection sessions, we targeted the preterite/imperfect, ser/estar, gustar-type verbs, the subjunctive, and the conditional. On the first day of class, participants were given a pretest on all five structures. The following five sessions focused on each of the five target structures, respectively. In each experimental session, some groups first saw a grammar lesson on the target structure, whereas others did not. We included this Lesson/NoLesson contrast to investigate the effectiveness of captions and textual enhancement in the absence of explicit instruction. Second, participants saw an animated video that varied in its captioning format: no captions (“NoCaptions_NoLesson” or “NoCaptions_Lesson”); captions without enhancement (“Captions_NoLesson” or “Captions_Lesson”); or captions with textual enhancement via bolding on the target grammar form (“TECaptions_NoLesson” or “TECaptions_Lesson”). Participants were tested on their production of the target grammar through a translation task via an immediate posttest and two delayed posttests. Our findings reveal the following effects for grammar production: 1) positive effects of explicit instruction in the absence of captioning; 2) positive effects of captioning and textual enhancement in the absence of explicit instruction; 3) more robust effects of captioning and textual enhancement at immediate posttest than at delayed posttests; and 4) a stronger effect of captioning on the conditional than on the other target structures. The results for captioning confirm that multimodal pedagogical interventions can lead to a significant improvement in learners’ production, even in the absence of explicit grammar instruction. However, we also show that captioning effects exist for some, but not all, target structures. This outcome creates space for future investigation into the factors that mediate the effectiveness of multimodal FFI, such as prior knowledge or frequency of use. References: Cintro?n-Valenti?n, M.C., Garci?a-Amaya, L, Ellis, N.C. (2019). Captioning and grammar learning in the L2 Spanish classroom. The Language Learning Journal, 47(4), 439-459. Ellis, N. C. (2017). Salience in usage-based SLA. In S. Gass, P. Spinner, & J. Behney (Eds.), Salience in SLA (pp. 21-39). New York: Routledge. Lee, M., & Révész, A. (2018). Promoting grammatical development through textually enhanced captions. MLJ, 102(3), 557-577. Muñoz, C. (2017). The role of age and proficiency in subtitle reading. An eye-tracking study. System, 67, 77-86. Norris, J. & Ortega, L. (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta?analysis. Language Learning, 50(3), 417-528.