Published papers

Nonverbal information has been estimated to comprise roughly 66% of the information in any human interaction. In almost all communicative interactions, sighted listeners are able to augment, contextualize, and sometimes decipher information from the verbal channel by referring to the various facial expressions, gestures, and other body movements produced by speakers. Taxonomies of nonverbal communicative behavior are numerous in the literature, but this chapter will refer to the seminal Ekman and Friesen (1969) model, which categorizes nonverbal behavior into emblems, illustrators, affect displays, regulators, and adaptors. These nonverbal cues do not just serve listeners when operating in their L1s, however. In fact, research into the impact of the presence of nonverbal information in L2 listening has almost universally found that L2 listeners’ comprehension and memory of L2 speech are clearly facilitated by its inclusion. In addition to nonverbal communicative cues, other forms of visual information, such as setting and visual aids, can have a large impact on L2 comprehension and acquisition. The chapter begins with an overview of the functions of nonverbal communicative cues and other visual information in human interaction, with explicit attention to facial expressions, gestures, and visual aids. It then reviews the studies of their impact on L2 comprehension, learning, and assessment and makes recommendations for pedagogical practice and assessment, and proposes a research agenda to fill the remaining gaps in the literature.

The present article employs eye tracking methodology to investigate the influence of item type (explicit and implicit) on test-taker attention to visual cues in video L2 listening tests. The findings reveal that implicit items direct test takers’ attention toward nonverbal communicative cues related to affect more than explicit items, and that explicit items may direct more attention to illustrative gestures and objects. These findings raise questions about the construct addressed by implicit questions in multimodal video listening tests and call for careful consideration of test purpose when designing L2 listening tests.

Recent years have seen increasing calls to test L2 vocabulary knowledge under a meaning recall modality (Stoeckel et al., 2021), where learners write the meaning of words in their L1, as such tests may have a closer relation to vocabulary knowledge as it relates to reading proficiency than meaning recognition multiple-choice tests (Kremmel & Schmitt, 2016). However, a drawback of meaning recall tests is that they are difficult for teachers to score, particularly if they are not fluent in learner L1s. Online tests scored by AI may mitigate this challenge because large language models (LLMs) can represent multiple languages in a single, nuanced, and context-dependent space. In this study, a 150-item meaning recall test was taken by 611 Japanese university students and scored by two native Japanese researchers and three LLMs (Llama 3B-8, GPT4o, and Gemini 1.5) to determine whether correlations, internal reliability, and mean scores between humans and LLMs were comparable. This was found to be the case for GPT4o and Gemini 1.5. In addition, a many-facet Rasch measurement was used to isolate cases where LLMs rated responses differently than humans. The findings are discussed, and recommendations are given for improving AI scoring.

© The Author(s) 2020. Vocabulary’s relationship to reading proficiency is frequently cited as a justification for the assessment of L2 written receptive vocabulary knowledge. However, to date, there has been relatively little research regarding which modalities of vocabulary knowledge have the strongest correlations to reading proficiency, and observed differences have often been statistically non-significant. The present research employs a bootstrapping approach to reach a clearer understanding of relationships between various modalities of vocabulary knowledge to reading proficiency. Test-takers (N = 103) answered 1000 vocabulary test items spanning the third 1000 most frequent English words in the New General Service List corpus (Browne, Culligan, & Phillips, 2013). Items were answered under four modalities: Yes/No checklists, form recall, meaning recall, and meaning recognition. These pools of test items were then sampled with replacement to create 1000 simulated tests ranging in length from five to 200 items and the results were correlated to the Test of English for International Communication (TOEIC®) Reading scores. For all examined test lengths, meaning-recall vocabulary tests had the highest average correlations to reading proficiency, followed by form-recall vocabulary tests. The results indicated that tests of vocabulary recall are stronger predictors of reading proficiency than tests of vocabulary recognition, despite the theoretically closer relationship of vocabulary recognition to reading.

Nonverbal and other visual cues are well established as a critical component of human communication. Under most circumstances, visual information is available to aid in the comprehension and interpretation of spoken language. Citing these facts, many L2 assessment researchers have studied video-mediated listening tests through score comparisons with audio tests, by measuring the amount of time spent watching, and by attempting to determine examinee viewing behavior through self-reports. However, the specific visual cues to which examinees attend have heretofore not been measured objectively. The present research employs eye-tracking methodology to determine the amounts of time 12 participants viewed specific visual cues on a six-item, video-mediated L2 listening test. Seventy-two scanpath-overlaid videos of viewing behavior were manually coded for visual cues at 0.10-second intervals. Cued retrospective interviews based on eye-tracking data provided reasons for the observed behaviors. Faces were found to occupy the majority (81.74%) of visual dwell time, with participants largely splitting their time between the speaker’s eyes and mouth. Detected gesture viewing was negligible. The reason given for most viewing behavior was determining characters’ emotional states. These findings suggest that the primary difference between audio- and video-mediated L2 listening tests of conversational content is the absence or presence of facial expressions.

Sign languages present particular challenges to language assessors in relation to variation in signs, weakly defined citation forms, and a general lack of standard-setting work even in long-established measures of productive sign proficiency. The present article addresses and explores these issues via a mixed-methods study of a human-rated form-recall sign vocabulary test of 98 signs for beginning adult learners of Swiss German Sign Language (DSGS), using post-test qualitative rater interviews to inform interpretation of the results of quantitative analysis of the test ratings using many-facets Rasch measurement. Significant differences between two expert raters were observed on three signs. The follow-up interview revealed disagreement on the criterion of correctness, despite the raters’ involvement in the development of the base lexicon of signs. The findings highlight the challenges of using human ratings to assess the production not only of sign language vocabulary, but of minority languages generally, and underscore the need for greater effort expended on the standardization of sign language assessment.

© 2018 The Authors In response to changing stakeholder needs, large-scale language test providers have increasingly considered the feasibility of delivering paper-based examinations online. Evidence is required, however, to determine whether online delivery of writing tests results in changes to writing performance reflected in differential test scores across delivery modes, and whether test-takers hold favourable perceptions of online delivery. The current study aimed to determine the effect of delivery mode on the two writing tasks (reading-into-writing and extended writing) within the Trinity College London Integrated Skills in English (ISE) test suite across three proficiency levels (CEFR B1-C1). 283 test-takers (107 at ISE I/B1, 109 at ISE II/B2, and 67 at ISE III/C1) completed both writing tasks in paper-based and online mode. Test-takers also completed a questionnaire to gauge perceptions of the impact, usability and fairness of the delivery modes. Many-facet Rasch measurement (MFRM) analysis of scores revealed that delivery mode had no discernible effect, apart from the reading-into-writing task at ISE I, where the paper-based mode was slightly easier. Test-takers generally held more positive perceptions of the online delivery mode, although technical problems were reported. Findings are discussed with reference to the need for further research into interactions between delivery mode, task and level.

Background Verbs of utterance are some of the most fundamental verbs in the English language, yet their usage patterns are exceptionally diverse. Learners of English should be able to use these words correctly and comfortably, but without an understanding of their core meanings, acquisition of their various patterns of use can be daunting. Aims The present research investigates the differences between English learners’ and native speakers’ intuitions regarding the utterance verbs “speak,” “talk,” “say,” and “tell.” Sample The participants were eighty users of English in 4 proficiency groups (low, mid, high, and native). Method The participants were polled via questionnaire on their intuitions regarding various uses of the four utterance verbs. Data were analyzed and compared with descriptive statistics and t tests. Results Although the intuitions of learners of increasing proficiency increasingly resembled those of native speakers, the verbs “speak” and “talk” posed special problems, indicating a lack of understanding of these verbs’ core meanings. Conclusions Language educators are recommended to pay particular attention to these verbs’ more idiomatic uses (e.g., “talk politics”) to address these deficiencies.

© The Author(s) 2020. In this study we seek evidence of validity according to the socio-cognitive framework (Weir, 2005) for a new sentence repetition test (SRT) for young Deaf L1 Swiss German Sign Language (DSGS) users. SRTs have been developed for various purposes for both spoken and sign languages to assess language development in children. In order to address the need for tests to assess the grammatical development of Deaf L1 DSGS users in a school context, we developed an SRT. The test targets young learners aged 6–17 years, and we administered it to 46 Deaf students aged 6.92–17.33 (M = 11.17) years. In addition to the young learner data, we collected data from Deaf adults (N = 14) and from a sub-sample of the children (n = 19) who also took a test of DSGS narrative comprehension, serving as a criterion measure. We analyzed the data with many-facet Rasch modeling, regression analysis, and analysis of covariance. The results show evidence of scoring, criterion, and context validity, suggesting the suitability of the SRT for the intended purpose, and will inform the revision of the test for future use as an instrument to assess the sign language development of Deaf children.

Second language vocabulary acquisition has been modeled both as multidimensional in nature and as a continuum wherein the learner's knowledge of a word develops along a cline from recognition through production. In order to empirically examine and compare these models, the authors assess the degree to which the Vocabulary Knowledge Scale (VKS; Paribakht & Wesche, 1993), which implicitly assumes a cline model of acquisition, conforms to a linear trait model under the Rasch Partial Credit Model, and determine the dimensionality of the individual tasks contained on the scale (self-report, first language [L1] equivalent, and sentence) using DETECT. The authors find that, although the VKS functions adequately overall as a measurement model, Stages 3 (can give an adequate L1 equivalent) and 4 (can use with semantic appropriateness) are psychometrically indistinct, suggesting they should be collapsed into a single category of definitional knowledge. Analysis under DIMTEST and DETECT indicates that other forms of vocabulary knowledge measured by the VKS are weakly multidimensional, which has implications for continuum models of vocabulary acquisition.

The rise in the affordability of quality video production equipment has resulted in increased interest in video-mediated tests of foreign language listening comprehension. Although research on such tests has continued fairly steadily since the early 1980s, studies have relied on analyses of raw scores, despite the growing prevalence of item response theory in the field of language testing as a whole. The present study addresses this gap by comparing data from identical, counter-balanced multiple-choice listening test forms employing three text types (monologue, conversation, and lecture) administered to 164 university students of English in Japan. Data were analyzed via many-facet Rasch modeling to compare the difficulties of the audio and video formats; to investigate interactions between format and text-type, and format and proficiency level; and to identify specific items biased toward one or the other format. Finally, items displaying such differences were subjected to differential distractor functioning analyses. No interactions between format and text-type, or format and proficiency level, were observed. Four items were discovered displaying format-based differences in difficulty, two of which were found to correspond to possible acting anomalies in the videos. The author argues for further work focusing on item-level interactions with test format.

The present thesis employed a mixed-methods research design spanning over two studies and an intermediate instrument development step to investigate the interactions between the presence of nonverbal and other visual cues with examinees, individual items, and item task types in video listening tests. The first study employed eye-tracking methodology to determine the specific visual cues to which examinees attend when interacting with a video-mediated listening test. The findings of Study I then informed the development of a new video-mediated listening test for Study II, which investigated the effect of the presence of visual cues on item and item task difficulty via many-facet Rasch modeling and qualitative item analysis. Individual examinee differences were also explored in both studies with respect to gender, proficiency, and perceptions of the two formats. Study I found that examinees view facial cues an average of 81% of the time regardless of task type, but spend significantly more time oriented toward the listener (i.e., the character who is not speaking) when presented with an implicit item. Direct viewing of gestures, despite their prominent place in the nonverbal communication literature, only accounted for approximately 1.35% of the total video time. Study II found that the presence of visual information exerted a facilitative effect on all items, but that it was significantly more pronounced with implicit items, despite the fact that they were more difficult than the explicit items in the audio format. Finally, no substantive interactions between proficiency, gender, or perception of the formats were observed between either viewing behavior or the facilitative effect of video. The thesis ultimately raises questions about the usefulness of video listening tests for listening comprehension assessment, as the effect appears to do little more that raise scores, and, in the case of implicit items, may obviate the need to comprehend the linguistic input in order to make correct inferences.