Reviewing for bias and sensitivity: Beyond checklists 

Abstract:

Developing language test items is a huge responsibility and requires adherence to ethical principles and practices. While checklists for item development and review are necessary for the overall process, additional processes are essential to review for bias and sensitivity. This half-day workshop will begin with an overview of the ILTA Code of Ethics and Guidelines for practice. Next, participants will engage with each other to wrestle with realistic scenarios for item writing, review, and production. Based on the discussions from this small group work, the workshop coordinators will share some principles, practices and processes, and practicalities used to help item developers select, review and revise test items, and produce tests. Participants will develop both concrete and abstract approaches to rigorously review items and test forms. 

Bios:

Margaret E. Malone (Ph.D., Georgetown University) is Director of Assessment and Research at ACTFL. She has over three decades of experience in language test development, materials development, delivery of professional development, and teacher preparation through both online and face-to-face methods, data collection and survey research, and program evaluation. Her current research focuses on language assessment literacy, oral proficiency assessment and the influences of the Seal of Biliteracy on language teaching and learning. Previously, she was Associate Vice President of World Languages and International Programs at the Center for Applied Linguistics. Earlier in her career, she served as the first Language Testing Specialist for Peace Corps-Worldwide, where she managed oral language testing for 60 countries and 120+ languages.

Camelot Marshall, PMP, (Ph.D., Bryn Mawr College) is Principal Assessment Manager at ACTFL. She has over 30 years’ experience teaching Russian and graduate level second language acquisition courses, as well as designing, developing, and implementing test design and item writing processes. At present, she manages ACTFL’s test development program and employs innovative methods and approaches to develop world language proficiency and performance assessments. Prior to joining ACTFL, she was the assessment manager at American Councils for International Education, where she was instrumental in developing and implementing test development processes for ten critical languages.  

Applications of Gaussian Mixture Model in Language Testing and Speech Research

Abstract:

This is a world filled with Gaussian/Normal distributions. Quantitative data harvested in many practical domains, such as demography, marketing, as well as language testing, can be modelled by Gaussian distribution. Even a real-valued random variable whose actual distribution is unknown can also be assumed to have a Gaussian distribution. It is, therefore, reasonable to suppose that samples sharing a set of ‘similarities’ and those sharing another set of ‘similarities’ may constitute distinct groups generated from different Gaussians in either a univariate or multivariate space. With such a fundamental idea, Gaussian Mixture Model (GMM) as a machine learning algorithm models the internal groupings (e.g. clusters) as a mixture of several Gaussian Distributions; each of the clusters is defined by its own mean(s) and (co)variances. As such, GMM can be a useful methodological approach to identifying test-taker and performance profiles in language testing research. This workshop gives a broad introduction to the principle of GMM and how GMM can be used to perform unsupervised clustering and semi-supervised learning in language testing research. The workshop consists of two sessions. The first session will be a lecture-based meeting. During the second session, there will be a tutorial with several hands-on exercises on building GMM-based clustering models using Python for analysing datasets of speech and language testing. Prior Python skills are not essential; the base code for the exercises will be provided during the session.

Bios:

Yan Tang is an Assistant Professor in Speech Processing at the Department of Linguistics, University of Illinois Urbana-Champaign since 2019. Before that, he was a research fellow and senior research fellow for five years, working on a UK EPSRC project funded by the UK government and the BBC. Yan Tang completed his Ph.D. in Applied Linguistics with Sobresaliente Cum Laude at Universidad del País Vasco, Spain in 2014, focusing on computational speech and hearing. He also received a master’s degree in Software Systems and Internet Technology with Distinction at the University of Sheffield, UK in 2008. His research lies in a wide range on the spectrum of speech and hearing science from speech intelligibility enhancement, computational modelling of the auditory system, blind source separation to human speech production and perception in non-ideal listening conditions. His current research interests include computational modelling of informational masking in speech perception, speech-noise interactions in the spectro-temporal-modulation domain and their impact on speech understanding, and paralinguistic speech processing using machine learning.

Huiying Cai is a fourth-year Linguistics Ph.D. student at the University of Illinois at Urbana-Champaign (UIUC). She is a research assistant for the UIUC English Placement Test, assisting in test development and quality control. Her research interests mainly lie in language testing, with a particular focus on the interplay of language testing and computational linguistics. Her studies used natural language processing techniques on test-relevant corpora (e.g., rater comments and student essays) to provide empirical evidence for test validity arguments. Currently, her research intends to explore an automated scoring system for integrated writing tasks in a local context using machine learning techniques.