the algorithm marks down how you talk

Problem statement

Automated essay scoring (AES) systems — both legacy NLP-based engines and newer LLM-based scorers — produce scores that diverge from human ratings in group-dependent ways tied to dialect and first-language background. ETS's own evaluation of the e-rater engine on the GRE General Test reported standardized mean machine-minus-human differences of about −0.13 for the African American subgroup and roughly −0.19 to −0.22 for test takers from Taiwan on argument prompts, while test takers from China were overscored (+0.68 on issue prompts, +0.56 on argument). That the gap changes sign and size by group is what makes this hard: it is not one uniform penalty that can be subtracted out. ETS traced the mechanism to a modeling mismatch — human raters treated control of language as a threshold condition, whereas e-rater applied linear weighting across all features, insufficiently penalizing language errors while over-weighting organization and development features that correlate with essay length. The bias is structural, in the sense that it follows from how scoring models are fit, not from noise.

Why this matters

AES is deployed at high stakes and large scale: e-rater has been used operationally on the GRE General Test and the TOEFL test, and automated scoring is now common in university placement exams and K-12 writing assessment. When machine-human divergence is group-dependent, students from multilingual backgrounds, speakers of African American Vernacular English (AAVE), and L2 English writers can have their scores shifted in a direction that has nothing to do with the quality of their argument — affecting admissions, course placement, and academic self-concept. As LLM-based grading expands, deployment is outrunning fairness validation: Loukina, Madnani, and Zechner (2019) show, using simulated and real assessment data, that test-takers' native-language background can affect automated scores and that competing definitions of fairness cannot all be satisfied at once, so "is it fair?" has no single answer a vendor can certify.

What’s been tried and why it hasn’t worked

Monitoring machine-human score differences after deployment — the approach ETS took — detects the gap but does not close it: the GRE analysis located the cause in e-rater's linear feature weighting versus human raters' threshold treatment of language control, a difference that post-hoc rescaling cannot remove. Making training data more representative helps but is not sufficient on its own: Schaller et al. (2024), comparing shallow-learning, deep-learning, and LLM scorers on German secondary-school learner essays, found that training on skewed subsets did not itself produce biased scores but did produce low accuracy for out-of-distribution students, and concluded that training data are needed for all relevant user groups — including cognitive-ability groups, not only demographic ones. Assuming the problem is a dialect-feature penalty may also misdiagnose it: in an automated-scoring study of open-ended math responses, Siedahmed et al. (2025) varied four AAVE features against synthetically standardized controls and found that the number of dialect features per response did not change GPT scoring, while prompt-engineering choices did — i.e. the scoring harness mattered more than the dialect markers. And LLM scorers are not yet stable enough to be the fix: Pack, Barrett, and Escalante (2024), double-scoring 119 language-placement essays with PaLM 2, Claude 2, GPT-3.5, and GPT-4 on two occasions, found GPT-4 strongest on intrarater reliability and validity but interrater reliability against humans declining slightly over time across models.

What would unlock progress

Large-scale, multi-dialect, content-quality-matched essay corpora with paired human ratings that explicitly separate language form from content quality. Scoring architectures that decouple "content knowledge and argumentation quality" from "surface-level linguistic conformity to standard English." Fairness auditing frameworks specific to writing assessment defining acceptable differential performance across dialect/L1 groups, analogous to differential item functioning (DIF) analysis in psychometrics.

Entry points for student teams

A team could conduct a fairness audit of one commercially available AES system using a controlled essay set written in standard English vs. AAVE or L2 English with matched content quality, quantifying the scoring differential. Alternatively, a team could build a prototype scoring model that evaluates argumentation quality using semantic features while explicitly ignoring surface-level dialect markers. NLP, education measurement, and equity analysis skills apply.