education · digital · family: the model was never trained on this
the algorithm marks down how you talk
Automated essay scoring diverges by dialect and language background in high-stakes tests
Problem statement
Automated essay scoring (AES) systems — both legacy NLP-based engines and newer LLM-based scorers — produce scores that diverge from human ratings in group-dependent ways tied to dialect and first-language background. ETS's own evaluation of the e-rater engine on the GRE General Test reported standardized mean machine-minus-human differences of about −0.13 for the African American subgroup and roughly −0.19 to −0.22 for test takers from Taiwan on argument prompts, while test takers from China were overscored (+0.68 on issue prompts, +0.56 on argument). That the gap changes sign and size by group is what makes this hard: it is not one uniform penalty that can be subtracted out. ETS's expert review pointed to a modeling mismatch — human raters treated control of language as a threshold condition, whereas e-rater applied linear weighting across all features, insufficiently penalizing language errors while over-weighting organization and development features that correlate with essay length. The bias is structural, in the sense that it follows from how scoring models are fit, not from noise.
Why this matters
AES is deployed at high stakes and large scale: e-rater has been used operationally on the GRE General Test and the TOEFL test, and Schaller et al. (2024) describe AES as widely used in education in many countries. When machine-human divergence is group-dependent, students from multilingual backgrounds, speakers of African American Vernacular English (AAVE), and L2 English writers can have their scores shifted in a direction that has nothing to do with the quality of their argument — affecting admissions, course placement, and academic self-concept. As LLM-based grading expands, fairness validation lags behind it: Loukina, Madnani, and Zechner (2019) show, using simulated and real assessment data, that test-takers' native-language background can affect automated scores and that competing definitions of fairness cannot all be satisfied at once, so "is it fair?" has no single answer a vendor can certify.
What’s been tried and why it hasn’t worked
Monitoring machine-human score differences after deployment — the approach ETS took — detects the gap but does not close it: the GRE analysis pointed to e-rater's linear feature weighting versus human raters' threshold treatment of language control, and the alternative model forms ETS tried (logistic, unit-weighted, and classification-and-regression-tree models) narrowed some subgroup gaps and widened others without improving overall scoring performance. Making training data more representative helps but is not sufficient on its own: Schaller et al. (2024), comparing shallow-learning, deep-learning, and LLM scorers on German secondary-school learner essays, found that training on skewed subsets did not itself produce biased scores but did produce low accuracy for out-of-distribution students, and concluded that training data are needed for all relevant user groups — including cognitive-ability groups, not only demographic ones. Assuming the problem is a dialect-feature penalty may also misdiagnose it: in an automated-scoring study of open-ended math responses, Siedahmed et al. (2025) varied four AAVE features against synthetically standardized controls and found that the number of dialect features per response did not change GPT scoring, while prompt-engineering choices did — i.e. the scoring harness mattered more than the dialect markers. And LLM scorers are not yet stable enough to be the fix: Pack, Barrett, and Escalante (2024), double-scoring 119 language-placement essays with PaLM 2, Claude 2, GPT-3.5, and GPT-4 on two occasions, found GPT-4 strongest on intrarater reliability and validity but the interrater reliability of GPT-3.5 and GPT-4 decreasing slightly over time.
What would unlock progress
Large-scale, multi-dialect, content-quality-matched essay corpora with paired human ratings that explicitly separate language form from content quality. Scoring architectures that decouple content knowledge and argumentation quality from surface-level linguistic conformity to standard English. Fairness auditing frameworks specific to writing assessment defining acceptable differential performance across dialect/L1 groups, analogous to differential item functioning (DIF) analysis in psychometrics.
Entry points for student teams
A team could conduct a fairness audit of one commercially available AES system using a controlled essay set written in standard English vs. AAVE or L2 English with matched content quality, quantifying the scoring differential. Alternatively, a team could build a prototype scoring model that evaluates argumentation quality using semantic features while explicitly ignoring surface-level dialect markers. NLP, education measurement, and equity analysis skills apply.
Genes
Structural cousins: stuck the same way, in other fields
Sources
Chaitanya Ramineni & David Williamson, "Understanding Mean Score Differences Between the e-rater® Automated Scoring Engine and Humans for Demographically Based Groups in the GRE® General Test," ETS Research Report Series 2018(1), pp. 1–31, Anastassia Loukina, Nitin Madnani, Klaus Zechner, "The many dimensions of algorithmic fairness in educational applications," Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2019), Nils-Jonathan Schaller, Yuning Ding, Andrea Horbach, Jennifer Meyer, Thorben Jansen, "Fairness in Automated Essay Scoring: A Comparative Analysis of Algorithms on German Learner Essays from Secondary Education," Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Abubakir Siedahmed, Jaclyn Ocumpaugh, Zelda Ferris, Dinesh Kodwani, Neil Heffernan, Eamon Worden, "Nonstandard English and the Automated Scoring of Open-Ended Math Problems," Proceedings of the 18th International Conference on Educational Data Mining (EDM 2025), Austin Pack, Alex Barrett, Juan Escalante, "Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability," Computers and Education: Artificial Intelligence 6 (2024), 100234, Accessed 2026-08-20. go to source 1 ↗ go to source 2 ↗ go to source 3 ↗ go to source 4 ↗ go to source 5 ↗
Working notes
Notes made while writing and checking this brief, shown as written. They use working shorthand.
Work testing LLMs for essay scoring has expanded (Siedahmed et al. 2025), while algorithmic fairness in education is, by Schaller et al.'s (2024) account, rarely investigated empirically. Related to but distinct from `education-growth-mindset-structural-blind-spot` (which covers mindset research methodology) and `education-curriculum-assessment-misalignment` (which covers curriculum-test alignment). Loukina, Madnani, and Zechner (2019) draw on the fairness definitions in Berk et al.'s review of algorithmic bias in criminal-justice risk assessment, and note that the educational measurement community was concerned with the fairness of automated scoring long before the topic became widely popular.
Reconciliation 2026-08-20: Both original citations were wrong and the brief's headline statistic did not exist. (1) "Burchfield et al., 'Fairness in Automated Essay Scoring,' ACL BEA Workshop 2024" is not a real paper. The URL attached to it, https://aclanthology.org/2024.bea-1.18, is Schaller, Ding, Horbach, Meyer & Jansen, "Fairness in Automated Essay Scoring: A Comparative Analysis of Algorithms on German Learner Essays from Secondary Education" (BEA 2024) — verified and now cited under its real authors, with its actual finding (skewed training subsets produced low accuracy for out-of-distribution students, and representative training data are needed for all relevant user groups, cognitive as well as demographic). (2) The "Mizumoto & Eguchi, Computers and Education: AI, 2024" citation grafted two different papers together: Mizumoto & Eguchi's real paper is "Exploring the potential of using an AI language model for automated essay scoring" (Research Methods in Applied Linguistics, 2023), while the title given belongs to Pack, Barrett & Escalante, "Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability," Computers and Education: Artificial Intelligence 6 (2024), 100234 — that paper is now cited under its real authors and used for what it actually reports (119 placement essays, four LLMs, GPT-4 best on intrarater reliability, interrater reliability with humans declining slightly over time). (3) Retracted: the claim that "a fine-tuned DeBERTa-v3 model scored high-proficiency ESL essays 10.3% lower than native-speaker essays of identical human-rated quality," the claim that "approximately two-thirds of AES studies rely exclusively on standard English training data," and the "contrastive learning debiasing has been proposed (2026)" line. None could be traced to any real source and all are removed; the Source Notes sentence above about the 10.3% gap being larger for lower-proficiency writers is therefore void and should not be carried forward. (4) The empirical core is now Ramineni & Williamson's ETS Research Report on the GRE General Test, which is verified and carries real numbers: standardized machine-minus-human differences of about −0.13 (African American subgroup, argument prompts) and −0.19 to −0.22 (Taiwan, argument), versus +0.68 and +0.56 for test takers from China. That last figure required softening the brief's framing: the divergence is group-dependent and changes sign, so "systematically penalizes" was too strong as a universal statement and the title was changed from "Automated Essay Scoring Systematically Penalizes Non-Dominant Dialects" accordingly. (5) Added as a deliberate counterweight: Siedahmed et al. (EDM 2025) found that varying the number of AAVE features in open-ended math responses did not change GPT scoring while prompt engineering did — a partial null against the simple dialect-marker-penalty story, retained because the brief should not be built on the assumption it is trying to test. Source tier updated 2 → 1 (ETS research report + peer-reviewed workshop and journal papers).
Reconciliation 2026-09-30: Read against the sources on 2026-09-30; a second reader went over the repairs on the same day. Ramineni & Williamson (ETS Research Report 2018) was read in full text from the Internet Archive copy of the Wiley page (web.archive.org/web/20241122124727/onlinelibrary.wiley.com/doi/full/10.1002/ets2.12192; the live Wiley page refuses scripted requests). Loukina, Madnani & Zechner (BEA 2019) and Schaller et al. (BEA 2024) were read in full from the ACL Anthology PDFs. Siedahmed et al. (EDM 2025) was read in full from the proceedings PDF. Pack, Barrett & Escalante (2024) could be read only as its abstract and bibliographic record (DOAJ record and the Internet Archive copy of the ScienceDirect page); ScienceDirect serves a captcha to scripts and no repository copy was found. Changes: (1) "ETS traced the mechanism to" is now "ETS's expert review pointed to," and "located the cause in" is now "pointed to": the report says it proposes hypotheses about root causes and that "it appears that e-rater is not severe enough on language errors, overvaluing organization and development." (2) The claim that automated scoring "is now common in university placement exams and K-12 writing assessment" is in none of the sources; the sentence now gives what Schaller et al. say, that "AES is widely used in education in many countries." (3) "Deployment is outrunning fairness validation" is softened to "fairness validation lags behind it"; Schaller et al. say that while AES is widely used "it is unclear whether the algorithms used are fair to all groups of students," and Pack et al. say the validity and reliability of LLM scoring "is not well understood." (4) The claim that the weighting difference is one "that post-hoc rescaling cannot remove" is not in the report, which did not test rescaling as a remedy; it tested logistic, unit-weighted and CART models and found that none improved overall performance and that the effect on subgroup differences varied (the China difference fell under some models while the Taiwan difference grew). The sentence now says that. (5) Pack et al. were said to find interrater reliability "against humans declining slightly over time across models"; the abstract says it decreased slightly for GPT-3.5 and GPT-4 only. The text now names the two models and drops "against humans." (6) Quotation marks around "content knowledge and argumentation quality" and "surface-level linguistic conformity to standard English" are removed; neither phrase appears in any source. (7) The Source Notes sentence about a 10.3% gap being larger for lower-proficiency writers, already declared void in the 2026-08-20 note, is removed; the sentence after it ("the problem is worsening because LLM-based grading is expanding rapidly without fairness validation") now says only that work testing LLMs for essay scoring has expanded (Siedahmed et al.) while algorithmic fairness in education is "rarely investigated empirically" (Schaller et al.). (8) The Source Notes sentence attributing similar bias patterns to The Markup and ProPublica is in none of the sources, and its closing phrase calling education assessment "an emerging frontier" runs against Loukina et al., who say the educational measurement community "has been concerned with the fairness of automated scoring since long before this topic gained wide popularity"; the sentence now says what Loukina et al. do say. Looked at and left as written: the opening sentence, that both legacy engines and LLM-based scorers produce group-dependent divergence. The first pass narrowed it to the legacy engine; the second reader restored it, because Schaller et al. report that fairness issues based on demographic variables "were shown for large language models" in English essays by first- and second-language students, and Siedahmed et al. report earlier work on LLMs and dialect differences "demonstrating scoring biases." Confirmed in the sources and left as written: the Source line for all five papers; the standardized mean differences −0.13 (African American, argument, Table 5), −0.22 (Taiwan, argument, Table 5) and −0.19 (Taiwan, argument, Tables 15–16), +0.68 (China, issue, Table 4) and +0.56 (China, argument, Table 5); e-rater's operational use on the GRE General Test and the TOEFL test; human raters treating language control as a threshold with e-rater using linear weighting of all features, with organization and development most heavily weighted and correlated with essay length; Loukina et al.'s use of simulated and real data, the native-language effect, and the conclusion that all fairness definitions may not be achievable at once; Schaller et al.'s comparison of shallow, deep and LLM scorers on German secondary-school essays, the no-bias-but-low-accuracy result for skewed training subsets, and the call for training data on all relevant groups including cognitive ability; Siedahmed et al.'s four AAVE features, synthetic standardized controls, and the finding that dialect-feature count did not change GPT scoring while prompt engineering did; Pack et al.'s 119 placement-test essays, four models scored on two occasions, and GPT-4 best on intrarater reliability and validity. Not checked: anything in Pack et al. beyond the abstract, including whether interrater reliability is measured against the human raters. No title changed. Genome Tags untouched.