experimentswithoutfaces

Problem statement

The Institute of Education Sciences created SEERNet in 2021 to turn five widely used digital learning platforms (ASSISTments/E-TRIALS, MATHia/UpGrade, OpenStax/Kinetic, Canvas/Terracotta, ASU Online) into research infrastructure, so that education experiments could be run faster, cheaper, and at scale, with an explicit mandate for "equity-focused" research. The obstacle SEERNet's own guidance surfaces is that the platforms generally do not know who their students are: "a DLP may not have access to individual student or teacher demographic data (that is, without connecting to other data sources or embedding a survey) but may collect school-level covariates, including locale type … or Title I status." Answering the question policymakers most want answered — whether an intervention narrows or widens gaps by race, income, language status, or disability — requires linking platform records to student information systems held by districts, which means data-sharing agreements, de-identification protocols, and IRB review that a platform-embedded experiment was supposed to make unnecessary.

Why this matters

The promise of platform-based research is replication at scale: hundreds of A/B tests on real coursework instead of one $3M field trial. If that machinery can only report average effects, it will systematically miss heterogeneous effects, and equity claims — the stated purpose of the IES investment — will rest on school-level proxies (Title I status, urban/rural) that blur within-school differences. Researchers also cannot easily measure attrition or compliance for subgroups, and, as the report notes, some platforms do not even provide direct attrition data — it must be reconstructed from activity timestamps. The result is a research infrastructure whose cheapest studies are its least informative for the students the studies are meant to serve.

What’s been tried and why it hasn’t worked

SEERNet's approach so far has been procedural: advising researchers to explore existing data before designing a study, to negotiate with the platform about what can be varied, and to "weigh the tradeoffs and level of effort that may be required to access and integrate additional data, especially if that necessitates further data sharing agreements with educational institutions." Some platforms offer to broker district relationships and coordinate de-identification "such that the researcher never sees personally identifiable information," and some populate rosters from an LMS or SIS, which "could afford possibilities for linking with other sources of data given appropriate permissions" — but each of these is a per-study negotiation, which reintroduces the cost and delay the infrastructure was built to eliminate. Embedded consent-plus-survey collection of demographics works better for platforms serving adults (OpenStax, ASU) than for K–12 platforms where minors cannot consent and districts control the data. The report also flags that IRBs unfamiliar with platform research "may … be more conservative in their review process," and that researchers must find out whether a missing data type reflects "current technical constraints that could be relaxed in the future or if it is so by design." No standard, reusable mechanism exists for equity-relevant covariates to travel with a platform experiment.

What would unlock progress

A standardized, privacy-preserving way for platform experiments to obtain subgroup covariates without per-study bespoke agreements — for example, a district-side service that returns only aggregate subgroup treatment effects (or salted, hashed subgroup labels) computed against the platform's randomization, in the spirit of secure multiparty computation or differential-privacy releases used in census and health data; a model data-sharing agreement and IRB template specific to platform-embedded experiments; or validated methods for estimating heterogeneity from the school-level covariates platforms already hold, with honest bounds. Adjacent precedents: statistical linkage units in health (trusted third parties that link and de-identify), and ad-tech's aggregate-measurement APIs that report conversions by cohort without exposing users.

Entry points for student teams

A software/privacy team could prototype a "trusted linkage" service: the platform exports randomization assignment and outcome by pseudonymous ID, the district holds demographics, and the service returns only subgroup effect estimates above a minimum cell size — testing it on synthetic SIS and platform data and measuring what precision is lost relative to individual-level linkage. A policy/legal team could draft a model FERPA-compliant data-sharing agreement and IRB protocol for platform experiments and pilot it with one district and one open platform (ASSISTments' E-TRIALS is designed to host external researchers). A statistics team could quantify, using public NAEP or state data, how much subgroup heterogeneity is recoverable from school-level proxies alone. Skills: privacy engineering, education law, causal inference, human-subjects ethics.