Instructor-designed AI improved exam results in a trial across 19 universities, a preprint reports
The authors report a 0.32 SD gain on proctored exams at student level, from 4,768 students in the learning analysis of a trial covering 7,200 students in applied STEM classes at 19 universities, 17 of them in the US by our count. They found no statistically significant effect on engagement or satisfaction.

Classes that brought in AI tools designed by their own instructors improved more on proctored, AI-free exams, relative to their own earlier terms, than classes that waited a term, according to a randomised trial across 19 universities posted to arXiv on 14 November 2026. The preprint (arXiv 2611.19099) is led by Jun Li of the University of Michigan, lists 52 authors and has not been peer reviewed.
Who was studied
The paper describes two populations, and they differ:
- The whole trial: "The experiment involved 19 universities, 42 instructors, and 7,200 students", in 55 applied STEM classes, "over three academic semesters from spring 2025 to spring 2026". A class is "a course–instructor pair".
- The learning analysis: after exclusions, the exam result rests on "44 class-terms for learning (4,768 enrolled students)". A class-term is one term of one class. Both headline estimates below use this sample and compare each class's change from its own earlier terms, treatment classes against control classes (a difference-in-differences design).
Subjects ranged from statistics and machine learning to forecasting and decision analysis. By our count of the paper's ethics section, 17 of the 19 institutions are in the US; the others are the University of Hong Kong and the Indian Institute of Management Ahmedabad.
How it worked
Classes were randomly assigned "to implement their instructor-designed AI innovations either immediately (treatment) or with a one-teaching-term delay (control)". Each instructor designed their own tool, and simply "allowing ChatGPT on a homework assignment would not suffice". The tools "spanned 33 types, and seven pedagogical functions", including "individualized coaching agents (19 classes)" and "tutors (18 classes)". Instructors in both arms received technical and AI-design support.
Learning was measured with "uncurved scores from AI-proof assessments, typically from proctored, in-person, pen-and-paper examinations." Ten class-terms whose assessments did not meet that bar were dropped from the learning analysis.
The results
Scores were standardised within each class, so effects are given in standard deviations (SD), a unit based on the spread of scores within each class.
- Student level: "The student-level treatment effect estimate is 0.32 standard deviations, SD" (SE=0.13; P=0.019).
- Class level: "The class-level estimate is 0.26 SD" (SE=0.12; P=0.038).
The authors call these effects "sizable", citing "a median improvement of 0.10 SD" across pre-K–12 randomised interventions and "a mean improvement of 0.47 SD" for active learning in undergraduate STEM.
On the other outcomes: "We found no statistically significant effects for student engagement or satisfaction". The point estimates were −0.07 SD (student level) and −0.05 SD (class level) for engagement, and −0.11 SD and −0.10 SD for satisfaction, with P values from 0.29 to 0.60.
Earlier studies
The authors write that their "positive learning estimates contradict the negative estimates found in the other major studies on AI in education", citing three: a PNAS study of high school mathematics, a British Journal of Educational Technology paper and an Annenberg Institute working paper on a university trial. Two of this paper's authors, Hamsa Bastani and Alp Sungu, are also among the authors of the PNAS study. The authors call their use of frontline-driven AI their study's "key differentiator": "Whereas other studies use a top-down approach, imposing a uniform protocol devised by the researchers, we use a bottom-up approach".
The paper's reference list, which we read in full, does not include the study behind our 22 August report on a CEPR discussion paper which found that, in observational data on secondary students in one county in China, homework scores rose about 18% after students took up AI while monthly exam scores fell about 20%. The two differ in country, age group and design. On our reading they answer different questions, and neither settles the other.
Limits and caveats
The authors list four limitations:
- "the participating classes were primarily applied STEM, and effects in other disciplines and educational settings remain to be tested."
- "few classes were observed during a second or third implementation, limiting conclusions about the persistence of learning gains".
- "our engagement measures varied across sites and courses".
- "our RCT does not have enough statistical power to distinguish effects across different AI innovations and instructional settings."
The paper also sets out:
- Pre-registration: it "was pre-registered on the Open Science Framework (osf.io) on 28 February 2025, and amended on 6 March 2025, 15 August 2025, and 24 August 2026." The paper says the last amendment "removed these two options, neither of which had been used by participating instructors, and introduced the Simple IV specification", an alternative estimator.
- Analysis choices: the headline figures come from "regressions that is slightly simpler than the ones we pre-registered"; the authors say the pre-registered versions "yield nearly identical results". They did not pre-register the AI-proof exclusion, "as we did not anticipate the problem of AI use on the assessments."
- Non-compliance: two treatment class-terms "did not meaningfully implement their proposals". They are left out of the main regressions but kept in alternative ones, which the authors say give similar results.
- Data and code: restricted-use data "will be made available to qualified researchers with appropriate IRB approval" under a data use agreement or memorandum of understanding; code "is available to editors and referees upon request."
Funding and partners
The study "was supported by discretionary research funds from Northwestern University and the University of Michigan", and "The authors declare no competing interests." The paper also says: "We also co-developed Faculty Twin with Google's Rapid Innovation Team." Faculty Twin let instructors configure and deploy AI agents in a secure cloud environment. In the text we read, the paper does not say how many classes used it.
Why it matters
On our reading, this is a large randomised test of AI in university teaching, scored on exams that students could not use AI in. It is narrow by its own account: instructor-designed tools, applied STEM, mostly US universities, and mostly a single term of use (the paper says only three class-terms in the analysis sample came from a second or third implementation). The authors credit instructor design for the gain, but the trial compared those tools with a delay, not with centrally designed tools, so on our reading that point rests on comparison with other studies. It is a preprint with restricted data, and it leaves open whether the gains last or carry over to other subjects.
Ask Relay — he reads every question himself and replies personally by email.
