SL-005Stage 05: Analysis and reportingCompleted evidence synthesis

What higher-education AI tutoring studies actually measure

Research questionWhat can controlled studies tell educators about AI tutoring, learning time, and learner experience in higher education?

Review the synthesis View source context

SL-005 public status

Record status: Completed
Completed: 28 August 2026
Participants: None recruited
Findings: Bounded

COMPLETED EVIDENCE RECORD

A Learning Question With a Bounded Answer

This completed record is an evidence synthesis, not a new intervention study. We reviewed public research to identify what educators can responsibly learn from controlled AI-tutoring evidence and what remains unknown. The synthesis keeps learning outcomes, time, learner experience, and transfer limits separate so a promising result is not mistaken for a universal teaching recommendation.

Learning context: Higher-education science teaching, with particular attention to whether an AI tutor is designed around active learning and whether learning is measured beyond immediate task completion.
Decision this study should support: Help educators decide which features of controlled AI-tutoring studies are worth testing in their own teaching, and which claims still require independent, multisite evidence.

Synthesis objectives

01
Describe the focal controlled study's instructional contrast, learner population, outcomes, and reported time use.
02
Separate evidence about immediate learning from evidence about engagement, motivation, transfer, and scale.
03
Identify design features that should be retained or challenged in future teaching studies.
04
Produce a transparent boundary around what this synthesis can and cannot support.

SYNTHESIS DESIGN

What We Reviewed

This record makes its source frame, extraction choices, interpretation rule, and completion criteria visible. It does not estimate a new learner effect.

Record type: Completed rapid evidence synthesis; no pooled effect estimate
Evidence scope: Public controlled studies of AI tutoring in higher education, with a focal randomized crossover trial
Comparison: AI tutoring versus active-learning or instructor-led teaching as reported by each source
Extraction: Instructional design, learner sample, outcome definitions, time, implementation, and stated limitations
Interpretation: Narrative synthesis preserving study-level context and uncertainty
Completion rule: Source set, extraction record, synthesis, limitations, and public links reviewed and archived

Learning sequence

Frame

Define the educator-facing question and distinguish evidence about learning from evidence about experience or efficiency.

Screen

Check study design, setting, intervention description, comparison condition, and outcome reporting before drawing a conclusion.

Extract

Record what learners did, what the comparison group did, what was measured, and where the source leaves uncertainty.

Bound

Translate the evidence into a teaching decision without extending a single course result to all learners, subjects, or tutors.

SYNTHESIS OUTCOMES

What the Record Separates

The table distinguishes what the reviewed sources report from the limits and future outcomes they cannot resolve. No new participant outcome was collected here.

Outcome	Role	Operational definition	Timing
Learning performance	Synthesis outcome	Reported change in assessed learning in the source study, retained in the source's own measurement context.	As reported by each source
Time on task	Synthesis outcome	Reported instructional or study time associated with the AI-tutoring and comparison conditions.	As reported by each source
Learner experience	Context outcome	Reported engagement, motivation, or perception measures, kept separate from evidence of learning.	As reported by each source
Generalizability	Validity outcome	Setting, population, tutor-design, and implementation boundaries that limit transfer to other teaching contexts.	At interpretation

COMPLETED RECORD

Findings With Their Boundaries

These are the conclusions supported by the completed source review. They are not claims about a Santaros intervention or a universal teaching effect.

01
The focal randomized crossover trial reported greater learning in less time for its AI-tutor condition than its active-learning comparison in an authentic undergraduate physics course.
02
The focal study evaluated a deliberately engineered tutor with content-rich prompts and pedagogical scaffolding. Its result therefore supports testing that instructional design, not every generic AI tool.
03
The public evidence is not sufficient to claim durable transfer, broad subject generalization, or superiority across learners, teachers, institutions, and model configurations.
04
For educators, the strongest reusable lesson is methodological: measure learning directly, record time and implementation, and keep learner experience distinct from learning outcomes.

Limits of this record

This is a rapid synthesis with a narrow public source set, not a comprehensive systematic review or meta-analysis.
No Santaros participant data, new experiment, or independent replication was conducted.
The cited focal trial is one higher-education context and cannot resolve questions about long-term learning, transfer, equity, or changing model behavior.
Future updates may revise the synthesis when additional controlled studies and replications become available.

INTERPRETATION DISCIPLINE

An Interpretation That Keeps Uncertainty Visible

The synthesis keeps source context, design limits, and unresolved questions attached to every conclusion. It does not convert standards or one study into a general recommendation.

01
Preserve each source's comparison, outcome definition, and uncertainty rather than recomputing an incompatible common effect.
02
Do not pool a single focal trial with other designs when the intervention, learners, outcomes, or teaching context are not commensurate.
03
Separate reported learning, time, engagement, motivation, and transfer claims in the extraction record.
04
Treat the focal result as evidence about the evaluated tutor and course design, not as evidence that AI tutoring is universally superior.
05
State where an independent multisite study, longer follow-up, or direct replication is needed before an educational decision is widened.

RESEARCH INTEGRITY

Implementation, Access, and AI Disclosure

The record must be detailed enough to audit what learners experienced, what the AI system could do, and where qualified humans remained responsible.

Implementation record

Public source links, publication dates, and access date recorded in the evidence register
Study setting, learner sample, comparison condition, and reported instructional sequence extracted separately
Outcome definitions and time measures retained in source context
Claims about learning, experience, and generalizability checked against the source methods and discussion
Synthesis language reviewed for unsupported causal or universal recommendations

Equity and access

The synthesis does not assume that a result in one higher-education course applies to learners with different language, disability, access, or prior-knowledge conditions.
Paid-tool access, device requirements, support time, and instructor capacity remain implementation questions for future studies.
Learner agency, privacy, and human teaching responsibility are treated as conditions of responsible use, not optional add-ons.

AI-system disclosure

The focal source evaluated a purpose-built AI tutor with pedagogical scaffolding; the source's intervention is not interchangeable with a general chatbot.
No AI system was used by Santaros Labs to decide source inclusion, extract findings, or interpret evidence for this record.
The source's model, prompting, lesson design, and course context are treated as part of the intervention and must be reported in any replication.
This synthesis makes no claim about current model versions or tools that were not evaluated in the cited studies.

VALIDITY REGISTER

Main Risks and Planned Responses

These responses reduce specific risks. They do not eliminate uncertainty or guarantee that the final design will support a causal claim.

R01

Single-study overreach

Name the focal study and keep its course, learners, tutor, and outcome boundaries visible in every conclusion.

R02

Learning confused with engagement

Extract and report performance, time, engagement, and motivation as distinct constructs.

R03

Intervention drift

Describe the tutor's instructional design rather than treating AI tutoring as a single stable treatment.

R04

Publication and selection bias

Label the synthesis as rapid and bounded, retain negative or null evidence when found, and invite source corrections.

R05

Context loss

Carry setting, sample, comparison, and implementation details into the decision summary.

STUDY GATES

Status Changes Require an Exit Record

A stage label is a public claim. The record moves forward only when its stated exit condition is documented.

Question

Complete

Educator-facing question, constructs, and decision boundary recorded.

Evidence scope

Complete

Public source scope and inclusion rationale recorded.

Extraction

Complete

Study design, learning outcomes, time, experience, and limitations extracted.

Synthesis

Complete

Narrative synthesis completed without unsupported pooling or universal claims.

Reporting

Complete

Evidence brief, links, limitations, and update path made public in this case file.

EVIDENCE CONTEXT

Real Sources That Inform the Protocol

These sources are the public evidence and methods guidance reviewed for this completed record. They are not Santaros Labs outputs or endorsements.

Scientific Reports · 2025

AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting

Reports a randomized crossover trial in an authentic undergraduate physics course and describes the engineered tutor, comparison, learning outcomes, and limits.

Open source

Institute of Education Sciences · Living standard

Standards for Excellence in Education Research

Supports relevant student outcomes, implementation evidence, generalizability, open science, and transparent uncertainty.

Open source

UNESCO · 2023

Guidance for generative AI in education and research

Frames human-centred pedagogical design, privacy, equity, teacher capacity, and institutional responsibility.

Open source

PUBLIC RECORD

Artifacts That Keep the Record Auditable

Completed evidence brief
Source and extraction register
Limits and replication questions
Accessible HTML summary
Correction and update log

This record can be updated when sources, corrections, or a future replication change the evidence boundary.

Next portfolio record

SL-006Teaching reproducible research as a learning outcome

EVIDENCE COLLABORATION

Help Keep the Evidence Current

We welcome educators, learning scientists, and domain researchers who can identify a missing source, challenge an interpretation, or propose a careful replication.

Start a research inquiry

Corrections and updates are welcome. A cited source does not imply endorsement by Santaros Labs.