Decompose
Turn the review question into explicit claims, populations, exposures or interventions, comparators, outcomes, and time frames.
Study portfolio
SL-003Stage 02: Protocol draftingPlanned intervention study
Teaching evidence synthesis under scientific disagreement
Research questionCan structured AI-assisted instruction improve how learners represent consensus, uncertainty, contradiction, and evidence quality?
SL-003 public status
- Current stage
- Protocol drafting Stage 02 of 05
- Recruitment
- Not started No participants are being enrolled
- Results
- None This case file contains no study findings
- Last reviewed
- 28 August 2026 Public planning record
PLANNED STUDY CASE FILE
A Learning Question With a Testable Decision
This case file focuses on disciplined synthesis rather than summary fluency. Learners would map claims to sources, separate absence of evidence from evidence of absence, distinguish inconsistency from imprecision, and communicate how strongly a conclusion is supported. AI assistance would be evaluated as an instructional scaffold whose own omissions and unsupported claims must be detected.
- Learning context
- A systematic-review, journal-club, research-methods, or evidence-based-practice course using compact source packets that can be reviewed in full.
- Decision this study should support
- Determine whether structured AI assistance improves source-grounded synthesis and whether it creates new risks of overconfidence, omission, or citation laundering.
Protocol objectives
- 01
Estimate the contrast in source-supported conclusion accuracy between instructional conditions.
- 02
Measure detection and representation of material scientific disagreement.
- 03
Assess whether learners calibrate conclusion strength to evidence certainty.
- 04
Test transfer to a new evidence packet without the original prompts or examples.
DESIGN CANDIDATE
What Would Be Tested
Every element remains provisional until the protocol is registered. Unknowns are shown as unknowns instead of being filled with unsupported precision.
- Design candidate
- Randomized or counterbalanced comparison using multiple complete evidence packets
- Unit of assignment
- Individual learner, pair, or course section after contamination is assessed
- Conditions
- Structured AI-assisted synthesis workflow and conventional critical-appraisal worksheet
- Reference standard
- Independent expert annotations followed by documented adjudication
- Sample size
- Not set. Precision must reflect learner and evidence-packet variance
- Domains
- To be selected using licensing, expert availability, learner relevance, and leakage risk
Learning sequence
Map
Link each material statement to supporting, conflicting, indirect, or missing source evidence.
Weigh
Evaluate risk of bias, inconsistency, indirectness, imprecision, and publication or selection concerns.
Calibrate
Write a conclusion whose scope and confidence match the available evidence, then test transfer on a new packet.
MEASUREMENT
Outcomes Defined Before Observation
Candidate roles may change during protocol review. Any primary outcome will be fixed before data collection or access to relevant outcome data.
| Outcome | Role | Operational definition | Timing |
|---|---|---|---|
| Atomic claim support | Candidate primary | Proportion of material conclusion claims correctly supported by the cited source passages. | Final synthesis |
| Contradiction representation | Candidate co-primary | Accuracy and completeness when identifying material disagreement and its likely sources. | Final synthesis |
| Certainty calibration | Candidate secondary | Agreement between expressed conclusion strength and expert-adjudicated evidence certainty. | Final synthesis |
| Selective omission | Safety measure | Rate at which evidence that could materially change the conclusion is omitted. | Final synthesis |
| Delayed transfer | Candidate secondary | Performance on a new packet without access to the instructional scaffold. | Delay to be specified |
ANALYSIS DISCIPLINE
A Plan That Can Report Unfavorable Results
Final estimands, models, exclusions, missing-data rules, multiplicity decisions, and stopping conditions will be specified in the registered protocol where applicable.
- 01
Define atomic-claim and contradiction units before scoring begins.
- 02
Model repeated evidence packets and raters, with clustering by course or learning group where applicable.
- 03
Report agreement before and after adjudication and disclose rubric changes.
- 04
Analyze unsupported claims, omissions, and overconfident conclusions as separate error classes.
- 05
Test sensitivity to evidence certainty, domain familiarity, packet length, and alternative scoring thresholds.
RESEARCH INTEGRITY
Implementation, Access, and AI Disclosure
The record must be detailed enough to audit what learners experienced, what the AI system could do, and where qualified humans remained responsible.
Implementation record
- Search or source-selection rule, packet inclusion criteria, licenses, and source versions
- Expert annotation protocol, training, conflicts, independent judgments, and adjudication
- Instructional materials, allowed tools, time limits, and learner support
- AI system details, prompts, retrieval scope, source access, and generated citations
- All exclusions, corrupted materials, protocol deviations, and scoring revisions
Equity and access
- Choose source packets that do not make advanced English composition the unmeasured target skill.
- Assess prerequisite domain knowledge and provide equivalent orientation across conditions.
- Document accessibility of PDFs, tables, citations, and annotation tools.
- Avoid using sensitive learner characteristics for subgroup claims without adequate design, consent, and precision.
AI-system disclosure
- Model identity, date, settings, context limit, retrieval method, and source corpus
- Whether the system can access material outside the controlled evidence packet
- Prompt sequence for claim decomposition, evidence mapping, critique, and conclusion drafting
- Citation generation and verification steps
- Human review required before any AI-proposed conclusion is accepted
VALIDITY REGISTER
Main Risks and Planned Responses
These responses reduce specific risks. They do not eliminate uncertainty or guarantee that the final design will support a causal claim.
Expert disagreement
Collect independent annotations, publish the adjudication process, and retain disagreement where consensus is not justified.
Packet construction bias
Prespecify selection rules and include supporting, null, conflicting, indirect, and lower-certainty evidence.
Benchmark leakage
Use controlled source access and newly assembled packets where licensing and review permit.
Writing quality confounding
Score evidence representation separately from style and surface fluency.
False authority from citations
Require passage-level verification and score citation presence separately from citation support.
STUDY GATES
Status Changes Require an Exit Record
A stage label is a public claim. The record moves forward only when its stated exit condition is documented.
Concept
CompleteLearning objective, target inference, candidate packets, and error classes are recorded.
Protocol drafting
CurrentDomains, reference standard, primary outcomes, and analysis decisions are fixed.
Review
Not startedScientific, education, licensing, privacy, and accessibility reviews are documented.
Registration and execution
Not startedProtocol, packets, prompts, and scoring rules are time-stamped before data collection.
Reporting
Not startedAll result directions, uncertainty, disagreement, deviations, and artifact access are reported.
EVIDENCE CONTEXT
Real Sources That Inform the Protocol
These external sources provide context for design decisions. They are not Santaros Labs outputs, endorsements, or evidence that this proposed study has been completed.
The BMJ · 2021
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Informs transparent reporting of review questions, selection, synthesis methods, results, limitations, and availability.
Open sourceCochrane · Current handbook
Cochrane Handbook for Systematic Reviews of Interventions, Chapter 14
Frames structured presentation of findings and assessment of certainty across risk of bias, inconsistency, indirectness, imprecision, and publication bias.
Open sourceUNESCO · 2023
Guidance for generative AI in education and research
Supports human-centered pedagogical validation and attention to privacy, agency, inclusion, and institutional readiness.
Open sourceOPEN SCIENCE PLAN
Planned Public Open
- Preregistered protocol
- Source-selection and licensing record
- Expert annotation handbook
- Learner scoring rubric
- Error taxonomy
- Evaluation harness
- Analysis code and governed learner-data statement
Availability will depend on consent, ethics review, licensing, privacy, security, and institutional requirements. A restriction will be explained rather than presented as open access.
Next portfolio record
SL-004Mentored team learning in shared AI research workspacesMETHODS COLLABORATION
Improve This Study Before Registration
We welcome educators, learning scientists, domain researchers, statisticians, research software specialists, and governance reviewers who can strengthen the protocol.
A research inquiry does not imply study enrollment, institutional approval, funding, or authorship.