Published papers often make claims that readers cannot check from the paper alone. This opening module gives learners a vocabulary for that problem. It separates reproducibility from replication, explains what open science can and cannot solve, and introduces evidence-based screening.
- Define reproducibility, replication, transparency, and open science in plain academic language.
- Tell the difference between a supported judgement and a claim that still needs evidence.
- Choose Met, Not met, or Needs review and explain the choice with a paper-based source.
- Recognise when a reviewer needs to inspect a repository, follow up with authors, or leave the item unresolved.
You open a published paper that reports a promising effect, but the methods omit key decisions and the paper says only that “data are available on request.” You cannot tell what was measured, how the dataset was prepared, or whether the reported analysis can be checked. Transparent reporting helps readers understand a study’s design, materials, measurements, data preparation, analyses, and uncertainty; these details support checking computations and attempting a later study. Understanding Reproducibility and Replicability
This module gives you language for describing what evidence is available without overstating what that evidence proves. You will learn to separate a paper’s assertion from material you can inspect, and to record uncertainty honestly.
By the end of this module, you will be able to:
Distinguish reproducibility, replication, transparent reporting, and automated screening in a short research scenario.
Identify whether a paper statement is supported by an inspectable method, material, dataset, code file, or other source.
Assign Met, Not met, or Needs review to an evidence-review item and justify the choice.
Explain one ethical limit on sharing research material that contains learner or other human-participant data.
Reproducibility asks whether another researcher can use the original study’s methods, materials, and, where appropriate, its original data or code to obtain consistent results. In computational work, this commonly means recomputing the reported result with the same input data, analytical steps, code, and conditions of analysis. Understanding Reproducibility and Replicability
Replication is a new study that examines the same or a similar scientific question using newly collected data. It asks whether results or conclusions remain consistent across studies, potentially in a new setting, with a different population, or under changed conditions. Understanding Reproducibility and Replicability
For example, a learning analytics team shares its anonymised analytic dataset, code, and documented steps for a model predicting course completion. A second researcher runs those supplied materials and obtains the reported result: that is a reproducibility exercise. Another team collects data from a later course offering and tests the same prediction question: that is a replication.
Terminology differs between disciplines, so authors and reviewers should state what they mean rather than assume a universal definition. Definitions - The Turing Way
Open science includes practices that make scientific knowledge more available, accessible, and reusable, while supporting collaboration and wider participation. It can include publications, research data, metadata, software, source code, source materials, workflows, and protocols, where appropriate. Recommendation on Open Science
These practices can make it easier for readers to inspect how a result was produced, scrutinise a reported analysis, and reuse suitable materials. Openness supports scrutiny and critique, but availability alone does not establish that another person has run the analysis correctly or reproduced the finding. Recommendation on Open Science
Open does not mean “share everything publicly.” Restrictions can be justified to protect privacy, confidentiality, personal information, and human subjects; when open access is not appropriate, mediated access, anonymisation, or pseudonymisation may allow appropriate sharing. Recommendation on Open Science
For learner data, ethical judgement therefore remains necessary. Consider what participants agreed to, what identifiability or harm risks remain, and whether learners or other affected groups have a meaningful voice in research that concerns them; the source packet supports inclusive participation but does not provide a specific consent procedure for learner-data research. Recommendation on Open Science
A statement such as “our code is available” is an assertion. It becomes usable evidence for a reviewer only when the paper provides enough information to find or inspect the material—for example, a working repository location, a persistent identifier, a supplement, or a clear access process.
Transparent reporting supplies information about design, materials, variables, data collection, preparation, planned and unplanned analyses, and uncertainty. It can make a study capable of being checked, but it does not itself show that a reviewer has rerun the work or that a new study obtained the same result. Understanding Reproducibility and Replicability
Automated screening can help a reviewer locate reported statements, quotations, links, and possible gaps. Treat its output as a tool-supported review of reported evidence—not as proof that the study has been reproduced.
Optional review label | Meaning |
|---|---|
| The available material directly supports the item being reviewed and gives a reader a usable way to inspect it. |
| The available material does not provide the required evidence or access route. |
| The available material is insufficient, unclear, or conflicting. This label does not mean that the research failed. |
A fictional paper states: “The full analysis code and de-identified learner dataset are openly available.” A reviewer checks the paper’s supplement and finds a repository link. The repository contains two analysis scripts and a short readme, but it does not contain the dataset, a data dictionary, or instructions for obtaining restricted data. One script refers to a file that is not included.
The appropriate judgement for the item “The data are available for inspection or access is clearly explained” is Needs review. The claim says that a de-identified dataset is available, but the reviewer cannot find it or an access process. The reviewer should record the paper statement, the repository contents inspected, and the missing information. They should not label the whole study unreproducible, because they have not attempted to run the analysis and do not know whether access is provided elsewhere.