A reproducible paper lets readers compare a later analysis with the original report. Learners connect tables, figures, and claims to the relevant analysis and supporting outputs. They also write limitations that name the source of uncertainty, its likely effect, and the boundary it places on the conclusion.
By the end of this module, you will be able to:
Map a textual claim to the relevant result, table or figure, analytic output, and underlying data and code materials.
Distinguish planned primary and secondary analyses from exploratory analyses in a results narrative.
Revise a results paragraph to report an estimate, uncertainty, and a decision that materially affects interpretation.
Write a limitation that identifies its source, likely consequence, and boundary on the conclusion.
“The intervention improved student engagement” can sound decisive even when the study observed one self-selected group, used an uncertain measure, or produced an imprecise estimate. A reader should be able to locate the result in a table or figure, identify the analysed data and model behind it, and see the uncertainty that qualifies the interpretation. Quantitative reporting guidance calls for estimates and confidence intervals, clear distinctions among primary, secondary, and exploratory analyses, and reporting of missing data and analytic problems that could affect validity. Journal Article Reporting Standards for Quantitative Research in Psychology
Transparent reporting makes a claim inspectable; it does not prove that the claim is true. A useful limitations section therefore does more than announce that “limitations exist”: it explains what may have distorted, narrowed, or weakened a conclusion and where readers should stop extending it. Discussing Study Limitations in Reports of Biomedical Studies
A reported finding should have a visible route back to the work that produced it. Start with a precise definition of the outcome or construct, then connect the claim to the analysed dataset, preparation steps, analytic code, model or analytic procedure, and saved output. A table, figure, or excerpt should display the relevant evidence rather than merely decorate the narrative.
For quantitative work, a source pointer can name the table or figure, the analysis identifier or script, the dataset version or access route, and the output containing the estimate. Reporting guidance also calls for enough information about models, software, diagnostics, and anomalies for readers to understand what was estimated. Journal Article Reporting Standards for Quantitative Research in Psychology
For qualitative work, the route is different but equally important. Explain the data sources, how data were selected or collected, how the analytic process developed, and how excerpts, cases, or other supporting material ground the interpretive claim. Qualitative reporting guidance asks authors to demonstrate the analytic process and show that claims are warranted by the evidence. Journal Article Reporting Standards for Qualitative Primary, Qualitative Meta-Analytic, and Mixed Methods Research
In mixed-methods work, make both strands traceable and explain the connection between them. State how qualitative and quantitative results were integrated—for example, in a joint table that places themes beside estimates—rather than presenting two unrelated sets of findings. Journal Article Reporting Standards for Qualitative Primary, Qualitative Meta-Analytic, and Mixed Methods Research
Claim-to-evidence example
Claim: Students assigned to the feedback condition reported higher engagement at the final survey than students in the comparison condition, although the estimate was imprecise.
Evidence route: Outcome definition in Methods → cleaned analysis dataset →final_engagement_modelanalysis → model output → Figure 2 and Table 3 → uncertainty interval reported beside the estimated difference.
This route lets a reader ask productive questions: Was engagement measured as stated? Which cases entered the model? Were missing responses handled as described? Does the table match the output? Does the wording stay within what the design can support?
Label analyses according to their role. A primary analysis addresses the main hypothesis or question; a secondary analysis is planned but not central; an exploratory analysis examines patterns not specified as primary or secondary. These labels help readers assess how chance and analytic flexibility may have influenced a result. Journal Article Reporting Standards for Quantitative Research in Psychology
Report the result, not only a conclusion about it. For each material analysis, provide the estimate and its uncertainty—for example, a confidence interval where applicable—and identify the outcome, comparison, model, and analysed sample. If a null or inconclusive result is relevant to the stated question, report it rather than replacing it with a more favourable subgroup or exploratory pattern. Journal Article Reporting Standards for Quantitative Research in Psychology
Report decisions that could change interpretation. These include exclusions after data collection, transformations, outlier handling, missing-data procedures, adjusted analyses, and departures from the planned analysis. If a change occurred after analysis began, state that fact and give the rationale. Journal Article Reporting Standards for Quantitative Research in Psychology
Missing data deserve more than a count in a footnote. Report how much data were missing, the likely causes where they can be assessed, and the method actually used to address missingness. Readers need this information because missingness can alter who contributes evidence to a result. Journal Article Reporting Standards for Quantitative Research in Psychology
Keep measurement, selection, and implementation visible in the interpretation. Define how variables were measured; report relevant reliability information for the scores analysed where possible; describe how participants or cases were selected; and, for interventions, provide evidence about whether the intervention was implemented as intended. These details affect what the finding means and to whom or what setting it may apply. Journal Article Reporting Standards for Quantitative Research in Psychology
Do not turn an observed association into a universal or causal claim without support from the design. Discuss generalisability—also called external validity—as the extent to which a finding may apply beyond the studied sample and setting. Consider the target population, setting, measurement, time, and other contextual features that may limit that extension. Journal Article Reporting Standards for Quantitative Research in Psychology
A limitation should contain three parts:
Source: What feature of the design, data, measure, sample, analysis, or implementation creates the concern?
Likely consequence: How might it affect the estimate, interpretation, or ability to apply the finding? State the direction only when you have a reasoned basis; otherwise say that the direction is uncertain.
Boundary: What conclusion should readers not draw, or what population, setting, measure, or causal interpretation lies beyond the evidence?
This approach turns limitations into interpretive guidance. Authors are often well placed to explain the likely direction and possible extent of a bias because they know how the study was conducted, but they should also give the reasoning that lets readers evaluate that judgement. Discussing Study Limitations in Reports of Biomedical Studies
Weak limitation | Why it is weak | More useful version |
|---|---|---|
“The sample was small.” | It does not identify what the size changes or how readers should interpret the result. | “The analysed sample produced a wide uncertainty interval around the estimated condition difference. The data are therefore consistent with a range of plausible differences, so the result should not be treated as a precise estimate of the effect.” |
“Self-selection may be a limitation.” | It names a problem but does not explain the comparison or the conclusion at risk. | “Participants chose whether to join the programme, so participants in the two groups may have differed before the programme began. The observed difference cannot by itself establish that the programme caused the outcome difference.” |
“Our measure may not be perfect.” | It does not specify the measurement problem or its consequence. | “Engagement was measured through a single self-report item, which may not capture all aspects of engagement. The conclusion is limited to differences in this reported measure, not engagement in every form.” |
“Implementation varied across sites.” | It omits how variation affects interpretation. | “Delivery procedures differed across sites, so the study estimates outcomes of the programme as implemented in these sites rather than a single uniform intervention. Differences between sites may reflect implementation variation as well as other contextual differences.” |
“More research is needed.” | It is generic and leaves the current boundary unstated. | “A study that recruits a broader range of settings and records implementation consistently would test whether the finding extends beyond the participating sites.” |
Limitations are not a ritual list of defects. Include limitations that could materially affect the quality, interpretation, or applicability of the evidence, including problems in design and implementation. Discussing Study Limitations in Reports of Biomedical Studies
The dashboard intervention increased student engagement and should be adopted across universities. Engagement was significantly higher in the intervention group, proving that the dashboard works.
Students in the dashboard condition reported higher engagement at the end of the study than students in the comparison condition. The estimated difference and its uncertainty interval are reported in Table 2, with the corresponding model output in Analysis Output A.
This was the planned primary analysis; exploratory subgroup analyses are reported separately in Appendix B. The comparison was based on students who completed the final survey, and missing-response handling is described in Methods, “Missing data.”
One limitation is that students selected whether to participate in the dashboard condition. Pre-existing differences between participants may therefore contribute to the observed difference, and the result should not be interpreted as proof that dashboard access caused higher engagement or that the same result will occur across universities.
The revised paragraph gives readers a finding they can locate, identifies uncertainty rather than suppressing it, distinguishes the planned analysis from exploratory work, and sets a specific boundary on the causal and generalisable claim. Reporting this route and its limitations makes the conclusion easier to inspect; it does not establish that the conclusion is true.