The final module brings the six areas together in a supervised paper review. Learners use an evidence-review tool in a deliberate order: choose a permitted paper, inspect the extracted text, select relevant sections, examine the evidence behind each judgement, and document any override. The central habit is simple: state what the evidence supports, state what it does not, and leave uncertain cases open.
By the end of this module, you will be able to:
Select a permitted paper and identify the sections relevant to a defined review question.
Extract exact quotations and record a page, section, table, figure, or other source location for each judgement.
Distinguish whether a reported data, code, or preregistration link is reachable, accessible, documented, reusable, or executable.
Record a Met, Not met, or Needs review judgement and an evidence-based reason for any override.
A review goes wrong when the reviewer accepts an automated conclusion without checking the source passage, its location, or the material behind a reported link. The result may look consistent and efficient while confusing what a paper states with what has actually been inspected. Automation bias is the tendency to over-rely on automation, and it can lead people to accept incorrect advice rather than verify it. Automation bias: a systematic review of frequency, effect mediators, and mitigators
A defensible review makes its evidence trail visible: the review question, selected text, exact quotations, source locations, link observations, and final human judgement. A structured record is the bridge between what a source reports and the decisions made during review. Chapter 5: Collecting data | Cochrane
Begin by defining one review question that can be answered from a paper and, where relevant, from the materials it declares. For example: Does the paper state where the analysis code can be accessed? or Does the paper identify a preregistration record and explain departures from it? Choose a paper that you are permitted to review under the course, journal, institutional, or research-team context.
Next, select only the text that can reasonably answer the question. Relevant sections commonly include the methods, data-availability statement, code-availability statement, supplementary materials, acknowledgements, declarations, results, and preregistration statement. A paper may distribute relevant information across several reports or sources, so do not assume that one article contains every detail. Chapter 5: Collecting data | Cochrane
For each possible answer, inspect the exact quotation and its source location before deciding. Record the wording exactly, then note where you found it—for example, “Data availability statement,” “Methods, Analysis subsection,” “Table 2 note,” or a page number in the PDF. Recording direct quotations and their locations reduces the risk that a later reader must rely on your memory or interpretation. Chapter 5: Collecting data | Cochrane
Then make the decision that the selected evidence supports—no broader. A missing statement in the selected sections does not prove that code, data, or a registration does not exist; it means the evidence currently inspected does not establish the point. References, repository names, identifiers, or claims outside the selected material may justify follow-up, such as checking a linked record, another report of the study, an erratum, or asking the authors for clarification. Chapter 5: Collecting data | Cochrane
A link is not a conclusion about the material behind it. Check the level that your review question requires, and state clearly which levels you did and did not inspect.
Check | Meaning | What to record |
|---|---|---|
| The link opens or resolves at the time of checking. | URL or identifier, date checked, and any error or redirect. |
| You can obtain the relevant material, rather than merely reach a landing page. | Whether files, records, or access instructions are available to you. |
| The material includes enough explanation to identify its contents, purpose, structure, and conditions of use. | README, metadata, codebook, variable description, licence, or access instructions. |
| The available material appears to have the permissions and contextual information needed for an appropriate reuse attempt. | Licence or permission statement, documentation, and relevant restrictions. |
| Code can be run in an appropriate environment with its required inputs and dependencies. | This requires an execution attempt and a record of what happened. |
A paper-based review can establish what the paper says and may identify a declared link, but it cannot establish that restricted material is obtainable, that files are complete, or that code runs. An automated tool may organise candidate passages, suggest evidence, and check whether a link responds, but it cannot substitute for human inspection of access conditions, documentation, reuse conditions, or an execution attempt.
Do not turn a reachable link into a claim of reproducibility. Repository records and access statements are useful evidence, but a reviewer should report the observed level precisely: for example, “link reachable; files not inspected,” or “documentation present; reuse conditions unclear.” Data collected for review should remain connected to its source and distinguish source information from later calculations or interpretations. Chapter 5: Collecting data | Cochrane
Use labels as optional shorthand, not as proof of practical reproducibility.
Met: The selected evidence directly supports the defined review question. Record the quotation, location, and any link-check scope.
Not met: The selected evidence directly contradicts the criterion, or the relevant selected material clearly establishes that the required item was not provided. Do not use this label merely because an automated search found nothing.
Needs review: The evidence is incomplete, ambiguous, inconsistent, outside the selected material, inaccessible, or requires a check that you have not performed. Use it when the responsible conclusion is to leave the issue open.
Use Needs review when a paper names a repository but does not give a usable identifier, when a registration is mentioned without enough detail to locate it, when the link reaches a page rather than the claimed materials, or when relevant reports disagree. Multiple sources can contain conflicting information, and unresolved uncertainty may require follow-up rather than forced classification. Chapter 5: Collecting data | Cochrane
You may override automated output when the quotation is irrelevant, incomplete, mislocated, misread, contradicted by clearer evidence, or when the automated result implies more than the evidence supports. Write a brief reason that points to evidence rather than preference: “Override to Needs review: the extracted sentence names a repository, but the declared materials and access conditions were not inspected.” This makes the decision auditable and helps another reviewer understand the boundary between the tool’s suggestion and your judgement.
Human accountability matters because over-reliance on automated advice can introduce errors. Training and emphasising user accountability are identified as possible mitigators of automation bias. Automation bias: a systematic review of frequency, effect mediators, and mitigators
Fictional paper claim: “All analysis code and de-identified data are available in the project repository.”
Evidence found: In the paper’s data-availability statement, the reviewer finds the exact sentence: “All analysis code and de-identified data are available in the project repository.” The reviewer records the quotation, the section heading, and the repository link as the source location. An automated assistant highlights the same sentence and suggests Met.
Link check: The repository link is reachable and opens to a project page. The reviewer can see a short description, but does not inspect the files, cannot determine whether the data are obtainable, and does not attempt to run any code.
Human judgement: Needs review.
Reason recorded: “The paper declares shared code and data, and the repository link is reachable. The available material was not inspected for access, documentation, licence, completeness, or executability; the claim is therefore not verified beyond the declaration and reachable link.”
This decision does not say that the materials are absent or unusable. It states the narrower conclusion supported by the review record and prevents a link check from becoming an unsupported claim about reuse or execution.