A second run can check the calculation. A second study asks whether the finding survives another encounter with the world.
Imagine a team studying whether a quieter reading room helps people finish a task. They collect observations, analyze a spreadsheet and publish a promising result. A colleague downloads their files, runs the analysis and obtains the same table. That is useful news. But nobody has yet invited a new group of readers into the room.
This hypothetical example separates two achievements that a headline can easily collapse into “scientists confirmed it.” One checks the path from recorded observations to a reported answer. The other takes the question back to the world.
Same files, different question
The National Academies’ 2019 report on reproducibility and replicability uses a specific distinction: computational reproducibility concerns consistent results from the same data and analytical procedure; replicability concerns consistency between studies that collect their own data to address the same question. Terminology varies between disciplines, so a paper’s actual procedure matters more than the label alone.
In our reading-room example, reproducing the table could catch a missing file, an undocumented exclusion or a calculation that does not match the published description. It gives another person a route through the work instead of asking them to trust a polished chart.
Yet the same files contain the same limitations. Perhaps the study recruited only volunteers who already preferred silence. Rerunning their records does not bring in people who declined to participate. A recoverable answer is not automatically a broadly applicable answer.
A result can repeat with its mistake intact
The report’s chapter on computational reproducibility makes a crucial qualification: reusing erroneous code can reproduce an incorrect result. Agreement between two runs is not an independent inspection of every assumption. Documentation must also identify data, methods and the computational environment; a file that worked on its author’s machine may depend on an unrecorded software version.
Suppose our imaginary spreadsheet accidentally records seconds as minutes for one group. A faithful rerun might preserve the mistake perfectly. An independent check of the units asks something the rerun alone did not: does the recorded quantity mean what the analysis says it means?
That is why “the code ran” and “the reasoning was checked” deserve separate sentences. A reader should be able to tell whether someone merely executed the package, inspected the transformations or independently rebuilt the calculation. None of those activities is worthless. Their evidential reach is different.
New data do not require identical numbers
Now imagine a second team collecting fresh reading-room observations under comparable conditions. Their average difference will probably not match the first team’s final decimal. The question is whether the results are consistent in light of their uncertainty, not whether the tables look like photocopies.
The report’s chapter on replicability emphasizes comparing both the closeness of estimates and their uncertainty. A second study may be too imprecise to settle the issue. Disagreement can also expose differences in conditions worth investigating rather than immediately proving misconduct or making the original conclusion untouchable.
For our hypothetical readers, room lighting, task difficulty or recruitment could matter. Those are candidate explanations to test, not excuses to invent after every disappointing result. A useful follow-up makes its comparison clear: which conditions were held similar, which changed and what outcome would count against the proposed explanation?
Read the verb before trusting the headline
When an article says a finding was “confirmed,” look for the action behind the word. Were the original files rerun? Were measurements collected again? Was the result tested in a different population? A classroom demonstration, a computational check and a new empirical study should not silently trade places.
Our discussion of facts and interpretations offers a related habit: keep the observation separate from what you conclude from it. Here the observation might be that an analyst recovered a published table. The stronger claim—that the effect reliably holds for new people—still needs its own support.
A trustworthy research story does not need the biggest possible verb. It needs a precise account of what was checked, what survived that check and what remains open.
Original conceptual illustration by M.R. Orion. Its bars are schematic, not experimental data. The reading-room study is an invented explanatory example.
M.R. Orion
