Sunday, September 15, 2013

"Replication" and "Reproducibility"

We might be inclined to use the terms "replicate" and "reproduce" interchangeably. However, in the context of scientific verification, a distinction has been drawn between them.

The 2 December 2011 issue of Science devoted a special section to "Data Replication and |Reproducibility". Among the contributors was Roger Peng, whom I follow on the Simply Statistics blog. In one of his posts, Roger defines the two terms under discussion:

"......I define “replication” as independent people going out and collecting new data and “reproducibility” as independent people analyzing the same data. Apparently, others have the reverse definitions for the two words. The confusion is unfortunate because one idea has a centuries long history whereas the importance of the other idea has only recently become relevant. I’m going to stick to my guns here but we’ll have to see how the language evolves."
And the discussion has continued.

Recently, Roger has produced a three-part post, titled "Trading a New Path for Reproducible Research": Part 1; Part 2; Part 3.

It's definitely worth a read if you're involved in data-based research.

© 2013, David E. Giles

1 comment:

  1. Here is what you can find at the replication section of the Journal of Applied Econometrics.

    "There are two aspects to replication of results published in JAE:

    Replication in a narrow sense: This would involve checking the submitted data against the primary sources (when applicable) for consistency and accuracy, checking the validity of computations either directly or by carrying out the estimation (including the computation of standard errors) using other computer packages.

    Replication in a wide sense: This assumes that replication in the narrow sense has been successful, but asks if the substantive empirical finding of the paper can be replicated using data from other periods, countries, regions, or other entities as appropriate. Replicators are required to submit their output, preferably using different software from that used by the original article, as well as a short write up of their results. In cases where the results are not replicable, the authors will be given the opportunity to review the submission."