"Replication" and "Reproducibility"

We might be inclined to use the terms "replicate" and "reproduce" interchangeably. However, in the context of scientific verification, a distinction has been drawn between them.

The 2 December 2011 issue of Science devoted a special section to "Data Replication and |Reproducibility". Among the contributors was Roger Peng, whom I follow on the Simply Statistics blog. In one of his posts, Roger defines the two terms under discussion:

"......I define “replication” as independent people going out and collecting new data and “reproducibility” as independent people analyzing the same data. Apparently, others have the reverse definitions for the two words. The confusion is unfortunate because one idea has a centuries long history whereas the importance of the other idea has only recently become relevant. I’m going to stick to my guns here but we’ll have to see how the language evolves."

And the discussion has continued.

Recently, Roger has produced a three-part post, titled "Trading a New Path for Reproducible Research": Part 1; Part 2; Part 3.

It's definitely worth a read if you're involved in data-based research.

© 2013, David E. Giles

Can Your Results be Replicated?

It's a "given" that your empirical results should be able to be replicated by others. That's why more and more journals are encouraging or requiring that authors of such papers "deposit" their data and code with the journal as a condition of acceptance for publication.

That's all well and good. However, replicating someone's results using their data and code may not mean very much!

Ten Things for Applied Econometricians to Keep in Mind

No "must do" list is ever going to be complete, let alone perfect. This is certainly true when it comes to itemizing essential ground-rules for all of us when we embark on applying our knowledge of econometrics.

That said, here's a list of ten things that I like my students to keep in mind:

1. Always, but always, plot your data.
2. Remember that data quality is at least as important as data quantity.
3. Always ask yourself, "Do these results make economic/common sense"?
4. Check whether your "statistically significant" results are also "numerically/economically significant".
5. Be sure that you know exactly what assumptions are used/needed to obtain the results relating to the properties of any estimator or test that you use.
6. Just because someone else has used a particular approach to analyse a problem that looks like yours, that doesn't mean they were right!
7. "Test, test, test"! (David Hendry). But don't forget that "pre-testing" raises some important issues of its own.
8. Don't assume that the computer code that someone gives to you is relevant for your application, or that it even produces correct results.
9. Keep in mind that published results will represent only a fraction of the results that the author obtained, but is not publishing.
10. Don't forget that "peer-reviewed" does NOT mean "correct results", or even "best practices were followed".

I'm sure you can suggest how this list can be extended!

© 2013, David E. Giles

More on Multiple Bubbles

In a recent post I highlighted a new EViews Add-in package, written by Itamar Caspi. His rtdaf package facilitates the application of the Right-Tail Augmented Dickey-Fuller tests that are "....designed to detect the presence of an unobserved bubble component in an observed asset price and to date-stamp its occurrence".

In part, the testing procedures are based on Phillips et al. (2013b). If you're following this literature, there are two other recent papers by those authors that are a must-read - Phillips et al. (2013,a,b)

References

Phillips, P. C. B., Shi, S., and Yu, J., 2013a, Specification sensitivity in right-tailed unit root testing for explosive behaviour. Oxford Bulletin of Economics and Statistics, forthcoming.

Phillips, P., S. Shi, and J. Yu, 2013b. Testing for Multiple Bubbles 1: Historical episodes of exuberance and collapse in the S&P 500. Working paper.

Phillips, P., S. Shi, and J. Yu, 2013c. Testing for Multiple Bubbles 2: Limit theory of real time detectors.  SMU Economics and Statistics Working Paper Series, No. 05-2013.

© 2013, David E. Giles

Some More Papers for Your "To Read" List

For better, or worse, here are some of the papers I've been reading lately:

• Chambers, M. J., J. S. Ercolani, and A. M. R. Taylor, 2013. Testing for seasonal unit roots by frequency domain regression. Journal of Econometrics, in press. 
• Chicu, M. and M. A. Masten, 2013. A specification test for discrete choice models. Economics Letters, in press. 
• Hansen, P. R. and A. Lunde, 2013. Estimating the persistence and the autocorrelation function of a time series that is measured with error. Econometric Theory, in press.
• Liu, Y., J. Liu, and F. Zhang, 2013. Bias analysis for misclassificaiton in a multicategorical exposure in a logistic regression model. Statistics and Probability Letters, in press.
• Thornton, M., 2013, The aggregation of dynamic relationships caused by incomplete information. Journal of Econometrics, in press.
• Wang, H. and S. Z. F. Zhou, 2013. Interval estimation by frequentist model averaging. Communications in Statistics - Theory and Methods, in press.   

© 2013, David E. Giles

Francis Smart on the "ThinkNum" Data Resource

Francis Smart has a nice post on his Econometrics by Simulation It's titled, "ThinkNum - A New Interactive Public Database and Graphing Tool".

As Francis discusses, ThinkNum is essentially a competitor of Quandl, but I'll let you read Francis's post yourselves.

© 2013, David E. Giles

Right-Tail Augmented Dickey-Fuller Tests in EViews

A few days ago, I received an email from Itamar Caspi, a regular follower of this blog. Itamar has developed a really nice EViews "Add-In" package that facilitates the application of "Right-Tail Augmented Dickey-Fuller" tests.

His Add-in package, named rtadf, covers four tests:

1. ADF.
2. Rolling ADF.
3. sup ADF (SADF). See Phillips et al. (2011).
4. Generalized SADF (GSADF), see Phillips et al. (2013).

New Econometrics Journal

There's a relatively new econometrics journal on the block. The Journal of Econometric Methods, published its second issue last month.

The Editors are Jason Abrevaya, Bo Honore, Atsushi Inoue, Jack Porter, and Jeffrey Wooldridge. The Aims and Scope of the journal are described as follows:

"The Journal of Econometric Methods welcomes submissions in theoretical and applied econometrics of direct relevance to empirical economics research. The journal aims to bridge the widening gap between econometric research and empirical practice. We aim to publish papers from top scholars in econometrics, but submissions must (i) consider a topic of broad interest to practitioners and (ii) be written in a style that is targeted at practitioners. Subject to these requirements, the journal will consider submissions in all areas of econometrics.

We will not consider submissions that are application-specific. While econometric methodology should be thoroughly illustrated with empirical data, such methodology should be useful above and beyond the specific application considered."

The articles published to date are of a very high standard, and I'm looking forward to seeing more.

© 2013, David E. Giles

From My Reading List...........

Here are a few of the papers that I've been reading over the past week or so:

• Amisano, G. and J. Geweke, 2013. Prediction using several macroeconomic models.Working Paper Series NO. 1357, European Central Bank.
• Arnold, B. C. and H. K. T. Ng, 2011. Flexible bivariate beta distributions. Journal of Multivariate Analysis, 102, 1194-1202.
• Johansen, S. and B. Nielsen, 2013.  Outlier detection in a regression using an iterated one-step approximation to the Huber-skip estimator. Econometrics, 1, 53-70.
• Magnus, J. R. and A. L. Vasnev, 2013. Practical use of sensitivity in econometrics with an illustration to forecast combinations. B. A. Working Paper No. 04/2013, The University of Sydney Business School, University of Sydney.
• Pendakur, K. and S. Sperlich, 2010. Semiparametric estimation of consumer demand systems in real expenditure. Journal of Applied Econometrics, 25, 420-457.
• Solon, G, S. J. Haider, and J. Wooldridge, 2013. What are we weighting for? Working Paper 18859. National Bureau of Economic Research.

© 2013, David E. Giles

Should I do a Ph.D.?

When it comes to discussions with students, there's one question that's a "hardy perennial". There's no simple answer, whatever your discipline is, but Tim Hopper is tackling the question, Should I do a Ph.D?, with a series of interviews.

The first interview is with John Cook, whose blog, The Endeavour, I like to follow. So, I was especially interested in what John had to say. He comes at the question as formally trained mathematician, with broad "real world" experience, but I found my self nodding appreciatively at most of what he had to say.

Tim provides links to some other related material that addresses the important question that he's raised, and I'm looking forward to seeing his upcoming interviews on this topic.

© 2013, David E. Giles