No "must do" list is ever going to be complete, let alone perfect. This is certainly true when it comes to itemizing essential ground-rules for all of us when we embark on applying our knowledge of econometrics.
That said, here's a list of ten things that I like my students to keep in mind:
- Always, but always, plot your data.
- Remember that data quality is at least as important as data quantity.
- Always ask yourself, "Do these results make economic/common sense"?
- Check whether your "statistically significant" results are also "numerically/economically significant".
- Be sure that you know exactly what assumptions are used/needed to obtain the results relating to the properties of any estimator or test that you use.
- Just because someone else has used a particular approach to analyse a problem that looks like yours, that doesn't mean they were right!
- "Test, test, test"! (David Hendry). But don't forget that "pre-testing" raises some important issues of its own.
- Don't assume that the computer code that someone gives to you is relevant for your application, or that it even produces correct results.
- Keep in mind that published results will represent only a fraction of the results that the author obtained, but is not publishing.
- Don't forget that "peer-reviewed" does NOT mean "correct results", or even "best practices were followed".
I'm sure you can suggest how this list can be extended!
© 2013, David E. Giles
Put comments in your code, otherwise you will forget.
ReplyDeleteDon't use methods you don't yet understand.
ReplyDeleteIf you really want to be a scientist, beware of points 3 and 4. if you always follow them, you never get important results. No assumption (which is the essense of "common/economic sense" should overrun data as it is.
ReplyDeletePeter Kennedy also had his ten commandments for applied researchers (lots of overlap with your list)
ReplyDelete1. Thou shalt use common sense and economic theory.
Corollary: Thou shalt not do thy econometrics as thou sayest thy prayers.
2. Thou shalt ask the right questions.
Corollary: Thou shalt place relevance before mathematical elegance.
3. Thou shalt know the context.
Corollary: Thou shalt not perform ignorant statistical analyses.
4. Thou shalt inspect the data.
Corollary: Thou shalt place data cleanliness ahead of econometric godliness.
5. Thou shalt not worship complexity.
Corollary: Thou shalt not apply asymptotic approximations in vain.
Corollary: Thou shalt not talk Greek without knowing the English translation.
6. Thou shalt look long and hard at thy results.
Corollary: Thou shalt apply the laugh test.
7. Thou shalt beware the costs of data mining.
Corollary: Thou shalt not worship R2.
Corollary: Thou shalt not hunt statistical significance with a shotgun.
Corollary: Thou shalt not worship the 0.05% significance level.
8. Thou shalt be willing to compromise.
Corollary: Thou shalt not worship textbook prescriptions.
9. Thou shalt not confuse significance with substance.
Corollary: Thou shalt not ignore power.
Corollary: Thou shalt not test sharp hypotheses.
Corollary: Thou shalt seek additional evidence.
10. Thou shalt confess in the presence of sensitivity.
Corollary: Thou shalt anticipate criticism.
For the full article (and a very amusing explanation of the above), see Peter Kennedy "SINNING IN THE BASEMENT: WHAT ARE THE RULES? THE TEN COMMANDMENTS OF APPLIED ECONOMETRICS" Journal of Economic Surveys, Vol 16 N.4, 2002
Thanks Angelo! Vintage Peter - I miss him.
DeleteExcellent list. Thank you very much. I think it would be ok to add the maximum of George E. P. BOX "... all models are wrong, but some are useful." This is to raise awareness among users (and modelers) that econometric models are only useful scientific tools, not magic crystal ball that predicting the future without error.
ReplyDeleteI prefer Tarpey's maxim: all models are right, most are useless. He argues that if we regard models as approximations to the truth, we could just as easily call all models right. Slides here: http://andrewgelman.com/wp-content/uploads/2012/03/tarpey.pdf
ReplyDeleteFor practitioners: replicate before you try to implement!
ReplyDeleteThanks for the list prof. Giles
Well done Dave, Excellent post.
ReplyDeleteI think Canada may invade the US after this!
What about a rewording of what Heckman calls Marshak's maxim: Start with a question, not a method. Then find the relevant method to answer your question.
ReplyDeleteI like this list (particularly number 1). I would add: "Treat all estimated regression coefficients as indicating partial correlations rather than as structural parameters; then ask what different structural models are consistent with the correlation".
ReplyDelete