Russell Roberts Unable to Respond to the Answer that I gave him?

In a previous post Russell Roberts asked for a single example of a useful instrument in empirical work.

So again, my question to my better-read colleagues in the profession--give me an example of a statistical analysis that relies on instrumental variables, for example, that is done well enough, that is so iron-clad, that it can reverse the prior beliefs of a skeptic. . . . .

I wrote him back:

1) I don't put a huge amount of weight often on instrumental variables, but let me give one example from my own work on giving women the right to vote. The instrument there is whether states voluntarily or were forced to give women the right to vote. We found that both types of states experienced a similar increase in government growth after women were given the right to vote. If it was simply increased liberalism by men that caused both suffrage for women and government growth, you should see that in states that reached a critical mass to voluntarily give women the right to vote, but not in others where states were forced to given them a vote.

After several responses back and forth I wrote Russell that: "The best that I can see in you response to what I wrote is 'So let me try again. Very few econometric analyses persuade skeptics.' You have a similar response on your blog. It does not appear to me that this is a helpful response nor is it very direct. Explain why my responses, particularly the instrument regarding women's voting, doesn't meet your concerns. You asked for an example and I gave you one that used an instrument. I have yet to see one paper challenge the instrument that we used on women's voting, and you provide no logical objection."

(Cut)

Russell asks that I not print his reply.

Roberts and Hanson Rounds II and III

Unfortunately, Russell Roberts seems unwilling to respond to my previous comments on his postings on empirical work. If he wants to talk about the weaknesses of empirical work, be specific. If he wants to talk about empirical work free from political biases, have the guts to point to specific examples where this bias exists and say how he would do it differently. What is this work missing. He cites Ed Leamer (a former professor of mine), but he ignores that the work that he does discuss does follow Leamer's recommendations. So then what would Roberts do differently?

No Roberts tries to impose the obiligation on those who might know the empirical literature better.

Robin is taking my observation about pragmatism and applying it to handguns. I didn't mean to. I brought up pragmatism in order to highlight the general dangers of excessive faith in reason. Assuming that econometric analysis always trumps an anecdote is an example of the potential dangers of econometric analysis. Yes, relying on anecdotes is lousy science. But lousy econometrics is lousy science, too. What my podcast with Ayres made me realize is that lousy econometrics may be the norm rather than the exception.

Such cynicism can come cheap. It also seems to leave us with anecdotes. Well, there's also common sense, intuition and general lessons gleaned from experience and empirical work that is less prone to manipulation.

So again, my question to my better-read colleagues in the profession--give me an example of a statistical analysis that relies on instrumental variables, for example, that is done well enough, that is so iron-clad, that it can reverse the prior beliefs of a skeptic.

OK, let me give Russell a response. These points and other similar ones are in my book Freedomnomics. The second point below is also in More Guns, Less Crime. I realize that Russell hasn't had time to read either book, but before he comments more on these types of empirical work, he might benefit from reading them.

1) I don't put a huge amount of weight often on instrumental variables, but let me give one example from my own work on giving women the right to vote. The instrument there is whether states voluntarily or were forced to give women the right to vote. We found that both types of states experienced a similar increase in government growth after women were given the right to vote. If it was simply increased liberalism by men that caused both suffrage for women and government growth, you should see that in states that reached a critical mass to voluntarily give women the right to vote, but not in others where states were forced to given them a vote.

2) Regarding correlation and causation that is precisely why some research try to provide many qualitatively different empirical tests. For example, with right-to-carry laws: 1) violent crime falls, 2) the size of the drop increases over time the longer the law is in effect because more permits are issued, 3) there are differences between violent crimes where a criminal comes in contact with a victim who might be able to defend herself and a property crime where there is no contact (violent crimes fall relative to property crimes), 4) there are differences between different types of violent crimes (e.g., between murder generally and multiple victim public shootings because the probability that someone will be able to defend themselves with multiple victim public shootings is much higher than the case where there is only one or two victims present), 5) a comparison between adjacent counties on opposite sides of a state border, and 6) differential benefits across different types of victims.

Russell, try to come up with an alternative explanation for these different findings.

As a general comment, I am disappointed how vague Roberts' discussion is and how filled it is with platitudes.


Finally, Robin Hanson summarizes where Russell might be coming out on all this: "Russ finally answers as I'd originally expected: he relies on simpler clearer data and theory. " I think that there is a lot of regression and empirical work that uses very simple approaches (see the above reference to women voting or I would even argue many issues involving concealed handguns such as permit holders being extremely law-abiding and not posing a risk by themselves), So Russell, what do you have to say to that? In addition, Russell, I don't think that Ayres conceded anything to you on Friedman's Monetary History.