Response to Malkin's Op-ed Below is Malkin’s op-ed with commentary by me (my comments are indented and in italics and start at the bottom of the page with the numbered responses corresponding to the numbers in the supporting document). (Note that two other discussions on this issue have been posted since February 2003 and involve a general discussion of the two other polls that ask about brandishing that have been done over the previous two decades as well a response to other attacks are available at the bottom of the page found here.) Despite being sent this information several times, she has not responded to any of these points. Steve Malzberg and Karen Hunter, co-hosts of a morning drive time show on WWRL (1600 AM) in New York, offered to let Malkin discuss these claims with me on the air, but she was unwilling to participate. It is disappointing that she will make allegations in print and on radio shows, but that she is unwilling to defend these assertions when I am present.

The general evidence for the survey is available here. The beginning of that document provides a brief abstract of the primary points.

An overview of the evidence is this: A) The survey was redone and the redone somewhat smaller survey produced similar results. In fact that survey data was already available at www.johnlott.org when Malkin wrote her piece.
B) The survey results in the single paragraphs in the two books where I have referenced this survey data was biased against the claim that I was making. I argued that the simple defensive brandishing or warning shots are not news worthy. The higher the rate of defensive brandishing or warning shots, the easier it is to explain why the media is not biased when it doesn't cover most defensive gun uses. If I wanted to show that the media was more biased, I should have used the surveys with lower defensive brandishing rates. I have also explained why the length of the time people are asked to recall events over can explain the difference in the four surveys on brandishing that have been done over the last twenty years (two designed by me and two by Gary Kleck).
C) Two people who took the survey have said that they took it. One person, James Hamilton, was interviewed by Professor Jeff Parker at GMU. As to the second person who took the survey, James Lindgren claims that David Gross took a different 1996 survey, but Gross's statements as well as the survey data from the 1996 survey indicate that Gross took my 1997 survey. The data from the 1996 survey is available from me or from the ICPSR under Hemenway's name. Other people were able to confirm various other aspects, such as the timing of when the survey was done and that I talked to people at the time of the survey. I have also supplied my tax records from 1997 to Joe Olson a tax law professor and other professors that show large payments for research assistants. Many others have confirmed many other aspects of what happened.
Bottom line: Science involves replication and I have always made my data available to others. In this case, I redid the survey and made that data available to anyone who wants access to it.

The other Lott controversy
Michelle Malkin
February 5, 2003

For those few of us in the mainstream media who openly support Second Amendment rights, research scholar John Lott has been -- or rather, had been -- an absolute godsend.

Armed with top-notch credentials (including stints at Stanford, Rice, UCLA, Wharton, Cornell, the University of Chicago and Yale), Lott took on the entrenched anti-gun bias of the ivory tower with seemingly meticulous scholarship. His best-selling 1998 book, "More Guns, Less Crime," provided analysis of FBI crime data that showed a groundbreaking correlation between concealed-weapons laws and reduced violent crime rates.

I met Lott briefly after a seminar at the University of Washington in Seattle several years ago and was deeply impressed by his intellectual rigor. Lott responded directly and extensively to critics' arguments. He made his data accessible to many other researchers.

But as he prepares to release a new book, "Bias Against Guns," next month, Lott must grapple with an emerging controversy -- brought to the public eye by the blogosphere -- that goes to the heart of his academic integrity.

The most disturbing charge, first raised by retired University of California, Santa Barbara professor Otis Dudley Duncan and pursued by Australian computer programmer Tim Lambert, is that Lott fabricated a study claiming that 98 percent of defensive gun uses involved mere brandishing, as opposed to shooting.

When Lott cited the statistic peripherally on page three of his book, he attributed it to "national surveys." In the second edition, he changed the citation to "a national survey that I conducted." He has also incorrectly attributed the figure to newspaper polls and Florida State University criminologist Gary Kleck.

1) The reference to the survey involves one number in one sentence in my book. Compared to the 98 percent number there was an earlier survey by Kleck that found 92 percent of defensive gun uses involved brandishing and warning shots and because the survey was asking people about events that occurred over a long period of time it is likely that it over emphasized more dramatic responses. (My number that is directly comparable to the 92 percent estimate is about 99 percent.) My point in the book was that defensive gun use rarely involves more “newsworthy” events where the attacker is killed and either survey would have made the general point. A general discussion of the different methodologies is provided here.

I never attributed my survey results to Kleck. What happened was that Dave Kopel from the Independence Institute took an op-ed that I had in the Rocky Mountain News and edited it for his web site. In the editing he added the incorrect reference to Kleck. (Statements from Kopel and others are provided in the supporting documents ). The two pieces are identical except for the reference to Kleck. As to the claim that I attributed the number to newspaper polls, that claim involves a misreading of two different sentences in an op-ed (see the material addressed in the second half of the link to point (1)). As to using the plural, that was an error. Given the years that have passed since I wrote the sentence, I cannot remember exactly what I had in my mind but the most plausible explanation is that I was describing what findings had been generated by the polls, in other words I was thinking of them as a collective body of research. I had been planning on including more of a discussion on the survey in the book, just as I have in my book that came out early this year, but I had a hard disk crash (see response (2)) and I lost part of the book along with the data.

More importantly, the survey results that I used were biased against the claim that I was making. The relevant discussions in both of my books focus on media bias and the point was that the lack of coverage of defense gun uses is understandable if most uses simply involve brandishing where no one is harmed, no shots fired, no dead bodies on the ground, no crime actually committed. If others believe that the actual rate of brandishing is lower and I had used the results of Kleck, it becomes MORE difficult to explain the lack of news coverage of defensive gun uses. The two short discussions that I have on this issue in my two books thus choose results that are BIASED AGAINST the overall point that I am making, that the media is biased against guns.

Some issues involving the source for Malkin's claims can be found here, here, and here.

Last fall, Northwestern University law professor James Lindgren volunteered to investigate the claimed existence of Lott's 1997 telephone survey of 2,424 people. "I thought it would be exceedingly simple to establish" that the research had been done, Lindgren wrote in his report.

Unfortunately, Malkin fails to mention that Lindgren is not an unbiased observer since I had written a journal article in Journal of Law & Politics critiquing some of his work months before he "volunteered to investigate" these claims.

It was not simple. Lott claims to have lost all of his data due to a computer crash.

2) As to the “claim” that I lost my data in a computer crash on July 3, 1997, I have offered Malkin the statements from nine academics (statements attached), four of whom I was co-authoring papers with at the time and who remember quite vividly also losing the data that we had on various projects. David Mustard at the University of Georgia spent considerable time during 1997 helping me replace gun crime data. Other academics worked with me to replace data on our other projects. Just so it is clear, this computer crash basically cost me all my data on all my projects up to that point in time, including all the data and word files for my book, More Guns, Less Crime, and numerous papers that were under review at journals. The next couple of years were hell trying to replace things and the data for this survey which ended up being one sentence in the book, was not of particular importance. However, all the data was replaced, including not only the large county level data, the state level data, as well as the survey data, when the survey was redone.

He financed the survey himself and kept no financial records.

* Unlike many academics, I have never asked for government support for my research. Nothing different or unusual was done in this case. While we still have the tax forms that we filed that show we made large expenditures on research assistants that year, my wife keeps our financial documents for the three years required by the IRS. I have provided my tax records from that year to several professors. Among them is a tax expert, Professor Joe Olson, at Hamline University in Minnesota, and he can verify this information. I have checked with the bank that we had an account with, but they only keep records five years back. Since wild claims have been made about the costs of the survey, some notion of its scope would be useful. The survey was structured so that over 90 percent of those questioned would only have to answer three short questions and those were usually completed in under 30 seconds. Less than one percent of those surveyed would actually answer as many as seven questions and even in that case the survey only took about two minutes. The appendix in The Bias Against Guns provides a description of the survey when it was replicated.

He has forgotten the names of the students who allegedly helped with the survey and who supposedly dialed thousands of survey respondents long-distance from their own dorm rooms using survey software Lott can't identify or produce.

* I have hired lots of student RAs over the years. Since I have been at AEI in the last couple of years I have had around 25 people work for me on various projects. The students in question worked for me during the very beginning of 1997. While I can usually reconstruct who has worked for me, it requires that I have that material written down. The information on these students was lost in the hard disk crash and given that I had lost data for other projects such as three revise-and-resubmits that I had at the Journal of Political Economy it was not a particularly high priority.

I don’t have the original CD with telephone numbers from across the country that was used to obtain telephone numbers, but I have kept one that I obtained later in 1997 when I was considering redoing the survey and I still have that available.

Assuming the survey data was lost in a computer crash, it is still remarkable that Lott could not produce a single, contemporaneous scrap of paper proving the survey's existence, such as the research protocol or survey instrument.

3) I have statements from two people who took the survey and other confirmatory evidence. As to the written material, being asked for written material six years after the survey is a long time. After the survey was done, the material was kept on my computer. In addition, I have moved three times (Chicago to Yale to Pennsylvania to AEI) as well as changed offices at Chicago and Yale since the summer of 1997. Yet, besides the statements from the academics who can verify the hard disk crash as well as the statement of those who participated in the survey, I do have statements David Mustard, who I had talked to numerous times about doing the survey with me during 1996 and who remembers after that us talking about the survey after it was completed. He is “fairly confident” that those conversations took place during 1997. John Whitley and Geoff Huck also have some recollections. Russell Roberts, now a professor at George Mason, was someone else that I talked to about the survey, but he simply can’t remember one way of the other. I didn’t talk to people other than co-authors about the survey and the research that I was doing on guns generally. This is because of the often great hostility to my gun work and also because I didn’t want to give those who disliked me a heads-up on what I was doing. I did have the questions from the survey and they were reused in the replicated survey in 2002.

After Lindgren's report was published, a Minnesota gun rights activist named David Gross came forward, claiming he was surveyed in 1997. Some have said that Gross's account proves that the survey was done. I think skepticism is warranted.

4) David Gross is the only person who Malkin mentions and she doubts his statements. Gross, a former city prosecutor, does have strong feelings on guns, but that is one reason why he remembers talking to me about the survey when I gave a talk in Minnesota a couple of years after the survey. There was no other gun survey on the questions that I asked during 1997. And another survey that was given close in time, during the beginning of 1996, was dramatically different from mine (e.g., the 1996 survey was done by a polling firm (not by students), was very long with at least 32 open ended questions (not something that could be done in a few minutes), involved Harvard (not Chicago), did not ask about brandishing, etc.). What Gross remembers indicates that it could only have been my survey.

Malkin also selectively quotes Lindgren. Lindgren told the Washington Times that, “I interviewed [Mr. Gross] at length and found him credible.” Mr. Gross has also responded to later statements made by Lindgren.

I have also had a second person who participated interviewed by Jeff Parker, the former associate dean at the George Mason University Law School. Parker interviewed both James Hamilton as well as Hamilton's sister, who claims that James told her about the interview when it occurred, and he can verify this information.

Lindgren claimed that Gross had instead answered a quite different survey done by Hemenway at Harvard, but when Hemenway finally released the data from both his 1996 and 1999 surveys and the age and other information about Gross and Hamilton do not match any subject interviewed in either survey.

 Lott now admits he used a fake persona, "Mary Rosh," to post voluminous defenses of his work over the Internet.

* When Julian Sanchez asked about the similarities between my writings and those posted under this Internet chat room pseudonym during this past January I did admit it immediately. (Sanchez had put up a post on his blog site asking for help in identifying someone who was cutting and pasting many of my responses from other places in chat room discussions. Because a dynamic IP address was being used, Sanchez could only identify the posting as coming from someone in southeastern Pennsylvania. When I found that he was asking for help in identifying the poster I admitted that I was using the pseudonym.) I had originally used my own name in chat rooms but switched after receiving threatening and obnoxious telephone calls from other Internet posters. Ninety some percent of the posters in the chatroom were pretty clearly using pseudonyms. The fictitious name was from a family e-mail account we had set up for our children based on their names (see latter discussion), on a couple of occasions I used the female persona implied by the name in the chat rooms to try to get people to think about how people who are smaller and weaker physically can defend themselves. Virtually all the posting were on factual issues involving guns and the empirical debates surrounding them. All that information was completely accurate.

"Rosh" gushed that Lott was "the best professor that I ever had."

*This was a family email account and I was not the only person who posted using this account.

She/he also penned an effusive review of "More Guns, Less Crime" on Amazon.com: "It was very interesting reading and Lott writes very well." (Lott claims that one of his sons posted the review in "Rosh's" name.)

*The e-mail account was set up by my wife for my four sons (Maxim, Ryan, Roger, and Sherwin in birth order) and involves the first two letters of each of their names in order of their birth. Maxim wrote several reviews on Amazon.com using that e-mail account and signed in using maryrosh@aol.com, not “Mary Rosh.” His posting included not only a review of my book, but also reviews of computer games such as Caesars III.

For whatever it is worth, a recent glich at Amazon.com revealed that it is quite common practice for authors to actually write positive anonymous reviews of their own books. The New York Times story on this revelation was actually quite sympathetic, which contrasts with the attack that the New York Times had on me when it also incorrectly claimed that I had written the review of my book.

Just last week, "Rosh" complained on a blog comment board: "Critics such as Lambert and Lindgren ought to slink away and hide."

By itself, there is nothing wrong with using a pseudonym. But Lott's invention of Mary Rosh to praise his own research and blast other scholars is beyond creepy. And it shows his extensive willingness to deceive to protect and promote his work.

*It would have been helpful if Malkin had actually read the text of what I wrote.

Some Second Amendment activists believe there is an anti-gun conspiracy to discredit Lott as "payback" for the fall of Michael Bellesiles, the disgraced former Emory University professor who engaged in rampant research fraud to bolster his anti-gun book, "Arming America." But it wasn't an anti-gun zealot who unmasked Rosh/Lott. It was Internet blogger Julian Sanchez, a staffer at the libertarian Cato Institute, which staunchly defends the Second Amendment. And it was the conservative Washington Times that first reported last week on the survey dispute in the mainstream press.

*The January 23rd story in the Washington Times could not accurately be described as a negative story. Professor Dan Polsby is quoted as saying that I was “vindicated.” Even Lindgren, a critic whose academic work I have criticized in the past (Journal of Law and Politics, Winter 2001), is characterized by the Times as believing that “ the question appears to have been at least partially resolved . . . “ and he did say that David Gross was a credible witness.

In an interview Monday, Lott stressed that his new defensive gun use survey (whose results will be published in the new book) will show similar results to the lost survey. But the existence of the new survey does not lay to rest the still lingering doubts about the old survey's existence.

*She never asked me any questions about whether the old survey was done.

The media coverage of the 1997 survey data dispute, Lott told me, is "a bunch to do about nothing."

*This quote is totally taken out of context. Some people had accused me of violating federal regulations regarding federal approval for human experiments while I was at Chicago. Malkin’s telephone call focused on that claim, and that is what my quote referred to.

I wish I could agree.

I spent years replacing the data lost in the hard disk crash. The county level crime data was replaced and given out to academics at dozens of universities so that they could replicate every single regression in More Guns, Less Crime. I have also made the data for my other book The Bias Against Guns available at http://www.johnlott.org/cgi-bin/login.cgi . The data for my other reserach has also been made available. The survey was also replicated and obtained similar results to the first survey and the new data has been made available since the beginning of the year. When asked I have even made my data available before the research was published. I don't think that there are any academics who have had a better record then I have in making my data available to other researchers. For an example of just on of my recent critics who has refused to share his data see here . I have provided Malkin with the information noted here, but she has never replied to e-mails that I have sent her.

Home

Johnlott.org (description of book, downloadable data sets, and discussions of previous controversies)

Academic papers:

Social Science Research Network

Book Reviews:

For a list of book reviews on The Bias Against Guns, click here.

---------------------------------
List of my Op-eds
---------------------------------

Posts by topic

Appalachian law school attack

Baghdad murder rate

Arming Pilots

Fraudulent website pretending to be run by me

Ayres and Donohue

Stanford Law Review

Mother Jones article

Vin Suprynowicz quote

Links

Craig Newmark

Eric Rasmusen

William Sjostrom

Dr. T's EconLinks.com

Interview with National Review Online

Lyonette Louis-Jacques's page on Firearms Regulation Worldwide

The End of Myth: An Interview with Dr. John Lott

Cold Comfort, Economist John Lott discusses the benefits of guns--and the hazards of pointing them out.

An interview with John R. Lott, Jr. author of More Guns, Less Crime: Understanding Crime and Gun Control Laws

Some data not found at www.johnlott.org:

Updated Media Analysis of Appalachian Law School Attack

Since the first news search was done additional news stories have been added to Nexis:

There are thus now 218 unique stories, and a total of 294 stories counting duplicates (the stories in yellow were duplicates): Excel file for general overview and specific stories. Explicit mentions of defensive gun use increase from 2 to 3 now.

Journal of Legal Studies paper on spoiled ballots during the 2000 Presidential Election

Data set from USA Today, STATA 7.0 data set

"Do" File for some of the basic regressions from the paper