A guest post by Will Baude, Adam Chilton, and Anup Malani.

At a recent symposium on Developing Best Practices for Legal Methodology, we proposed a set of principles for rigorous demonstration of claims about legal doctrine. Legal scholars, advocates, and judges commonly make such claims, we thought, but without a systematic demonstration of supporting evidence. One question we received from many of the participants at the symposium was whether this was indeed a common phenomenon.

So to get a better sense of how frequently doctrinal claims are made without systematic support, we reviewed every article published in the last completed volume of ten top law reviews. For each article, we had a research assistant read the abstract and record any claim about the state of legal doctrine. The research assistant then read the article and recorded the evidence that was provided as support: at most a single case, multiple cases, or some form of a systematic review (that is, define the entire set of cases that was relevant to the claim and the evidence to support it).

The results of this review are in the above table. Our analysis suggested that roughly 50% (69 of 139) of articles included a claim about the state of legal doctrine in the abstract. Of these 69 articles, only about 20% (14 of 69) provided any form of systematic review to support the doctrinal claim. The rest of the articles provided string cites to cases (and occasionally, academic articles as well), but did not explain how they identified the universe of cases or whether they are representative.

This strikes us as suboptimal. To be clear, we have no particular reason to think that the doctrinal claims made in these articles are wrong. And we do not fault anybody for failing to adhere to a norm that does not yet exist. But our argument is that are important reasons that legal academia should develop a standard that helps legal analysts more rigorously document their claims about the state of legal doctrine.

Here are five reasons. First and most obviously, a more rigorous demonstration of evidence makes it easier for readers to evaluate the truth of a claim. Second, it is easier for readers to assess how confident to be in a claim. Third, it can prevent mistakes, even by experts. Fourth, it increases general progress in the field because it makes it easier for future research to build on the work from the past. Fifth, it helps reduce the risk or perception of bias.

Tomorrow we will present our proposal for how doctrinal claims can be made more systematically. If you are interested in reading more, you can read our essay on the topic that is now forthcoming in the University of Chicago Law Review.

A guest post by Will Baude, Adam Chilton, and Anup Malani.

How can the distinctive methodology of law be pursued carefully and rigorously? Much of the core task of a lawyer, a judge, or a researcher amounts to assessing substantive legal materials, like constitutional provisions, legislation, or court decisions. But these materials are often canvassed and analyzed non-systematically. Although there are already extensive debates about how to interpret statutes or constitutional text or precedents, those debates tend to center on substantive disagreements about the legal effect of the provisions and not the more methodological question of *how* a given interpretive approach should be pursued. The academy can help by developing best practices for careful analysis.

On May 6th-7th, the University of Chicago Law Review will bring scholars together for a symposium on this subject: Developing Best Practices for Legal Analysis.  In addition to ourselves, the participants for the event include Oren Bar-Gill, Omri Ben-Shahar, Curtis Bradley, Melissa Carlson, Vince Chhabria, Frank Easterbrook, Richard Epstein, Richard Fallon, Tom Ginsburg, Abbe Gluck, Katerina Linos, Florencia Marotta-Wurgler, Bernadette Meyler, Richard Posner, Fred Schauer, Lawrence Solum, Barbara Spellman, Nick Stephanopolous, Cass Sunstein, and Adrian Vermeule.

A guest post by Adam Chilton:

UCLA law professor Lynn LoPucki recently posted an article examining 120 empirical papers that were published in 15 leading law reviews and the Journal of Empirical Legal Studies. LoPucki coded these articles to see who was writing them, and what kind of data they used. The paper’s primary finding is that articles in the dataset written by law professors without a PhD were more likely than those with a PhD to code original data instead of using existing dataset sources. From this fact, LoPucki concludes that there is a “reduction in coding resulting from the hiring of more J.D.-Ph.D.s” that is undesirable for the legal academy. 

I have two thoughts about this interesting finding. First, I don’t think that we can infer from LoPucki’s paper that law professors with PhDs actually code less data. The reason is that LoPucki’s units of analysis are papers and not professors. It could be the case that law professors with PhDs code new data for a lower percent of their empirical articles, but still collect new data for a much higher absolute number of papers. LoPucki’s claim could be true–and I don’t have data to suggest otherwise–but I don’t think that it’s demonstrated in his current paper.

Second, I’m not persuaded by the claim that the measure of a good empirical legal studies paper is whether it codes original data. To be fair, LoPucki’s argument is not that coding original data is necessarily important for all disciplines. Instead, his argument is that  coding original data is important for legal scholarship because it typically requires direct engagement with legal materials like cases or statutes.

I think that for all empirical papers–whether written by law professors or economics professors–the question of whether to collect original data or use existing data sources should depend entirely on what data are already available. For some topics it would be a waste of resources to reinvent the wheel, and for other topics collecting new data may be absolutely necessary. I think that an empirical legal paper is valuable if it credibly brings new evidence to an important topic, not if the dataset was coded specifically for the paper.

A guest post by Adam Chilton and Mila Versteeg:

As the graph above shows, about 84% of countries now have prohibitions against using torture enshrined within their constitutions. Given the recent revelations that even wealthy and democratic countries like the United States have engaged in torture, it’s worth asking whether these constitutional prohibitions have helped to actually reduce torture.

In a short article on the Washington Post’s Monkey Cage blog, we explain our recent research that finds that constitutional torture bans have not reduced torture. Although the available data on rates of torture around the world obviously isn’t perfect, a few widely used datasets have been developed. After analyzing these datasets, we’ve found that countries with torture prohibitions torture more than those without, and that rates of torture do not go down after torture prohibitions are adopted.

Although our Monkey Cage post relies on raw data to make our argument, in our academic article on the topic we use a variety of more sophisticated empirical methods. We specifically rely on a new method developed by the political scientist Yoanthan Lupu designed to address the endogeneity concern that plagues human rights research: decisions to adopt human rights protections are systematically related to human rights practices. Although Lupu uses this method to study human rights treaties, we use it to study constitutional rights. The basic idea is that we first estimate countries’ constitutional preferences using a technique known as ideal point estimation, and then match countries together based on those preferences and a range of other variables. We are then able to estimate the effect of torture prohibitions on torture rates.

When using this method, and a range of more conventional regression techniques, we do not find any evidence that constitutional torture prohibitions have reduced torture. Moreover, not only do the estimated effects of constitutional prohibitions never achieve statistical significance in the dozens of regression models we use in our paper, the estimated effects are consistently substantively small as well. Our findings thus suggest that if we want to reduce levels of torture around the world, we have to find a better way to do it than just enshrining torture prohibitions in constitutions. If you’d like to learn more about our research on the topic, our paper is available on SSRN.

A guest post from Adam Chilton:

When do attempts by Chinese firms to acquire American companies generate political backlash in the United States? In a paper that was just published, The Political Economy of Inward FDI: Opposition to Chinese Mergers & Acquisitions, I explored this topic with Dustin Tingley, Chris Xu, and Helen Milner.

As the above figure from our paper shows, investments from China in the United States have increased dramatically in the last 15 years. During that time, Chinese firms have attempted or completed Mergers & Acquisitions with over 500 American companies. These M&As have even included Chinese firms taking over large publicly recognized American companies like IBM’s Thinkpad division and Smithfield Foods. Although some of these transactions have gone largely unnoticed, others have generated considerable backlash from both the public and government officials.

To test what factors predict political opposition to these deals, we utilized a dataset of 569 attempts by Chinese firms to complete M&As with American companies between 1999 and 2014. For each transaction, we searched a number of news databases to determine whether each transaction generated political opposition. We then collected information on a variety of factors about the companies involved and the relevant industries for each transaction so we could test what factors are associated with M&As generating political opposition.

We found that political opposition is more likely when the American companies targeted by Chinese firms are in security sensitive industries or economically distressed industries. We also found evidence of reciprocity: opposition is more likely when the American companies face restrictions from acquiring firms in the same sectors in China. If you’re interested in learning more about the results, or the methods and data we used, the paper is available on SSRN.

A Guest Post by Adam Chilton and Christopher Whytock:

Discretion to make political or legal decisions is frequently given to one branch of government based on the belief that it is better suited to make a particular kind of decision than another branch. For example, much of administrative law is premised on the idea that administrative agencies are better positioned to make decisions about how to carry out their missions than judges. When these claims of comparative institutional competence are made, however, they are rarely based on systematic empirical evidence. This is in part because there are rarely opportunities to evaluate what happens when two different branches of government are tasked with making the kind of decision.

 In a paper published last week, Foreign Sovereign Immunity and Comparative Institutional Competence, we empirically evaluate one comparative institutional competence claim by taking advantage of a situation where Congress moved the authority to make a certain kind of decision from the State Department to the courts. That situation was created by the passage of the Foreign Sovereign Immunities Act (FSIA) of 1976.

There is a longstanding principle of customary international law that governments should not be subject to suit in the courts of a foreign country. In 1952, however, the United States adopted a new, restrictive theory of when sovereign immunity should be granted. This paved the way for foreign governments to be sued in American courts over their “commercial activities” (like breaking a contract with an airplane manufacture) but not their “public acts” (like passing legislation that limits what kinds of airplanes are allowed to fly in their airspace).

Of course, whether any particular action taken by a foreign government constitutes a commercial activity or a public action is not always clear. When suits against foreign governments arose, initially it was the State Department that was forced to make these calls. It was not long though before critics began to argue that the State Department was making politically motivated decisions—for example, that immunity was awarded to important countries even though a specific suit was clearly based on commercial activities that should prevent immunity from being granted. In 1976, Congress responded to these criticisms by moving the authority to make foreign sovereign immunity decision from the State Departments to the courts.

To leverage this change in authority, we built a database of immunity decisions made before and after the passage of the FSIA. By controlling for the facts underlying each dispute and the characteristics of the parties involved, we are able gain some traction on how these two different branches of government have made foreign relations decisions. In contrast to previous studies that evaluated a small number of cases qualitatively, our study does not reveal evidence of systematic bias in the State Department’s immunity decisionmaking, while it does identify potential political influences on the courts’ decisionmaking. Although there are admittedly some limitations to our data and approach, these results still challenge the frequently made argument that the State Department is worse at making legal determinations free from political interference than other branches of government. If you want to read more about our method and results, you can find the paper at SSRN.