Developing Systematic Reviews for Legal Research

A guest post by Will Baude, Adam Chilton, and Anup Malani.

Yesterday we argued that claims about legal doctrine are frequently made without sufficient support, and argued that this can be a problem that should be remedied. Today we will briefly sketch our proposed solution.

In response to analogous concerns, other disciplines have developed methods for conducting “systematic reviews” of prior findings on a research question. Individual studies may have limited statistical power to answer a research question and their conclusions may be bound by the specific circumstances in which they were conducted. By contrast a review could aggregate the data and contexts from multiple studies to yield both a more precise and generalizable study. But it is important that the reviews be conducted in a rigorous way that minimizes the kind of bias and error we have discussed.

Although there are a number of variants of systematic reviews, they generally have four steps. First, the researcher should state clearly the question she is attempting to answer. Second, the research must justify and be transparent about how she defines and obtains the sample for her review. Third, the research ought to explain any weighting that is applied to the cases in the sample. Fourth, the researcher needs to justify and be transparent about the manner in which she analyzes the sample cases she reviews.

We argue that the insights of systematic review can be extended to law. It is true that legal analysis tends to be qualitative rather than quantitative, and that legal analysis is more likely to embed normative issues, but in our view those are not reasons to ignore the lessons of systematic review. On the contrary, a rigorous methodology allows us to separate positive and normative contributions and move arguments forward.

In our new essay, we explain exactly how we’d extend these four steps to legal analysis, and provide an example of a systematic review in action. Read it if you’d like to learn more.