Carnegie Commons Blog

Evidence for Improving Schooling: Role of RCTs and the “Tiers of Evidence” Framework

The term “evidence-based” is mentioned some 60 times in the new Every Student Succeeds Act. The act’s attention to how research can and should play a more central role in improving schooling is a much-needed antidote in our field where fads tend to run strong but supportive evidence is often weak or non-existent.

But left much less clear is the practical question: “What does evidence for local improvement actually look like?” The act makes references to the Tiers of Evidence set out in the I3 program and operative in the What Works Clearinghouse. However the body of such evidence is quite limited, and questions are now being raised about its validity and utility (reference Smith and Ginsberg) and even whether this would actually be the best guidance we could possibly afford educators were it somehow magically available.

Educators are not alone in this regard. These same questions are being raised about other practical improvement endeavors. Recently I came across a video of a session organized by the Development Research Institute at NYU. The session focused on a then recently released book, “”Poor Economics”, that sought to synthesize findings and draw out the practical implications from a large body of RCTs carried out on fighting global poverty. (add reference) The subsequent commentary by Angus Deaton, a Nobel Prize winning economist at Princeton University, offers a cogent critique of the “RCT as Gold Standard” paradigm. And, since the Tiers of Evidence Framework rest on the latter argument, implicit here are words of caution about the limits of this framework as well. Toward the end of his commentary, Deaton offers his own view as to what evidence for practical improvement actually looks like. Although not specifically referencing improvement science or networked improvement communities, Deaton’s argument is quite consistent with the six improvement principles.

We present below key segments from Deaton’s presentation. For those interested in viewing the entirety of the conference go here.

Deaton begins by talking about the nature of evidence and its relevance for policy aimed at improving practice. “What is it that RCTs actually tell us and don’t?”