The Importance of Interpretation

The concept of evidence-based policy was examined in a recent article by Pawson and colleagues (2011).The authors discussed the current trend of "evidence-based everything" and the impact this approach can have on policy making. They examined the example of proposing a policy banning smoking in a car when there are children present and the difficulty in providing conclusive evidence to support the policy.

Pawson and colleagues highlight the ongoing theme of their article in the following Donald Rumsfeld quote:

There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we now know we don't know. But there are also unknown unknowns. These are things we do not know we don't know."_

Many times the "known knowns" in research are somewhat conditional and can be interpreted differently depending on context and/or circumstances. In their example, the authors reviewed literature that measured the air quality in the back seat of a car when the driver was smoking. Despite the fact that this measurement provides a clear, numerical output, which is argued by policymakers to be evidence for the smoking ban policy, the authors highlighted the various conditions that could have impacted the measurement, such as open windows or blowing air conditioning.

Research is embedded in a complex system and as a result, findings are not always as black and white as we would like to think. The article's authors provide an extensive argument for the complexity of conducting research in the real world, where there are no true constants and all evidence is more adequately described as a conditional explanation, rather than empirical proof, of the findings. They describe the role of the researcher in converting "unknowns" to "knowns", and in weighing the significance of the information they reviewed, quite powerfully:

_"In short, the evidence does not deliver the legislative decision. It does, however, provide the grounds on which the policy maker can make a more informed decision."_

This explanation helps to highlight that data does not stand on its own, but rather always requires interpretation. I found this to be a particularly good reminder for evaluation research. It's important to feed the data back to the programs that we're evaluating and gather the program staff's interpretation to incorporate into the findings. At CRC, we have begun to create "learning circles" with program staff in certain domains to discuss data and collect their interpretation of it. We present quantitative and qualitative data we've collected and then ask the staff to reflect and help us explain why we're seeing certain patterns. We recognize that the staff members are the experts in their field and this process has allowed us to include their insight.

What processes do you go through to interpret data?

Reference:  Pawson, R., Wong, G., & Owen, L. (2011).Known Knowns, Known Unknowns, Unknown Unknowns: The Predicament of Evidence-Based Policy. _American Journal of Evaluation, 32_(4), 518-546.