More Medicaid Power

Drum actually posted an explanation of power analysis last night and how it relates to the Medicaid study:

There are several things to say about the Oregon study, but I think the most important one is this: not that the study didn’t find statistically significant improvements in various measures of health, but that the study couldn’t have found statistically significant improvements. It was impossible from the beginning.

Here’s why. The first thing the researchers should have done, before the study was even conducted, was estimate what a clinically significant result would be. For example, based on past experience, they might have decided that if access to Medicaid produced a 20 percent reduction in the share of the population with elevated levels of glycated hemoglobin (a common marker for diabetes), that would be a pretty successful intervention.

Then the researchers would move on to step two: suppose they found the clinically significant reduction they were hoping for? Is their study designed in such a way that a clinically significant result would also be statistically significant? Obviously it should be…

So here’s the question: if the researchers ended up finding the result they hoped for (i.e., a reduction of 16 people with elevated GH levels), is there any chance that this result would be statistically significant? I can’t say for sure without access to more data, but the answer is almost certainly no. It’s just too small a number. Ditto for the other markers they looked at. In other words, even if they got the results they were hoping for, they were almost foreordained not to be statistically significant. And if they’re not statistically significant, that means the headline result is “no effect.”

Now, they didn’t design the Oregon program or they would have insisted upon more numbers for this very reason.  That said, how they probably should have been much more thoughtful about how they brought these findings too light given the inherent problems.

About Steve Greene
Professor of Political Science at NC State http://faculty.chass.ncsu.edu/shgreene

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: