Medicaid, statistical power, and health vs. wealth

The big news for policy wonks last week was a new study of Oregon Medicaid recipients that used a natural experiment to assess the impact of Medicaid.  Short version: No statistically significant” improvement on measured physical health outcomes over two years, but significant improvements in mental health and financial health.  Conservatives have been crowing about how it is therefore pointles to expand Medicaid, as Obama care does.  A lot of really smart responses.  Let’s start with Jon Cohn:

The big news is that Medicaid virtually wiped out crippling medical expenses among the poor: The percentage of people who faced catastrophic out-of-pocket medical expenditures (that is, greater than 30 percent of annual income) declined from 5.5 percent to about 1 percent. In addition, the people on Medicaid were about half as likely to experience other forms of financial strain—like borrowing money or delaying payments on other bills because of medical expenses.

That may sound obvious—of course people with insurance are less likely to struggle with medical bills. But it’s also the most under-appreciated accomplishment of health insurance: Whatever its effects on health, it promotes economic security. “The primary purpose of health insurance is to protect you financially in event of a catastrophic medical shock,” [emphasis mine] Finkelstein told me in an interview, “in the same way that the primary purpose of auto insurance or fire insurance is to provide you money in case you’ve lost something of value.” And while only a small portion of people will experience financial shock in any given year, over time many more will—which means many more will benefit from the protection that Medicaid provides.

Damn straight.  Yes, medical insurance keeps my family healthier, but mostly it keeps me out of the poorhouse.

Meanwhile, Chait takes this on as only Chait can:

Okay: The case for Medicaid expansion is not as strong as I had thought. Now for the caveats: The case for Medicaid expansion is overwhelmingly strong. If a study found that puppies survive steep falls at a higher rate than expected, then you could say the case for throwing puppies out of skyscraper windows has marginally weakened, but would remain extremely strong. Indeed, data notwithstanding, either throwing puppies out of skyscrapers or throwing people off Medicaid are both acts of sadism…

We know that Medicaid makes people happier and less poor. We have trouble proving its impact on their physical well-being because proof of the benefits of medicine remain elusive. Unless we want to stop thinking of basic medical care as a life necessity, and we don’t, the case for Medicaid remains unimpeachable.

Meanwhile, Drum and the Incidental Economist take an important look at why Medicaid did not seem to improve outcomes.  Drum:

In fact, the study showed fairly substantial improvements in the percentage of patients with depression, high blood pressure, high cholesterol, and high glycated hemoglobin levels (a marker of diabetes). The problem is that the sample size of the study was fairly small, so the results weren’t statistically significant at the 95 percent level.

However, that is far, far different from saying that Medicaid coverage had no effect. It’s true that we can’t say with high confidence that it had an effect, but the most likely result is that it did indeed have an effect. The table below shows the point estimates. Note also that in all cases, the use of prescribed medication went up, in some cases by a lot.

Here’s the thing, if you are finding 17 and 18% differences but they are not statistically significant at the p<.05 level, it means you just did not have enough statistical power to properly test your hypotheses.  Now, there’s nothing that can be done about that in this case– there were just only so many people– but it does mean that you need to think about this in full context and realize, exactly as Drum suggests, that in all likelihoood Medicaid did improve health outcomes, we’re just not 95% confident.  Kind of like if you gave one treatment to 10 people and 8 improved but only 5 improved in the control group.  Yeah, the treatment probably worked, but there’s just not enough people to give you statistical confidence.  Now, change those numbers to 800 of 1000 and 500 of 1000 and you can be pretty damn sure.  There just weren’t enough people in this study to be pretty damn sure, but in context, the results certainly are suggestive.  And as Chait and Cohn point out, even if there was not any physical improvement (in a narrow two year period), it sure means something to not go bankrupt and to have better mental health.

About Steve Greene
Professor of Political Science at NC State

4 Responses to Medicaid, statistical power, and health vs. wealth

  1. Mark says:

    I guess that’s true, but what is the point of statistical tests if we are just going to ignore the confidence intervals? Your example extending improvement in 8 out 10 people in the treatment group compared to 5 out of 10 in the control group out to a sample of 1000 doesn’t make sense. Sure, we might see that result. But the point is that the difference between 8 and 5 is not meaningful at all because it could easily be random. You can’t assume it will hold with a larger sample, which is what the confidence interval is telling us. So sure the point estimate is negative, but if we repeated this study, it could just as easily be smaller or even positive.

    You simply cannot look at point estimate (-2.43) and claim a 17% drop because that estimate means nothing. It could be anywhere from -8 to 3, which is a huge range. 95% is not an overly onerous statistical bar to clear. Even if we dropped it down to 90%, that probably would not be statistically significant.

    I read and like most of Kevin Drum has to say, but he’s way off here. There is no world in which the “most likely” likely result is an effect. This is why we use statistics to analyze data and have confidence intervals. You can’t just ignore them when you don’t like the results. This is science, not wishful thinking. “Most likely” means nothing in this context.

    That being said, this study does show the value of expanding Medicaid. It clearly helps people, and some health care is better than none.

    • Steve Greene says:

      No, Drum gets this. I don’t know the p values, but if p=.11, it is still fair to say something is most likely, even if you cannot say “statistically significant.” And, I know you can’t say anything meaningful with 5 and 8 out of 10, but you can with 500 and 800 out of 1000, that’s my point.

      • Mark says:

        That’s true for a p-value of .11, but none of them for the numbers he highlights are that low. For one of the estimates, the p-value is 0.61. You can’t plausibly or honestly look at that p-value and confidence interval and say that represents a drop of 18%. It’s entirely random. The “drop” he highlights isn’t a drop at all; it’s 0. I’d be more likely to accept that if the values just barely didn’t reach significance but that’s not the case.

        My overall point is that his comments focus on the wrong thing. Why is he trying to make an argument that these specific differences are real when is there so much else in the report that points to the benefits of Medicaid? It might come through in terms of physical health outcomes, but it does in other places, which you point out. He should be making that argument instead of trying to claim the differences he highlights matter when the evidence shows they don’t.

      • Mark says:

        I meant it might NOT come through in terms of physical health outcomes. Stupid typos.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: