Daniel over on Crooked Timber takes a
whack at Steven Milloy's
critique of the soft drink study that was discussed last week. (Sorry, I had my first week of classes this week, so I'm just getting back to all this now). Obviously, Steven Milloy can defend himself if he wants, but I figured a few points of elaboration on
my original post regarding his essay are warranted. For the record, I don't know Milloy or have any other contact with him except for having read the essay in question.
The testable hypothesis in the study is the null hypothesis that an increase in soft drink consumption like that in the study (less than one per week to one or more per day) will dramatically increase the risk of weight gain and type 2 diabetes. Milloy's piece critiques the study in two ways. First, it argues that there were major methodological problems with the study that undermine its reliability, and that once corrected for, many of the findings of the study are much more questionable than touted. Second, that whatever the findings of the study, the authors have oversold their conclusions, in that the conclusions that they draw are not supportable by what the study actually found. I characterized the overall critique as "devastating," a characterization with which Crooked Timber took issue.
First, Crooked Timber states that Milloy is "a bit of a hack, who got his start with a bit part towards the end of the single largest and most impressive work of intelelctual dishonesty of the previous century, the effort to discredit the scientific work on the linke between tobacco and lung cancer."
Others open their responses to Milloy in a similar fashion.
Now, Milloy's article is a critique--he is saying that the authors of the soft drink study have failed to carry the burden of rebutting the null hypothesis that soft drink consumption causes diabetes. I have no idea why Milloy's position on smoking would make the JAMA article on soft drinks more or less persuasive. Either it is methodologically correct or it isn't--the article stands or falls on its own merits. Any defects that Milloy may or may not have does not increase or decrease the JAMA article a whit.
So the real issue is the merits of the critique. First, Milloy says that the once the researchers "statistically adjusted their results for bodyweight (a risk factor for dabetes) and for caloric intake (a proxy measure of consumption of sweetened foods other than soda), the 83 percent increase [in type 2 diabetes prevalence] dropped to an even more statistically dubious (and soft-pedaled) 32 percent increase." Now it seems to me that Milloy is obviously correct here--bodyweight and non-soda caloric intake seem to me to obviously relevant to trying to isolate the marginal effect of the increased soda consumption. So the 83 percent figure is really an irrelevant number--nonetheless, the
Washington Post reported on page 1, in the second paragraph of the article, that those who drank more than one serving a day "had more than an 80 percent increased risk of developing Type 2 diabetes" than those who had less than one a month. Then a few days later the Post stated in an
editorial, "those who had one or more drinks containing sugar or corn syrup per day were 83 percent more likely to develop Type 2 diabetes than those who drank less than one such drink per month."
But, the reader might object, the researchers cannot be responsible for how the Washington Post characterizes their research. Well, it turns out that the Post story was lifted directly from the
press release from the Harvard School of Public Health, which states in paragraph 2, "Those who reported drinking sugar-sweetened sodas more than once per day showed an increased risk for type 2 diabetes of more than 80 percent compared to women in the study who drank less than one per month...." the 32 percent figure, by contrast, appears nowhere in the press release.
Ok, so assume for the moment that Milloy is correct that there is a 32 percent increase in the prevalence of type 2 diabetes in the subject population. Here's the excerpt from Milloy that Daniel quotes, "When the researchers statistically adjusted their results for bodyweight (a risk factor for diabetes) and for caloric intake (a proxy measure for consumption of sweetened foods other than soda), the 83 percent increase dropped to an even more statistically dubious (and soft-pedaled) 32 percent increase. That result is of the same magnitude as the study's reported 21 percent increase in diabetes among consumers of more than one diet soft drink per day."
This is where Daniel gets to the heart of his critique, writing about this passage: "Think about this for a second. According to Milloy, the correct (even the ethical) thing to have done in presenting the results of this study would have been to have headlined the '32 per cent increase' (1.32 relative risk) that one gets in a model which controls for body mass and caloric intake. This is equivalent to suggesting that the correct way to think about the health risks associated with soft drinks is to deal with a model under which somebody goes from drinking one can of Coke a week to more than one per day, but reduces their consumption of other foods so as to maintain a constant total caloric intake. Given that the entire reason why people worry about soft drink consumption is the sugar in the drinks, does this make any sense at all?"
Perhaps I have misread Milloy's point, but I don't read the claim at all the same way as Daniel does. Now what I understand Milloy to be saying is that those who drank more than one REGULAR soft drink per day had a 32% increase in diabetes prevalence, whereas those who drank more than one DIET soft drink per day had a 21% increase. The point, as I understood it, is that there was a noticeable increase in diabetes prevalence REGARDLESS of whether a person drank alot of regular or diet soda. Moreover, there is little difference in the increased prevalence between the two different types (its not clear to me if the 32% versus 21% is a statistically significant difference--Milloy's point seems to be the more colloquial one that the difference is not very large). Milloy's conclusion, therefore, is that the study has not shown that it is the sugar in the regular soft drinks that is causing the increased diabetes prevalence, because there is an increased diabetes prevalence from soft drink consumption regardless of whether it is sugared or non-sugared. If so, Milloy states that this suggests that some other independent factor is to blame, such as lifestyle or genetics, such that those who drink a lot of soda (regardless of which type) to be prone to contracting type 2 diabetes. A hypothesis would be something like those who drink alot of soda don't exercise as much or don't eat as healthy. I have no idea whether this is true or not, but if Milloy is correct then it suggests that the study misses the point by focusing on sugared drinks. It seems to me this is also why Milloy says that it matters what people's other caloric consumption is--if soda consumption is corellated with and is a proxy for other lifestyle habits, we would want to know what people's other caloric consumption (note that the reference point is non-soft drink calorie consumption).
Now that's how I understand the Milloy critique, which seems quite powerful to me if correct. To be honest--and I'm not being rhetorical here--I honestly can't make sense out of how Daniel is characterizing how he understands the claim made by Milloy here. I understand the point of Milloy's critique to be the comparison between regular and diet soda consumption and what that says about the role of sugar in the study. Daniel interprets Milloy as making some othe point about increasing soda consumption and decreasing other foods and the proper statement of the diabetes prevalence, and I don't read Milloy that way. Daniel says that the model (as he summarizes it) is a "stupid model." If Daniel is correct in the way he characterizes Milloy's point, then as I noted, it doesn't really make sense to me either. But it seems to me that what Milloy is intending to say, and actually says as I read it, is not actually stupid. As a general practice, if we assume that an author is trying to make a serious claim, it seems to me that a more logical intellectual policy is to adopt the reading that construes Milloy's claim in a manner that makes sense (leaving aside whether it is right or wrong), rather than to construct it in a manner that does not make sense or is "stupid." In other words, given the choice between construing a claim as a straw man versus a serious claim, it is a better practice to adopt the latter reading on the assumption that is what the author intended.
In addition, other credible commentators have made critiques of the study that are similar to Milloy's. A
story on a Bay Area TV website referred to the comments of Karmeen Kulkarni of the American Diabetes Association who said that similar results might be found if researchers studied another food with little nutritional value, such as chips, cakes or cookies. Kulkarni, like Milloy, stressed that lifestyle factors are critically important in determining the effect of soft drink consumption. The story states, "She [Kulkarni] said women in the study who drink more sugary beverages tended to live a less healthy lifestyle — smoking more, working out less, eating more calories and less fiber and protein. The research also relied on the women to write down what they ate, making it less reliable."
Milloy similarly notes that the study does not control for genetics or lifestyle issues--although he goes on to add that the "real explanation for the reported weight gain more likely lies in the women's geneteics and their overall lifestyles." Now, I don't know whether that last part is true or not--but then again, all I have ever claimed is that Milloy's critique of the weaknesses in the study and its conclusions are very strong. I have never said that in this piece Milloy offers a compelling alternative hypothesis or support for an alternative hypothesis. But it seems obvious to me that lifestyle matters alot (just as Kulkarni observes) and that it would be pretty easy to control for that. It is less obvious to me how much genetics matter (could be a little or a lot) and it might be much more difficult to control for genetics. Nonetheless, it is obvious that these factors should be controlled for.
The quality of this study stands or falls on its own merits. I think that Milloy has identified some very serious methodological flaws in the construction of this study and the conclusions that are drawn from it. The authors of the study appear to be promoting the most expansive and least-supportable interpretation of the study and seem to have done little to address serious alternative interpretations of the data reported in the study. What Milloy may have said about other issues such as smoking, or what his own hypothesis is on this issue, is largely beside the point for purposes of my assessment that this remains a very serious critique of the study.
Update:
To clarify a potential ambiguity, I meant to say that it is the JAMA article stands or falls on its own merits (ther original post inadvertently just said "the article" at one point, which might be read to refer to Milloy's article).