A Case Study in Bias

Two studies came out comparing the performance of schools or teachers. In the first case, Raj Chetty, John Friedman, and Jonah Rockoff came up with just about the most extensive and sophisticated study of teachers' value-added that I've ever seen. As highlighted in the New York Times, the study includes estimates for how much high-quality teachers improve their students' income years later, and also (see pp. 29 ff.) includes a new way to check for bias by looking at how cohorts of students change performance when a high or low value-added teacher arrives from somewhere else. Very cool.

But such a study, implying that some teachers are better than others, and that teacher quality can be revealed by how well their students do on tests (conditioning on prior achievement and student demographics), is disturbing to some people. Diane Ravitch tweeted at least 67 times the day the study came out, trying to undermine the study by questioning its lack of peer review (so far), the way in which it was conducted, and the very project of looking at test scores in the first place.

 In the second case, there's a group called Educate Now in Louisiana that released a PDF chart (available here) that merely lists the schools in New Orleans identified by whether they are Recovery School District schools or voucher-accepting private schools, and then listing what percentage of students score above basic on English and Math in grades 3-5. That's all. No attempt to control for the individual students' prior achievement, no attempt to control for any student demographic variables such as poverty, no attempt to control for the fact that students are eligible for vouchers only if they had been attending a failing public school, no statistical analysis whatsoever.

This is as primitive as it gets, and is a horrible way to judge the merit of voucher schools (as I explained here).

 Did Diane Ravitch tweet 67 times criticizing this purported attempt to compare voucher schools to public schools? No: right in the midst of her incessant criticism of an immeasurably superior study, she sent out one tweet that said, "How did voucher schools in New Orleans do?" followed by a link.

Ravitch here displays the worst sort of intellectual bias: when what looks like one of the best studies out there doesn't fit her ideology, she acts as if it is far more questionable than the baloney that she otherwise is happy to plug. To be sure, it's OK to ask questions about the new value-added study, what it means, how it was done, and whether it was oversold in the media. But it's not OK to pass along a worthless analysis of the merits of vouchers.

Anti-reformers need to think a bit more objectively about whether they want someone like this as their standard-bearer. It only does them discredit to treat Ravitch as any sort of objective scholar.

"Starch Consumption Raises Risk Of Breast Cancer Coming Back"

So read the headline on a news story recently. A new study found as follows:

Breast cancer survivors whose starch intake is above average have a greater risk of cancer recurrence compared to other breast cancer survivors, researchers from the University of California, San Diego explained at the 2011 CTRC-AACR San Antonio Breast Cancer Symposium, Dec. 6-10, 2011. The researchers added that it is in particular starch that raises the risk, and not just overall carbohydrates.

Plausible. But it turned out that all the researchers did was interview the women once a year about their diet, and women whose breast cancer came back were eating 2.3 GRAMS of carbohydrates more than the average per day, only half of which was starch. As far as I can tell, that's about as much as is in 1 tablespoon of cooked rice. So the headline was based on women claiming to have eaten an additional 1 tablespoon of cooked rice (or some equivalent) per day.

This does not strike me as a useful finding. There is no way that a once-a-year interview can pinpoint women’s carbohydrate consumption down to the tablespoon, and such a miniscule amount of starch surely can't be making a clinical difference anyway.

Charter Schools and Segregation

Charter schools are often accused of "segregation" merely for serving too many black kids. One recent example of this criticism comes from Zoe Burkholder of Montclair State University in New Jersey, who has an article in Teachers College Record lamenting the fact that DC Prep Charter School is 98% black.

She does concede that black parents have a good reason to choose DC Prep: "parents in D.C. can choose between a traditional public school racked with violence and high dropout rates, or a charter school that is safe and promises to teach at least two of the '3 Rs.'" She even admits that "maybe anyone would prefer a charter school like DC Prep under these conditions."

But she immediately backs away from agreeing that black parents ought to have the option of choosing such a school:

But that doesn’t make it okay, and here is why. When you step back from DC Prep, and successful charter schools like it, what you see is a public school that is racially and socio-economically segregated and inherently very different from the form and function of the majority of public schools in America. . . . Since Horace Mann first rode horseback through New England to sell the idea of tax-supported “common schools” for all children, Americans have dared to dream that public education will instill in our citizenry the many capacities necessary for self-government: critical thinking, civic engagement, tolerance for diversity, an appreciation for the arts and sciences, a knowledge of global affairs, a critical understanding of American history, and the capacity for civil debate.

I've said this about Diane Ravitch before: If you're going to oppose the so-called "segregation" of charter schools, even though it arises from the completely voluntary choices of black parents, you should think twice before waxing so eloquent about Horace Mann's day, when it was often illegal for black people to attend school anywhere. Nor is it historically correct that "Americans" wanted "tolerance of diversity" in public schools during the 100+ years of officially-mandated segregation.

In any event, Burkholder makes the same mistake that the highly publicized Civil Rights Project (headed by Gary Orfield) made: she compares DC Prep Charter School to "the majority of public schools in America."

That comparison is completely meaningless. We know that charter schools are much more likely to be located in inner-city neighborhoods where the demographics are much different from the national average. Indeed, if an inner-city DC or Atlanta charter school had demographics that resembled the broader United States, that school would instantly be accused of promoting segregation by gathering too many white students in one place.

What Burkholder should have done is compare DC Prep to nearby traditional public schools. On that ground, it turns out that a 98% black charter school in a heavily black area of northeastern D.C. isn't that unusual. The closest traditional public school to DC Prep is Noyes Elementary, which is 96% black, 3% Hispanic, and all of zero percent white.

Yes, racial imbalance still exists. But attacking charter schools does nothing to get rid of it.

Momentum in Sports

The Freakonomics blog makes a point that I think is wrong:

The best place to start is with a famous (for academia) paper from several years ago, called “The Hot Hand in Basketball: On the Misperception of Random Sequences.” As you can glean from that snazzy subtitle, the authors come down against momentum, arguing that a “hot streak” is really just a random sequence that we misperceive to be more meaningful than it is.

Ever try flipping a coin 100 times? You’ll be surprised at how many long, unbroken sequences of heads or tails you get. It’s easy to mistake that for a pattern, suggesting some kind of meaning or momentum, but it’s really just a pure illustration of randomness itself. The fact is that if you get 10 heads in a row, the next flip is no more likely to be heads (or tails, for that matter).

And so it is, for the most part, with hot hands and hot streaks and hot quarterbacks. In our Momentum video, you’ll hear Toby Moscowitz, the academic co-author of Scorecasting, discuss how pretty much everyone in football believes in momentum. But, having looked at a lot of NFL data, Moscowitz reaches a sobering conclusion: “There is a much stronger belief in momentum than is warranted by what we see in the data.”
* * *

Consider one example in our video, the Buffalo Bills’ redonkulous 32-point comeback against the Houston Oilers in 1993. As Chris “Mad Dog” Russo puts it: “You’re gonna tell me momentum had nothing to do with that game?!”

Okay, Chris, I’ll take a shot at telling you exactly that. You know why we’re still talking about that game? Because it was a massive anomaly – the kind of comeback that almost never happens. It was so rare that our brains have an easy time recalling it. (We do this with all anomalies – dramatic plane crashes, mass murders, and so on.) And when we recall something so easily, we tend to believe it’s far more common that it actually is.

The truth is that you’re bound to get a wild 32-point-comeback once in a while, just as you’re bound to get a streak of 10 or 12 heads too.

Here's the thing: a 32-point comeback might indeed be so rare that it fits within a statistically normal distribution as several standard deviations above the mean. This does NOT mean, however, that a 32-point comeback was itself a matter of random chance -- the 32-point comeback happened because of how a bunch of human beings performed on a given day, and their performance was not random at all. Their performance was affected non-randomly by their preparation and skills, their coaching, their choices of plays, and their confidence level (the latter of which would be dramatically affected if either team started to think that the "momentum" was heading in a particular direction).

Try an analogy: A 7-foot-tall man is a rarity, and if human height falls into a normal distribution, someone might make the following claim, akin to the dismissal of sports momentum: "This 7-foot-tall man's height might seem to have sprung from some genetic factor, but in fact, you find 7-foot-tall men in nature only as often as would be expected by chance. Therefore his height is just a matter of random chance, not genetics."

Well, the fact that this particular guy got the genes to be 7 feet tall might be random chance from the point of view of a statistician looking at all of humanity, but that in no way proves that his height was unrelated to genes. Similarly, the fact that one particular sports team had a huge amount of momentum on a particular day might be described as random chance, but that doesn't disprove the claim that it did have momentum then.

Songs I Like

UPDATE: Here's a Spotify list of most of the songs below.

Deas Vail, “Desire.”


The National, “Exile, Vilify.”


Eisley, "One Day I Slowly Floated Away."


Eisley, "Memories."


Deas Vail, "Shoreline."


Jeff Buckley, "Corpus Christi Carol."


Copeland, “Should You Return.”


With Lions, “Our Great Rise.”


The Future of Forestry, “If You Find Her.”


Jonsi, “Tornado.”


Gotye with Kimbra, “Somebody That I Used to Know.”


The Honey Trees, “To Be With You.”


Eric Whitacre, “Lux Arumque.”


Espen Lind, “Scared of Heights.”


Kimbra, “Settle Down.”


The Reign of Kindo, “The Moments In Between.”


Peter Groenwald, “Wreckage.”


Yael Naim, “New Soul.”


Bat for Lashes, “Sleep Alone.”


Civil Twilight, “Human.”


Digital Daggers, “Surrender.”


Shiny Toy Guns, "You Are the One."


Dubstar, "Stars."


Muse, Undisclosed Desires.

Erin McCarley, "Pitter-Pat."


Deadmau5 and Kaskade, “I Remember.”


Marc Martel, "Somebody to Love."


Dredg, "Information."


Eden's Edge, "Blue Moon of Kentucky."


Lo-Pro, "Reach."

George Michael, "I Can't Make You Love Me."


And just for fun: George Michael, “1, 2, 3.”

What Causes Student Achievement?

Dana Goldstein addresses that question:

As you can see, by estimating teacher effects at 20 percent, I've interpreted the research consensus quite generously. Matthew DiCarlo, a sociologist with the Shanker Institute, has looked at this same body of research and concluded that another 20 percent of the causes of student achievement gaps are "unobservable" (ex; differences in innate intelligence, statistical error, other mystery causes); and that the rest, about 60 percent, can likely be explained by all the myriad factors associated with socioeconomic status.

The parsing out of what causes student achievement seems very dubious. What if part of the way that socioeconomic status leads to higher achievement is that parents use it to buy houses in school zones with . . . better teachers? Seems very likely, but there's no way to tell with the usual models.

One way we could figure out how to divvy up responsibility would be to get 500 rich kids and randomly assign half to attend a school with teachers identified as horrible (but otherwise keeping everything else about the school the same, such as peers or school spending), and then compare them to the other rich kids who got to attend their regular school. Then you'd really be able to see how much rich kids were benefiting from being able to buy access to good teachers.

But you'd never be able to do such a study -- no one would sign up.

Memory and the Law

All of the studies on the fallibility of memory should make us question the weight given to eyewitness testimony.

Good collection of essays

From 52 prominent economists:

This is a compendium of fifty-four papers written by distinguished economists in response to an invitation by the National Science Foundation's Directorate for the Social, Behavioral and Economic Sciences (NSF/SBE) to economists and relevant research communities in August 2010 to write white papers that describe grand challenge questions in their sciences that transcend near-term funding cycles and are "likely to drive next generation research in the social, behavioral, and economic sciences." These papers offer a number of exciting and at times provocative ideas about future research agendas in economics. The papers could also generate compelling ideas for infrastructure projects, new methodologies and important research topics.

Contributors include James Heckman, Daron Acemoglu,
David Cutler, Esther Duflo, and Hal Varian.

Addicted to Running

Someone who qualifies: "Woman Completes Chicago Marathon and Delivers Baby."

Interesting article on Steve Jobs

Here.