What Scientists Should Learn From Economists

All those other enterprises, though, seem to have come to terms with the fact that there are going to be mis-steps along the way, while scientists continue to bemoan every little thing that goes awry. And keep in mind, this is true of fields where mistakes are vastly more consequential than in cosmology. We’re only a week or so into July, so you can still hear echos of chatter about the various economic reports that come out in late June– quarterly growth numbers, mid-year financial statements, the monthly unemployment report. These are released, and for a few days suck up all the oxygen in discussion of politics and policy, often driving dramatic calls for change in one direction or another.

But here’s the most important thing about those reports: They’re all wrong.

Chad makes an excellent point, but if I’m reading the post correctly it’s an admonition toward scientists, and I think that’s misplaced, or at least too narrow a focus. As a group, I think we have a decent handle on the difference between the levels of confidence one places in results at different stages of confirmation. Many scientists I follow on twitter were saying we need to be cautious about the BICEP2 results, and how we needed to wait for further analysis and confirmation — that’s the protocol, and it needs to be more widely acknowledged.

What’s missing is in the restraint of the media chain, which often includes the principal scientists; one should understand that they and the attached PR machine may tend to be a little aggressive in touting their results, and may have a bias to which they are blind; it’s why replication of experiments is important. However, everyone else involved has to slow down a little and consider the shortcomings of the system as well.

Is this a preliminary result/small sample size, or is this further down the line in terms of confirming the original discovery? (I’m assuming we’re over the hurdle of this being peer reviewed). If it’s early in the game, then these are much like the preliminary economic numbers Chad discusses — there will be revisions, and that needs to be explained. More data require more experiments, preferably by different research teams. Results have a way of disappearing when more data are examined — which is exactly what you should expect! But this doesn’t get much prominent discussion when BIG RESULT™ has been announced.

In the case of economic reporting, the public has been seeing this same style of reporting for decades — they’re used to it. They expect a certain level of wrongness from the folks who have predicted twelve of the last five recessions. What they’re used to in science reporting is a hyperbolic headline and the promise that it will result in a flying car really soon (and then, of course, the flying car never materializes) being reported in the same fashion as science that has a much longer pedigree of confirmation.

Scientists need to do better in getting the word out properly, to be sure. But my feeling is that the entire system needs to be reined in.