Chad has a post up about peer-review in the context of the BICEP2 results about inflation: Review and Replication

What ultimately matters, after all is not review but replication– that is, when somebody else tries to do the same experiment, do they get the same results? This is where issues with statistical flukes usually sort themselves out, and that’s not a problem that can be fixed by any amount of refereeing. A journal referee can look for obvious gaps in the analysis, but will never get to the level of detail of a repeat experiment or a follow-up measurement.

This has some overlap with something I wrote a few weeks back (Trust Me, Maybe?) wherein I argued that peer review is a demarcation where you can start taking claims seriously, but I realize I was thinking more about theory awaiting confirmation rather than experiment. Chad’s point that there are experimental efforts where peer review will be a formality of sorts, because we already know the experiment was carefully done, is correct. What’s important here is replication.

This reminds me of a description I recently saw (but I don’t know its origin): Peer review is a spam filter. In a case like BICEP, we’re already pretty sure it’s not spam.