Category Archives: Science-general

What We've Got Here is Failure to Communicate

Posted on by

That’s why they play the game

That claim of “statistical” predestination is confused.* We know the odds of a coin-toss or of the roll of a die before we ever pick them up. Those things are set, predetermined. But it was not preordained by the universe that Shaquille O’Neal would be a .527 free-throw shooter for his career. It’s weirdly mystical to speak of statistics that measure the outcome of an athlete’s performance over time as though they were that athlete’s “scientific” destiny.

And it’s ironic that such mysticism comes from those desperate to dismiss talk of being “in the zone” because they see that as uncomfortably mystical-sounding.

This is the second commentary of this sort I’ve seen over the weekend; unfortunately I can’t recall the first one or the article that precipitated it, but this one will suffice.

The sentiment is correct as far as it goes: anyone who is saying that statistics pre-ordain performance is wrong. But that’s a straw-man for the scientific discussion — it’s not the actual argument. I think that it’s obvious that there are parameters that affect one’s performance when one goes out sportsing. The analysis going into the phenomenon of “the streak” or being “in the zone” is not claiming otherwise, and there’s a fundamental misunderstanding of what the analysis is claiming that is in play here.

As an aside, I think it’s a similar misunderstanding that has developed with the so-called 10,000 hour rule: that you’ll become an expert when and only if you put 10,000 hours of work into your craft. Which is not to say that you’ll become world-class if you do so, but I’ve seen articles where it’s implied that the only thing standing between you and the pro sportsing circuit is the required investment of time. Bollocks. There’s innate talent for the craft as well, and the quality of your preparation, and perhaps more to it. A similar straw man has been constructed over this statistical analysis, via the retelling in a version of the “whisper game”. Purple monkey dishwasher.

The basic premise is that athletes get into “the zone” where the ball seems bigger and/or slower or perhaps the opposite, depending on what sport is being played, and the athlete does really well. Great. But we’re scientists, and we want a model of this, and the model here (for basketball, where the original study was done, by Gilo, Vallone and Tversky) is that when a player hits several shots in a row — they are “hot” — their odds of hitting shots is higher. That is, a player who hits e.g. eight shots in a row, does so because s/he is on a streak, and thus a higher shooting percentage is expected. In other words, the streaks should be deviations from a normal distribution.

The statistics, however, say otherwise. The streaks are completely consistent with a normal distribution — hitting those eight in a row is an expected consequence of some underlying probability of success and a large number of attempts. The model, which predicted a deviation, is wrong. There is no evidence of streaks.

The model isn’t rejected because of mysticism. It’s rejected because the prediction it makes is not observed, and if a model disagrees with the experiment, it’s wrong. Here’s where the subtlety comes in. What the results don’t say is that there is no such thing as “being hot”. It says that this model of streaks is wrong.

One also has to look at what wasn’t measured. The study was only an analysis of hitting shots based on making the previous shot (which is very much like the Gambler’s fallacy) but no analysis of a player that goes e.g. 16-for-20, or any other effects that might make the results end up being random. Far from being mysticism, these results just tell is where we might look next and where not to look next. However, we also know that any working model will have to give results that are consistent with the randomness we observe. We have a tendency to see patterns in randomness. This might have the same result as a lucky pair of socks — no real effect, but plenty of confirmation bias and apophenia.

The issue isn’t whether Shaq was somehow “predestined” by his 52.7% free-throw accuracy. But that number isn’t the real issue — the reality is that Shaq was a mediocre free-throw shooter, and the number reflects that. The question is whether Shaq performed differently from a player whose talent and preparation made him a free-throw shooter with ~50% chance of hitting each shot, and that answer (thus far) is no.

Physics and Mermaids

Posted on by

One of the things I was thinking about this past week, in between talks (or possibly during) at ICAP was the connection with the “impossible” drive that was arguably not actually validated by NASA in the news recently. I think the seed of this was planted during a talk about trying to measure the electron’s electric dipole moment (EDM), in order to rule out some of the extensions to the Standard Model (SM). There’s no connection in the physics, but it’s the concept of ruling out certain measurements that struck me.

You might hear the phrase that absence of evidence is not evidence of absence, and that’s true, as far as it goes. If scientists have not measured the strength of gravity a million km above the sun’s north pole, for example, it would not mean that there is no gravity there. But the aphorism doesn’t work when you have done testing and can reasonably expect to get a result if some model is true. Then your absence of evidence really isn’t an absence — you’ve measured something, and gotten a null result or a small result, which rules out a larger value.

Let’s say you wanted to make a determination of the existence of mermaids. This being a physics analogy, it wouldn’t be enough for a straight up-or-down statement of their existence — you’d have a model of the conditions under which they’d be found. Someone else might have a competing model, saying they existed, but under somewhat different circumstances. Then we could go out and search for the mermaids. We search the right kind of islands at the right time of day, and find nothing. Repeat as necessary, because statistics. That’s not going to absolutely rule out the existence of mermaids, but it puts a limit on how many mermaids are statistically likely to be out there. Depending on the conditions under which we searched, it might place stricter limits on one model over another — if another model said that mermaids existed in a somewhat different environment, our search of that “space” might not have been as thorough.

There is (not unsurprisingly) an xkcd cartoon related to this

We rule out phenomena, at increasingly better levels of confidence, the longer we properly observe and don’t see anything. (“Properly” because looking with blinders on, or the lens cap in place, doesn’t count. e.g. creationists will never find “transitional” fossils because they refuse to look.) Scrutiny in duration and/or in precision, yielding null results, pushes the limits back of where any new discovery might be.

We can also see this if we go back to the days of physics before quantum mechanics, over a hundred years ago. Albert Michelson had remarked, in 1894, “The more important fundamental laws and facts of physical science have all been discovered, and these are now so firmly established that the possibility of their ever being supplanted in consequence of new discoveries is exceedingly remote…Our future discoveries must be looked for in the sixth place of decimals.” Lord Kelvin had (supposedly) announced in 1900 that “There is nothing new to be discovered in physics now; All that remains is more and more precise measurement.”

The sentiment was wrong, of course: there was new physics lurking. But one part of this was correct: that new physics was lurking in the “sixth place of decimals” or beyond. One or two anomalies aside, the new physics wasn’t found where we had already looked — that’s physics we still use to this day, in the realm of what we typically observe — it was found as the tools got better.

Which is another reason why the “impossible” drive draws so much scrutiny. We’ve been down this road before, many times and not seen anything, which is why a claim that something is there (and was there all along) is met with so much skepticism. This doesn’t say that there can be new physics. What it says is that any new physics is going exceedingly likely to be found in the uncharted waters. Mermaid sightings were claimed in remote places, not the local beach.

You Can't Beet This. It's 24-Carrot Gold.

Posted on by

Lettuce See the Future: Japanese Farmer Builds High-Tech Indoor Veggie Factory

Shimamura turned a former Sony Corporation semiconductor factory into the world’s largest indoor farm illuminated by LEDs. The special LED fixtures were developed by GE and emit light at wavelengths optimal for plant growth.
…
By controlling temperature, humidity and irrigation, the farm can also cut its water usage to just 1 percent of the amount needed by outdoor fields.

Economics and Science?

Posted on by

What Scientists Should Learn From Economists

All those other enterprises, though, seem to have come to terms with the fact that there are going to be mis-steps along the way, while scientists continue to bemoan every little thing that goes awry. And keep in mind, this is true of fields where mistakes are vastly more consequential than in cosmology. We’re only a week or so into July, so you can still hear echos of chatter about the various economic reports that come out in late June– quarterly growth numbers, mid-year financial statements, the monthly unemployment report. These are released, and for a few days suck up all the oxygen in discussion of politics and policy, often driving dramatic calls for change in one direction or another.

But here’s the most important thing about those reports: They’re all wrong.

Chad makes an excellent point, but if I’m reading the post correctly it’s an admonition toward scientists, and I think that’s misplaced, or at least too narrow a focus. As a group, I think we have a decent handle on the difference between the levels of confidence one places in results at different stages of confirmation. Many scientists I follow on twitter were saying we need to be cautious about the BICEP2 results, and how we needed to wait for further analysis and confirmation — that’s the protocol, and it needs to be more widely acknowledged.

What’s missing is in the restraint of the media chain, which often includes the principal scientists; one should understand that they and the attached PR machine may tend to be a little aggressive in touting their results, and may have a bias to which they are blind; it’s why replication of experiments is important. However, everyone else involved has to slow down a little and consider the shortcomings of the system as well.

Is this a preliminary result/small sample size, or is this further down the line in terms of confirming the original discovery? (I’m assuming we’re over the hurdle of this being peer reviewed). If it’s early in the game, then these are much like the preliminary economic numbers Chad discusses — there will be revisions, and that needs to be explained. More data require more experiments, preferably by different research teams. Results have a way of disappearing when more data are examined — which is exactly what you should expect! But this doesn’t get much prominent discussion when BIG RESULT™ has been announced.

In the case of economic reporting, the public has been seeing this same style of reporting for decades — they’re used to it. They expect a certain level of wrongness from the folks who have predicted twelve of the last five recessions. What they’re used to in science reporting is a hyperbolic headline and the promise that it will result in a flying car really soon (and then, of course, the flying car never materializes) being reported in the same fashion as science that has a much longer pedigree of confirmation.

Scientists need to do better in getting the word out properly, to be sure. But my feeling is that the entire system needs to be reined in.

A Million Prescient Monkeys

Posted on by

A History of Books that Forecast the Future

As interesting as this is, it’s also an example of selection bias. Also: 2013 is the year for government spying on individuals, like this wasn’t happening earlier? really? But I digress…

Lots of stories appear to make predictions of the future, but are they really predictions or just fanciful things thought up by the author? What sci-fi devices haven’t come to pass? (How many have flying cars or superluminal travel of some sort, etc.?) That’s context that’s missing, because looking only at successful predictions (more on that in a moment) is the wrong way to look at it. If the author is truly a visionary maker of predictions, s/he has to be right more often than chance. It’s tough to measure that in an open-ended medium like storytelling, but one could at least do a systematic measure of it. Regardless, with myriad predictions, some are bound to be right. So what’s the success rate?

Also, how do you define success? For predictions that are vague it’s much easier to argue that it was successful, but of course vague predictions are next to useless precisely because they are vague. This is one element of how so-called psychics and their ilk make their livings – be vague enough that you can throw up your hands and declare success no matter what happens. I’m not familiar enough with the stories to know how much leeway the authors are being given.

The next step and the real trick — much harder IMO — is if the author was able to capture how society exploited the technology.