Category Archives: Science-general

Are You a Good Witch or a Bad Witch?

Posted on by

Star dust casts doubt on recent big bang wave result

[O]ther things apart from gravitational waves, such as dust, can emit polarised photons.

To minimise the chances of this effect causing a false signal, the BICEP 2 team pointed their telescope at a patch of sky far away from the Milky Way’s dusty disc. Then they used models of the dust in that part of the sky to estimate its effect on the polarisation. They found that this could account for no more than about 20 per cent of the signal that they attributed to gravitational waves last month.

But Mertsch says the models they used didn’t account for dust shells produced as the expanding remnants of supernovae slam into surrounding gas and dust.

So the results may not hold up. This is bad, right? We get all worked up about a result, and it turns out it might not be correct.

Except it’s not. Procedurally, this is good. This is exactly the way science is supposed to work. You do your best attempt at the research and looking for confounding effects that might be giving you a false signal. You have it peer reviewed, and you publish. As I’ve talked about before, that’s just the first hurdle. After that comes feedback from other scientists, including attempts at replication. And that’s where we are now — some attempts at replication are already ongoing, and here we have someone who has new information that might affect the conclusion. This is how science moves to get things right.

The bad part is that the first results get hyped, because they’re new, and there’s no restraint for waiting for the weight of evidence to pile up. Sometimes the first results won’t stand up to scrutiny. I recognize that this might weaken the confidence some have in science, but in reality it should strengthen it: other scientists are willing (enthusiastic, even) to stand up and say, “Wait a minute!” when all the ducks don’t line up. There’s no conspiracy to conform. Scientists questioning loudly trumpeted results like this gives me confidence that all of science is subject to the same kind of feedback.

Should You or Shouldn't You?

Posted on by

Should you get your PhD? (in science)

I added the parenthetical in science because I’m not sure how well the advice works well outside of it — it may have less applicability outside of physics. I have quibbles with a few things, as I’m an experimentalist and an atomic physicist, and Ethan is a theorist and is trained as an astrophysicist. There are bound to be some differences, but I think most of it is going to hold up for science PhDs in general.

There are plenty of brilliant people who get them, of course, but there are also plenty of people of average or even below-average intelligence who get them. All a PhD signifies, at the end of the day, is that you did the work necessary to earn a PhD. There are many people who have PhDs who will dispute this, of course. There are plenty of people who are insecure about their lives, too, and base their entire sense of self-worth on their academic achievements and accolades. You probably have met a few of them: they are called jerks.

This was probably the biggest surprise in grad school to me — how much the ratio of intelligence to stubbornness actually was in the student population, vs. the larger value I had naively expected it to be.

Don't Handle With Care

Posted on by

Antifragility and Anomaly: Why Science Works

I think I like this terminology.

Antifragility is the true opposite of fragility. Unlike mere robustness, it is the ability to actually profit from misadventure. A porcelain cup is fragile, and shatters if dropped. A plastic cup, being robust, will not be any the worse for such an experience, but it will not be any the better for it either.

…

Among the things that Taleb lists as fragile are scientific theories. Scientific theories are indeed vulnerable to disproof, since they must be tested against reality. The simplest way to describe this is to say that they must be falsifiable by experience, a criterion associated with the name of Karl Popper. In the popular imagination at least, however well established the theory may be from past experience, it could at any time be refuted in the future by a single observation that differs from what is theoretically predicted. If so, scientific theories would indeed be fragile, since they could not survive a single shock.

But that is not what really happens. Well-established theories have already explained a wide range of observations, and will not readily be destroyed by a single counterexample. On the contrary, they usually emerge all the stronger for accommodating to it.

Of course theories can break, but it requires contrary evidence in such a way that we can’t simply narrow the scope of the idea, i.e. the holes are in all through it rather than at an edge. Phlogiston, for example, or the early models of the atom.

Trust Me, Maybe? Redux

Posted on by

Chad has a post up about peer-review in the context of the BICEP2 results about inflation: Review and Replication

What ultimately matters, after all is not review but replication– that is, when somebody else tries to do the same experiment, do they get the same results? This is where issues with statistical flukes usually sort themselves out, and that’s not a problem that can be fixed by any amount of refereeing. A journal referee can look for obvious gaps in the analysis, but will never get to the level of detail of a repeat experiment or a follow-up measurement.

This has some overlap with something I wrote a few weeks back (Trust Me, Maybe?) wherein I argued that peer review is a demarcation where you can start taking claims seriously, but I realize I was thinking more about theory awaiting confirmation rather than experiment. Chad’s point that there are experimental efforts where peer review will be a formality of sorts, because we already know the experiment was carefully done, is correct. What’s important here is replication.

This reminds me of a description I recently saw (but I don’t know its origin): Peer review is a spam filter. In a case like BICEP, we’re already pretty sure it’s not spam.

I Would Do Anything for Blogs, but I Won't Do That

Posted on by

Third of a short series. Summary of and commentary on some talks at ScienceOnline 2014

The session was called Standards in blogging, referring to scientific and journalistic standards. The question before the group was should blogging have some sort of code of conduct, much like journalists do (or are supposed to have), and the research standards scientists aspire to?

It was broken down into the areas of

* Background – do you have to read the original paper? Do you contact the authors? Discuss the science with other parties?

* Neutrality and balance

* Use of anonymous sources

In broad terms it was recognized that blogging standards must be informal, because there isn’t a formal path like journalism school that could teach the standards. So we sort of have to deal with this on our own.

I think the room was generally surprised that reading the original paper had to be brought up, because if you’re critiquing a paper, of course you have to read it. It was clear from the discussion of contacting the author(s) and potential competitors that the bulk of the audience was comprised of bloggers who were not always schooled in the topics they wrote about, which led into the question of how to convey when you are speaking from a position of expertise, or interpreting others’ expertise, which may confuse readers who aren’t familiar with the author. I’m not sure there was a clear solution on that. Caveat emptor, I guess.

On the subject of anonymous sources, we discussed the fact that peer review is already anonymous, so blogging about a paper already involves anonymity. While I seem to recall a sentiment that anonymous quotes were frowned upon, one of the contrary observations was that some scientists might not want to go on the record calling out problems with a paper published in their field.

Most of this really doesn’t affect me, though. All of the discussion through the first 2/3 of the session seemed to be from the non-scientist or scientists blogging subjects outside their field of expertise. I finally jumped in to give my perspective, that I don’t interview people so a lot of the points were moot, because what I am offering is my informal opinion, based on some level of expertise. That limits the scope of my discussion, of course, but this is a hobby. The one part that does apply is that I do consult other physicist before writing some posts. I works with some smart people, so I can often double-check that I’m not being an idiot (doesn’t always work, though), but not in a way where I would quote them. More like the journalists’ deep background, being a second source.

On the subject of neutrality and balance, my view is that I blog so that I don’t have to be balanced or neutral. In the room there was a definite aversion to false balance — those stories you read where you get both sides of the story, even if the other side is crackpottery. But to me true balance and neutrality are not important in blogs the way they are in traditional journalism. In the applicable type of post I’m saying what I think, or translating some bit of science for a wider audience based on what I know. (The rest of the time I’m complaining about how someone else didn’t do it right or just posting interesting links). To me, that ability to freely add your own bias or perspective is the whole point of blogging — you can do this without it having to go through someone else’s journalistic filter. (also not having to go through someone’s editor to decide if the story is worthy)

End of the Road?

Posted on by

Can Science ever be “Settled”?

You can reach a point on basic concepts where the sheer weight of the evidence tells you that the idea is right, even though some details are left to be worked out. Science is an iterative process — we strive to be less wrong than the iteration before, and we work out details that are out on the edges of what we’ve been able to investigate. Sometimes, as in the case of quantum mechanics and relativity, those edges reveal a whole new set of behaviors, but as interesting and useful as those paradigm shifts have been, one needs to recognize that the physics that came before is still valid over a wide range of sizes and speeds. Classical treatments still work well for macroscopic objects and speeds not a large fraction of c.

Similarly, we know that the planet is heating up and we know that humans have a significant effect on this. That’s settled, despite the sturm und drang of the denialists. What’s not settled is a matter of precision of the individual contributions and effects. Germ theory is settled science, even as we learn more and more about the details of microorganisms and how they behave. And so on for gravitation, the Big Bang, and evolution.

Thinking Like a Scientist

Posted on by

First of a short series (I hope). Summary of and commentary on some talks at ScienceOnline 2014

“What is Science Literacy”

I had some high hopes for this session, since this is a topic I’ve discussed before and care about. Unfortunately (for me) a fair fraction of the talk was dominated by discussion of science engagement. This is no small matter, and I concede that if you can’t engage with an audience they won’t become literate in the first place, and also that the audience seemed to be interested in that discussion, but I was hoping for more discussion on what literacy actually is. If you haven’t defined the problem, it’s hard to come up with an answer. I was anticipating more discussion on science not being a list of facts to be memorized and literacy being a combination of knowledge and the ability to apply the knowledge, which only came up late in the session, and not in a lot of depth.

At the beginning, though, the problem was framed in terms of a discussion the moderator (David Ng from UBC) had had with an 8 year-old, who asked (1) are unicorns real, and when that got a “no”, asked (2) could they be real (again, no), and finally (3) what if you actually saw a unicorn anyway, leaping over a rainbow. How would that change your answer?

The questions were in the context of the mythical creatures, rather than horses that might have something growing out of their forehead, so even though there might be some creature that looks like a unicorn and biology doesn’t rule such an animal out, it’s the magical things they do that tell us that they don’t and could not exist (violation of conservation of energy was offered as a prominent reason).

But what about question 3? We never really got around to answering that, but here’s my take, which I covered just last week: extraordinary claims require extraordinary evidence. The non-existence of unicorns and the reasons for this are quite well-established. If anyone were to report seeing a unicorn, the general reaction would be that they were mistaken — their eyes were tricked, or their video was a fake — and what evidence they had would be closely scrutinized, because it contradicts a large volume of careful science that has already been done. And THAT is a bit of science literacy — an understanding of the process by which we accept things as true or not within science. It’s too bad we ran out of time before the discussion could go there.

One other bit that came up was that the true goal of literacy is to get more people thinking scientifically even if they aren’t scientists, something with which I agree and tends to get lost in discussions that are focused on how many scientists we may or may not need, and falsely assumes that science understanding is or should be an all-or-nothing affair.

And the Number of the Counting Shall be Three

Posted on by

You need to a flashplayer enabled browser to view this YouTube video

Another great Veritasium video. I got to meek Derek at ScienceOnline this past week, and we were in the same dinner group one night but ended up at opposite ends of the table, so I didn’t get to chat as much as I would have liked.

The video shows a great example of how scientists have to not trick themselves by only doing tests that would confirm the mechanism of their hypothesis. You have to see if some other mechanism would work as well.

Trust Me, Maybe?

Posted on by

In case you missed it, Stephen Hawking was in the news recently for purportedly claiming there are no black holes. It got a fair amount of press, because it’s freaking Stephen Hawking and because it sounds like a bold claim. There was some back-and-forth on twitter about this, because all that was presented were some presentation notes (not rigorous work, i.e. no math) based on an arXiv paper (not peer-reviewed) and the actual discussion is more subtle, which means the popular science press kinda got it wrong (even more so in the headlines)There was also the observation that this got a lot more press because it was Hawking who presented it. Matthew Francis discusses this effect of celebrity in a piece at Slate.

In short, in science (as opposed to science journalism) we don’t automatically accept things from certain people just because of their status — that’s not how science rolls; if it’s wrong it’s wrong. What status offers is an increased opportunity to be heard, and credibility that your work is carefully done — you’re going to get the benefit of the doubt but not immunity to people checking your work. While this story would have gotten less press if it had come from someone else, it probably would have gotten some, because there are plenty of pop-sci “physicists say” stories that are based on similarly untested proposals. I think this grabbing any and all new results and tendency toward hyperbole are a problem. Overselling results, some of which will end up being wrong because they were plucked when the weren’t ripe, probably undermines peoples’ confidence in science. It certainly gives fodder to the anti-science types.

There’s a hierarchy to the acceptance of science. We have to remember that what we’re doing is to try and less wrong (and there are different degrees of being wrong), and to do so we have to have confidence that we are right, and it’s really easy to convince yourself that you’re right, so the true test is in objectively convincing other people.

So how do we convince people? We tell them about the ideas and let them check our work. We let others weigh in, and fix things that are wrong with the idea. We iterate.

The backdrop to any publicly shared science is that you’ve done internal reviews. If it’s theory, the idea has been developed and shared with colleagues and if it’s experiment the proper checks have been done to make sure there isn’t some systematic or calibration effect giving you the wrong answer, lest you end up suggesting the possibility that neutrinos are superluminal or something like that.

You might then give a talk at a conference on your idea or results. Conference talks aren’t peer-reviewed (in my experience, at least) and don’t contain enough detail for much analysis, though the conference proceedings may provide this detail.

Peer review during the process of publication provides another opportunity for knowledgable people to give feedback and object to shortcomings. But peer review is sort of a double-edged sword. It’s a dividing line in this hierarchy — one can generally dismiss claims that are not peer reviewed as a sort of “you must be this tall to ride this ride” — any popular article or discussion board claim that’s based on someone’s ill-formed idea should be taken with a huge grain of salt and conference talks a smaller one — but peer review does not carry with it any guarantee of correctness. There are plenty of peer-reviewed papers whose claims are later shown to be wrong — that’s part of the process.

As I see it, this is part of a larger problem in communicating science. People will take ideas at these various stages of development and run with them, whether it’s a crackpot who read a website treatise claiming that electrons are really tiny blue pyramids (and this explains e v e r y t h i n g!) or it’s a pop-sci article that gives entirely too much weight to a conference talk. Until other scientists have had a chance to give formal feedback to theory, or try and replicate results of an experiment, you have only limited confidence that the work is correct. The more confirmation you get, the higher the confidence level. What you really want is a bunch of experiments by many different groups.

If someone had measured a result for the deflection of light by the sun that disagreed with general relativity, what would happen? Back in 1920, it would have called into question Eddington’s experiment, and several more rounds of measurement would have to be done. If it happened today, much like the superluminal neutrino result, the vast majority of the feedback would be saying the experiment must have some error to it. (Though undoubtedly a handful of people would jump in with their version of modified GR to explain the result) Once the weight of experimental result hits a certain critical mass, the expectations swing away from needing data to confirm a theory to needing exceptional data to disprove it. Science journalists understandably don’t want to wait this long to write about results, of course, which is a bit of a conundrum, but I think the system has a tendency to overstate the confidence we have in our scientific musings and findings, even without the additional amplification of having Hawking’s name attached to things.

(another discussion that may come up at ScienceOnline)

The Neverending Story

Posted on by

The eternal tug of war between science journalists and scientists. A graphical story.

Timely, as ScienceOnline is coming up this week, and this topic is sure to come up.

One of the interesting points Bee addresses is that science journalism isn’t treated like sports — nobody writes sports stories for people who know nothing about the sport being reported on. Some knowledge is assumed. I share the frustration of reading a science story and not being able to figure out what science they’re actually talking about, since it’s been simplified so much. I don’t think the multi-level story she describes will come to be anytime soon, but it’s an interesting idea.