laboratory glasswareSkim the news headlines most days, and you’ll see news reports about scientific studies on all sorts of topics. The trouble is that many of the news stories offer soundbite analysis and quick conclusions, many of which are misleading or lack relevance or rigor. And we as citizens often go no further than skimming the headlines and adding their bold assertions to our body of “knowledge” about health, the environment, education, and more.

Neither we nor reporters usually delve into these studies and examine the details in depth. Who has the time?

We can’t put every scientific study we hear about under a microscope, but we can develop the critical thinking skills that will help us better assess the validity and relevancy of these studies and help students to do the same.

One thing we can do is consider the studies themselves. A recent AlterNet article, “6 Ways Scientific Studies Can Trick You,” highlights some of the tactics (sometimes intentional, sometimes not) that can affect the conclusion of a study, such as:

1. Start with a wrong assumption.

2. Throw out data you don’t like.

3. Set improper threshholds.

4.  Share findings that aren’t statistically significant.

5.  Design the study to get the desired results.

6.  All of the above.

Although we’re taught that science is an objective undertaking, the truth is that variables, data, and framing can be manipulated.

Dr. Ben Goldacre’s TEDx talk “Battling Bad Science.” is very helpful in demonstrating how evidence can be distorted and studies can be manipulated.

In fact, there are so many instances of manipulation, inaccurate information, errors, and even fraud and plagiarism occurring in scientific studies now, that websites like Retraction Watch have popped up to monitor and report on them.

Additionally, all of us, scientists and citizens alike, bring along our own biased lenses. Just one example:

For hundreds of years, the common practice (and common thought) has been to require conducting experiments on animals, and the belief that animal experiments are relevant and necessary to help humans. But recent research is causing some to begin questioning the validity of studies using animal testing (at least for testing the safety of chemicals), saying that:

“The results of these experiments challenge the longstanding scientific presumption holding that animal experiments are of direct relevance to humans. For that reason they potentially invalidate the entire body of safety information that has been built up to distinguish safe chemicals from unsafe ones. The new results arise from basic medical research, which itself rests heavily on the idea that treatments can be developed in animals and transferred to humans.”

These new studies themselves need to be evaluated, but the potential ramifications throw a whole lot of previously-accepted science into question.

And then there’s the way scientific studies are reported. As Gary Gutting mentions in a recent New York Times commentary:

“Media tend to present almost any scientific result they report as valuable for guiding our lives, with the entire series of reports accumulating a vast body of practical knowledge. In fact, most scientific results are of no immediate practical value; they merely move us one small step closer to a final result that may be truly useful. Too many news reports present experimental results as providing good advice on which we can reliably act.”

Gutting goes on to offer the idea of a labeling system for scientific reporting to help clarify the validity and importance of a study:

“Is it merely a preliminary result (a small-scale heuristic study meant to suggest a hypothesis that will itself require many stages of further testing before we have a reliable conclusion)?  Is it a larger-scale observational study (showing a correlation but by no means establishing a causal connection)?  Is it a large-sample randomized controlled test (establishing a causal connection, given specific conditions)?  Or, finally, is it a well-established scientific law that we know how to apply in a wide range of conditions?”

Scientific studies and their reporting offer a valuable opportunity to hone critical thinking skills and to remind us not to blindly accept what we read or are told. In fact, since science is wrapped in such a cloak of authority and credibility, it’s vital that we look beyond those news headlines and dive deeper into the details — and teach students to do the same.

~ Marsha

Continue the conversation! Leave your comment below, and “like” and share this post via your social media sites.