Red Flags of Bad Science
Like in any other domain of human endeavor there is a spectrum of quality and accuracy in science.
Some double-blind, placebo-controlled human research is excellent and profoundly helpful to us in living better but I will show here that some “science” is tantamount to a ranty, opinionated blog, some is funded by the pharmaceutical industry and really is just marketing masquerading as science.
I’ll encourage you to not place blind faith in the establishment of science the way that your ancestors likely placed blind faith in the catholic church or whatever their religious persuasion was.
As Nassim Taleb articulates in Antifragile:
the unconditional belief in the idea of scientific prediction regardless of the domain, the aim to squeeze the future into numerical reductions whether reliable or unreliable. For we have managed to transfer religious belief into gullibility for whatever can masquerade as science.
Information is the most powerful tool we have for living well, bad information really can limit us, science is the best source of helpful information, but it’s far from a crystal ball that can be stared into to ascertain absolute truth. If you’re willing to devote a little bit of extra attention and cognitive horsepower as a layperson you can easily recognize bad science.
Skeptical of Bad Science?
You may say…
Come’ on Jonathan, there’s no such thing as bad science! All science is good!
I’ll remind you that:
- For 30 years “science” and physicians promised the public that smoking cigarettes was good for you.
- For 40 years “science” and the FDA put out the ridiculous food pyramid recommending the public eat a diet mostly of bread, pasta, and dairy.
Has science cleaned up its act in recent times?
In 2015 Richard Horton, editor in chief of Lancet, the UK’s leading medical journal, stated:
“The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Science has taken a turn toward darkness.”
In 2010 Dr. Marcia Angell of the prestigious New England Medical Journal was equally disparaging of the state of science:
“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of the New England Journal of Medicine”
Editors, Publishers, Impact Factors, and Reprint Income
Editors would like to imagine they are simply gatekeepers who facilitate the interaction between authors who wish to…
I’m not saying that science is useless but that it should be scrutinized and just because something is labeled “science” you should not turn off your critical thinking skills.
Go to the Source
Probably the most common mistake that the layperson makes is reading an article online about science and assuming that they are actually doing meaningful research. A well-meaning person will Google a subject like…
- Climate change
- Or nutrition
…they will read the first few articles they find and give themselves a pat on back for acquiring a modicum more expertise on the subject.
While not totally useless, this information consumption really doesn’t count as researching the science on a topic. The same applies to listening to podcasts or watching Youtube videos.
Researching the science means actually reading the scientific studies, meta-analysis papers, books, and PubMed abstracts. Which are full of…
- Stale jargon
- Superfluous acronyms
- Scientific terminology
- And not exactly inspiring verbal prose
…they really are not fun to read but if you actually want to understand a scientific topic, it’s crucial to do.
Lifehack: Next time you find yourself in a debate with someone over some scientific topic. Just ask them when was the last time that they actually read a book specifically on the topic of contention.
Really? You don’t think IQ is a meaningful scientific measurement of intelligence? What was the last book that you read on IQ?
Almost always they will have NOT read a book on the topic. So then you can point out how really uninformed they are on the topic. As a rule in life:
If you haven’t read a book on a topic you really don’t know what you’re talking about in regards to it.
They have so little commitment to veracity and empiricism that they are not even willing to commit a couple of solid hours to understand the topic.
Then you can point out that their position is really just an arbitrary preference and that you guys debating this issue is really tantamount to merely arguing whether chocolate or vanilla ice cream is better.
Obviously, this argument really only works if you yourself have read a book on the topic. So better read up on the topics you anticipate debating!
Mainstream Media Misinformation
You may think that the New York Times, Huffington Post or The Guardian have high standards when it comes to fact-checking and accuracy but I’ll suggest that after reading a New study finds… type of story in these publications go and read the study itself.
Often it’s just a Google search away. If you take the time to do this often you’ll find a blatant error or misrepresentation of the science in the mainstream publication, particularly if there are political or financial implications of the science.
The job of journalists should be, to tell the truth, but more often they lie. The excellent book Trust Me, I’m Lying by Ryan Holiday explains why.
There’s a necessary class of merchants of science or science writers that people like myself, Dave Asprey and Steven Kotler fall into. Our job is to make the science more understandable and useful to the public but we make mistakes from time to time, our fact-checking is not perfect and our incentives are often not aligned with our earnest desire to educate people on how to use science to live better. I call us a necessary class because scientific publications really are quiet boring and nearly indecipherable to the general public, without marketing-minded people repackaging science it would not make it into the mainstream.
I think we can all agree that we would like science to be available and understandable to all so that it can be a democratizing force in the world.
However, amongst us, there is a real spectrum of genuine educators to charlatans who are just in it to make a buck. How can you spot a disingenuous middleman?
Lack of proper citations. Many science or health articles consist of several hundred words followed by a handful of links to studies that ostensibly support the conclusions presented by the writer of the article. But a lot of times the citations really don’t support the conclusions, this is why you should look for direct quotes from the studies or researchers themselves supporting conclusions as opposed to just links.
Omissions of unappealing information. Especially in the field of anti-aging products and Biohacking supplements, disingenuous science writers will omit information that hurts the case for the supplement or drug in question.
Clear conflicts of interest. We usually make our money a few different ways:
- Affiliate sales of products
- Advertising revenues
Often these income sources create a perverse incentive. If it’s clear that a writer has a financial incentive that makes them partial to a particular product or supplements you should further scrutinize their portrayal of the science surrounding it.
Ads. Online advertising creates a bad incentives for writers to put out a great quantity of work in lieu of quality. If you’re reading a science article on a website that has a bunch of ads in the sidebar and header and below the article that is further reason to question it.
Ghostwritten. Writers who know what they are talking about almost always put their real name and face on their work. They are proud of their work and stand behind it. There’s a lot of quintessential content farm websites that churn out a great quantity of articles about health, science, and supplementation. Invariably, the writers are invisible, they are often outsourced digital labors working for a few dollars an hour in a place like the Philippines or India.
Lack of author credentials. Writers who know what they are talking about will have some relevant credentials and experience.
Scientists will often spend decades myopically focused on a singular scientific niche, they’ll understand it profoundly but often they’ll lack holistic knowledge of how that science could be useful to laypeople. The writer’s job is to be a scientific polymath and a lot of times we can articulate applications of the science that can really help people.
However, if you’re reading an article online that contains one or more of these red flags that should trigger your skepticism.
If you really want to understand something, if you’re making a crucial decision based on science, go the source.
However, sometimes the source publication itself is bad science. Consider the case of the recent (hilarious!) hoax paper…
…that actually passed the peer review process and was published in a mainstream social science journal, Cogent Social Sciences.
The authors specifically wrote the least scientific paper that they could.
They specifically wrote a paper that should have been rejected by a scientific journal. A few examples of the absurdities it postulated:
Manspreading — a complaint levied against men for sitting with their legs spread wide — is akin to raping the empty space around him.
We conclude that penises are not best understood as the male sexual organ, or as a male reproductive organ, but instead as an enacted social construct that is both damaging and problematic for society and future generations. The conceptual penis presents significant problems for gender identity and reproductive identity within social and family dynamics, is exclusionary to disenfranchised communities based upon gender or reproductive identity, is an enduring source of abuse for women and other gender-marginalized groups and individuals, is the universal performative source of rape, and is the conceptual driver behind much of climate change.
Destructive, unsustainable hegemonically male approaches to pressing environmental policy and action are the predictable results of a raping of nature by a male-dominated mindset.
The authors intentionally made it as absurd and anti-scientific as possible, from the Skeptic.com article where they admitted their hoax
After completing the paper, we read it carefully to ensure it didn’t say anything meaningful, and as neither one of us could determine what it is actually about, we deemed it a success.
Not only is the text ridiculous, so are the references. Most of our references are quotations from papers and figures in the field that barely make sense in the context of the text.
The Conceptual Penis as a Social Construct: A Sokal-Style Hoax on Gender Studies
Note from the editor: Every once in awhile it is necessary and desirable to expose extreme ideologies for what they are…
Nearly a third of our references in the original paper go to fake sources from a website mocking the fact that this kind of thing is brainlessly possible, particularly in “academic” fields corrupted by postmodernism.
When researching the science it’s important to ask who benefits? Who is getting paid?
Many journals like Cogent Social Sciences operate with a pay-to-publish model. The authors of The conceptual penis paid $625, they published under fake names as part of a fake social research group, which Cogent Social Sciences did not catch. So for $625, approximately the same amount you would have to pay to score some cocaine, get drunk on champagne and bang a hooker in a chintzy hotel room in Vegas, anyone’s totally nonsensical opinions can be transformed into “science”.
How many other papers that have appeared within the pages of Cogent Social Sciences are similar pseudo-scientific none-sense? I would wager probably most of them.
Moralizing Should be Scrutinized
You should be very suspicious of science or scientists that make moral statements. Science and morality should not necessarily be separated like the church and state (or matter and anti-matter in the warp drive of the starship Enterprise to use a pseudo-scientific metaphor) but the more moralizing statements that bracket the science the more you should be suspicious of it.
Case in point, The conceptual penis is chuck full of moralizing statements — another major red flag that the peer reviewers at Cogent Social Sciences ignored.
To quote the hoaxers again
That is, we sought to demonstrate that a desire for a certain moral view of the world to be validated could overcome the critical assessment required for legitimate scholarship. Particularly, we suspected that gender studies is crippled academically by an overriding almost-religious belief that maleness is the root of all evil. On the evidence, our suspicion was justified.
Luckily for Biohackers the science of peak performance, health and antiaging is not very politicized relative to a field like social science which is just a minefield of misinformation and propaganda. Let’s keep it that way!
What (really) is pseudoscience?
You probably hear the word pseudoscience bandied about quite a lot.
You also hear it’s cousin pseudointellectual being used.
A whole lot less frequently you may hear the word I like even more than these two, sophistry.
And I’m sure you’re already sick of hearing the buzzphrase Fake News, which is definitionally quiet synonymous with pseudo-intellectualism.
Accusations of pseudoscience or pseudo-intellectualism almost always are a component part of some political or ideological narrative. Next family reunion bring up a scientific subject and it won’t be long before it devolves into an ideological or political standoff over the plates of foods.
Whether effective or ineffective, accusing the other side of pseudo-veracity is one of the most commonly used debate tactics.
How exactly would you know the difference between actual science and pseudoscience?
Actually, it’s quite simple if we look at the definition of pseudoscience
Pseudoscience purports to be an accurate description of physical law despite either 1) consistently failing experimental verification or 2) being unfalsifiable.
I think that second part of the definition is most interesting. Pseudoscience or Pseudointellectual movements are fundamentally unfalsifiable.
They make assertions about life, society or the world that cannot be disproven.
What are some examples?
If I say…
There are exactly 2.4 million rabbits in France at this moment.
…can you disprove it?
Can you disprove that there is NOT an invisible dancing penguin in the sky named Pete who gets very sad if you have sex with the wrong person?
Ok, those examples are pretty silly. What are some more practical ones?
I can’t really disprove that God created the universe 10,000 years ago because God in his infinite power would have the ability to create a universe instantaneously that appeared to be 13 billion years old.
So creationism or intelligent design is Pseudoscience.
As was postulated in a hilariously stupid “scientific” paper published in the journal Cogent Social Sciences…
Can I disprove that I, as a male, am NOT causing climate change by raping the empty space around me?
It’s so absurd that really I can’t disprove it.
So social science is, let’s be honest (mostly), Pseudoscience.
Can you disprove that redistributing the wealth of the top 1% of humanity to the rest of the world would create a global utopia?
Well, even though it’s been downright disastrous almost every time it’s been attempted on a smaller scale, it’s never been attempted on a global scale. So you can’t really disprove it.
Thus the economists and professors that advocate for Globalist Utopian Socialism (Or UBI) are Psuedointellectuals.
So next time you find yourself in the middle of a debate on an ostensibly scientific subject or you read in a magazine that some new groundbreaking study discovered something totally counterintuitive.
Just ask how what they are saying could be disproven…
If disproving it would be extraordinarily difficult or impossible then it’s unfalsifiable. It’s really NOT a scientific or intellectual position, it’s just a preference.
Newer scientific studies should be viewed as more credible than older ones. In researching the meta-analyses of Nootropics that I write I generally disregard studies that are over 30 years old.
In April of 2017 Pubmed announced they are actually standing up for science by publishing conflict of interest information on the study abstract page instead of allowing it to be buried behind an academic paywall.
This resulted from 62 scientists and physicians, and five United States senators petitioning the National Library of Medicine & National Institutes of Health to make public the funding sources of studies. This makes me more optimistic about institutions and politicians — in the news, it seems that all we hear about is how corrupt and incompetent institutions and politicians are but sometimes they actually act in our best interests!
So it’s reasonable to view science published after April of 2017 as a little more credible.
Red Flags of Bad Science
To quote Maciej D. Zatonski on Quora:
Here are some classic examples that indicate that what you are reading is b******t:
- No control group
- No double blinding
- Results not replicated
- No peer review
- Sensationalised headlines
- Bonus: Betteridge’s Law of Headlines says, that if the title has a question, the answer to that question is “No” 
- Conflicts of interest
- Misinterpreted results
- Causation assumed when only correlation exists
- Conclusions not drawn from results
- Too small sample or sample not representative
- Selective reporting of data
- Claimed “Galileo’s Gambit” (results from a single study that contradict mainstream science are claimed to be the only correct ones; discredited authors are claimed to be “persecuted” and therefore right)
- Anecdotes used instead of evidence (“I know there is no evidence homeopathy works, but it worked for me!”)
- Changes of conspiracy and collusion
- Stressing or appealing to status or authority of someone (real or fictitious)
- Repetition of discredited arguments
- False dichotomy / failure to realise there are more than only two opposing outcomes
- Cherry picking of data / ignoring majority of evidence (3 studies show that climate change is not real, while ignoring 3000 studies that show otherwise)
- Wishful thinking (there is no evidence for existence of gods, but we wish they were real)
- Appeal to ancient wisdom (“it must be true because many people were saying this for a long time before”)
- Use of pseudoscientific language
- Daily Mail articles / Fox “news”
- Discrediting an argument because it comes from a person you dislike
- Appeal to nature (“it is natural, so must be healthy”; well… earthquakes, scorpions, cyanide and botulinum toxin are natural too!)
- False balance — counter argument to scientific fact claimed to be of equal value/weight to a counter-argument (theory of embryological foetal development vs stork theory of human birth; theory of evolution by natural selection vs creationism, etc)
- Appeal to common sense or alarmism (usually done to receive extra funding for pointless research)
Dr. John Ioannidis — The Bad Science Expert
A Greek American doctor; over the past few decades, he has very methodically and none-sensationally shown the light on bad science.
An exhaustive article that appeared in The Atlantic interviewed him thoroughly at the University of Ioannina Medical School campus
Ioannidis… delivered what felt like a coup de grâce: wasn’t it possible, he asked, that drug companies were carefully selecting the topics of their studies — for example, comparing their new drugs against those already known to be inferior to others on the market — so that they were ahead of the game even before the data juggling began? “Maybe sometimes it’s the questions that are biased, not the answers,”
Lies, Damned Lies, and Medical Science
Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong. So why are…
On the integrity of the scientific process
We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously.
The science journal Nature concurred in a 2006 editorial:
“Scientists understand that peer review per se provides only a minimal assurance of quality, and that the public conception of peer review as a stamp of authentication is far from the truth.”
Quality and Value: The True Purpose of peer review
Nature is the international weekly journal of science: a magazine style journal that publishes full-length research…
Double-Blind, Randomized Placebo-Controlled and Pre-Registered
You’ve likely heard before that the gold standard of science is the double-blind, randomized, placebo-controlled study
- This means that approximately half the subjects are given a placebo. This should prove that the drug or supplement being studied outperforms the placebo effect. Which is actually pretty meaningful because the placebo effect is powerful!
- The patients are not told if they are receiving a placebo nor are the researchers told if they are administering the placebo or the real thing. You can imagine if the researchers know they are giving a placebo to an unwitting patient they will act a bit bashfully which the patient may notice, blunting the placebo effect. This is why it’s crucial that studies are double-blinded.
The Replication Crisis
As Dr. Rolf Zwaan explains in this interview, it was discovered in 2015 that out of 100 recent studies in the fields of cognitive psychology and social science, less than a third were actually replicable.
Dr. John Ioannidis from the paper Why Most Published Research Findings Are False:
“Several methodologists have pointed out… that the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study…”
Why Most Published Research Findings Are False
There is increasing concern that most current published research findings are false. The probability that a research…
The best solution to the replication crisis would seem to be Pre-Registration, meaning that…
- The hypothesis, study design, and methodology are Pre-Registered with journals before conducting the study.
- This keeps the researchers focused on determining if the data confirms or denies the hypothesis.
- This prevents researchers from expressing a bias in research or searching for a desired result.
Dr. Ioannidis goes on to clarify the place of Pre-Registration in studies:
“In some research designs, efforts may also be more successful with upfront registration of studies, e.g., randomized trials… Registration would pose a challenge for hypothesis-generating research. Some kind of registration or networking of data collections or investigators within fields may be more feasible than registration of each and every hypothesis-generating experiment. Regardless, even if we do not see a great deal of progress with registration of studies in other fields, the principles of developing and adhering to a protocol could be more widely borrowed from randomized controlled trials.”
In 2016 Forbes listed pharmaceutical as the most profitable amongst major industries — beating out banking, financial services, tobacco, and the IT industry. This creates an appreciable (and irresistible?) incentive for these firms to yield their influence to pervert science.
Consider the sordid history of Bayer Pharmaceutical which has recently merged with the globally hated Monsanto.
It would be difficult to call the current iteration of the company evil based solely on their participation in WW2 70 years ago, however many still consider it one of the least benign corporate citizens in the world because of it’s Factor VIII scandal in the 1980s; when it turned out that a product of their’s infected 6,000 hemophiliacs with AIDS they paid out a $600 million settlement and then outrageously proceed to continue to sell the product in other international markets, infecting unwitting people around the world with the deadly virus.
I don’t ascribe to the fringe conspiracy theories about the purposeful evilness of these companies; I think it’s a combination of the perverse sticks and carrots that our globalized world offers in combination with the outgroup preference that a corporate culture engenders naturally amongst its employees.
Ben Goldacre an Epidemiologist points out that
Industry funded trials are four times more likely to have a flattering result
The book Adaptogens is especially critical of pharmaceutical science:
“The great majority of published research is so deeply flawed that it should be considered essentially worthless.” So wrote John Ioannidis, PhD,
Ioannidis singled out the following types of studies as being particularly likely to lead to a worthless result:
Studies with a small sample size;
Studies that consider a small number of possible effects;
Studies whose outcomes are poorly or subjectively defined;
Studies in which financial conflict of interest is a factor;
Studies in which the researchers are prejudiced by being unduly wedded to a particular outcome;
and Studies of a topic that is currently “hot.”
John Ioannidis clarified further on conflict of interests in studies
“The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.”
From The Atlantic article:
…drug-company research wasn’t measuring critically important “hard” outcomes for patients, such as survival versus death, and instead tended to measure “softer” outcomes, such as self-reported symptoms (“my chest doesn’t hurt as much today”)
Not everything pharmaceutical is bad, in fact, some of the best smart drugs are products of pharmaceutical companies, but where the moral hazard is so great you should be skeptical.
Be Suspicious of Bias Confirming Science
You want to scrutinize science that really underlines the obvious moral position of the author or publisher. Again, a lot of what you see published is opinion masquerading as science.
- A really lefty Liberal arts college produces social science saying that women are actually better at engineering (or whatever) than men.
- Evangelical Christians produce an Intelligent Design Biology textbook.
- A vegan that publishes nutrition research that eating meat causes inflammation. (I’d be more likely to believe this last one!)
I’m not suggesting that you totally disregard science that supports a preferred bias.
I’m suggesting that when you are reading a study, Google search the authors, publication or establishment and just take about 30 seconds to look at their social media.
If their social media has a really strong ideological or moralizing bent that the study coincidentally supports that should be a red flag, I’ll encourage you to further scrutinize the study.
- Is the study focused mostly on the data?
- Or is it full of a bunch of moralizing opinions?
- Do the researchers admit in the study that they were particularly surprised by any of the findings?
Conversely, a lot of times you’ll find studies or researchers that mention specifically that their findings challenged their biases or preconceptions yet they published them anyways, I would take this as a sign of integrity.
On its surface, this topic might seem depressing. Perhaps you’re feeling like a little kid that has just found out that Santa Claus doesn’t exist. As Biohackers we really love science;
- We lionize scientists.
- We relish that discovery feeling of learning new things.
- We get excited about new scientific findings that allow us to live better.
But the truth is that a lot of science has been corrupted by human bias, bad incentives and bureaucratic institutions.
My takeaways from researching this topic thoroughly are…
- Quality matters a lot more than quantity when it comes to studies. If there is a recent gold-standard double-blind, randomized, placebo-controlled pre-registered study done on a given Nootropic I’ll mostly disregard the lower quality studies done on it.
- I’ll more seriously consider statistically significant amounts of the anecdotal accounts and Biohacker reports about a given Nootropic.
For example, recently I caught some flack on Reddit from skeptics about a meta analysis podcast of Vitamin B17, which is a bit of a controversial vitamin.
Unsurprisingly, they pointed to some science done many decades ago that declared B17 a hoax, even though there is more recent science done that is more positive about it. The science that is negative about B17 is replete with red flags and it was actually published by a pair of researchers that had in the 1950s defended big tobacco and declared smoking cigarettes healthy; something that we now know is absurdly anti-scientific.
As I mention in my podcast, in my research of B17 I looked through +600 anecdotal reports and did not find one incidence that confirmed the negative B17 “science”.
In this kind of case, it’s pretty clear that the anecdotal data should be held in higher esteem than the published “science”.
I hope you’ll practice similarly nuanced critical thinking in your scrutinizing of the science and your personal Biohacking practice.
Originally published at www.limitlessmindset.com.