One thing I notice a lot of advocacy websites falling victim to is citing research without really understanding what that research means. A lot of the time they’re pulling from secondary, third, fourth, or fifth resources because the information they get comes from news reports that are also reporting information from secondary, third, fourth, or fifth resources. A secondary resource, for example, would be a textbook or another article describing the research findings.
The issue with this is very rarely do these articles go into detail about the research procedure, the methods, or any resulting confounding variables. The issue with THAT is the only information YOU receive are the results. And when the results say something like “Eating a Grapefruit Everyday Will Cure Your Anxiety”, you want to believe it. What you’re missing is that the study had 20 subjects, no control variable, and the subjects–by the way–were worms.
I’ll cover some basic ways to judge whether or not a paper you read–and you should be reading the actual paper–is a valid result or something that needs a lot more investigation.
First, if you haven’t yet read any of the series Is Psychology A Science posted here on this website, you can find the first in the 4-article series right here at this link. It will give you background information on why this is important from a scientific perspective.
Contrary to what you may believe, you don’t need to be a seasoned researcher to read these papers. You don’t need to know how to do an ANOVA test or a Chi Squared test. You don’t need to know what p <.05 really means. You don’t even need to know a lot on the subject the paper reports on. All you need is some basic knowledge. Let’s go step by step.
What does a paper consist of?
What’s great about research papers is that you always know what to expect. There will be the first paragraph, which is called an Abstract that tells you what the paper is about and often contains one sentence describing the results.
Often a full-length introduction is followed. This is important when the study is backed by, or has a lot of reference for, other experiments. Here you will find the importance of the subject and what other researchers have found and how this study will differ.
Next commonly comes Methods. This is where all the materials are listed and the procedure (including participants) are described. This is done in great detail so that peer review and (if/when published) practitioners and psychologists and everyone who SHOULD be reading the study, can determine validity. Here, you will learn the ins and outs of the experiment and/or the outside resources used to gather data.
The Results follow. Here you find the final statistics. You will find step-by-step results for how each participant (or group of participants) and sometimes you will see charts, graphs, info-graphics of all types that represent the data.
Discussion comes last. Here you find what those results mean and any recommendations for further research. This is where you will find confounding variables (discussed below) and another place to scrutinize the validity of the study.
Afterward comes acknowledgements, often who financially supported the research. This can be particularly important when reading studies about medication; if the pharmaceutical company that makes Risperdal (Johnson & Johnson) is providing funding for Risperdal research, the paper is subject to publication bias. This can be harder to detect and won’t be discussed here. I will be writing a separate post on this.
Then the reference list, which are other studies cited in the paper.
This layout is important. If the paper you’re reading isn’t like this, it’s not a research paper and what you’re reading is a secondary source, meaning a summation of the original research. This could be so far from the actual research finding, that it’s completely useless.
Is this studying what the researchers meant to study?
When reading a paper, it’s important to sniff out whether what’s been observed is what was actually meant to be observed.
This is known as construct validity. Research in psychology can be tricky because most of the time you’re observing behavior and behavior can be caused by many factors. For example, if we’re trying to study whether or not television violence influences real violence in young children, we need to define what violence is and what constitutes aggressive behavior. If we don’t, trying to find a relationship between behavior and the amount of violent television being watched will be hopeless; you won’t see any relationship, you’ll just see aggressiveness and violent television as separate entities. They may look correlated, but when properly studied you may see different results.
If a paper related to television violence and children do not have operational definitions (the definitions of the behavior we’re looking for), than it’s safe to say that paper isn’t really showing much.
A lot of the time this gets twisted in the media. For example, you’ll read statistics everywhere that those labeled with schizophrenia have a shorter life span of about ten years. A lot of people take that as “schizophrenia kills”. But is that really what reports and studies are saying?
No. There are other factors: taking hardcore, organ-destroying medication for decades is one of them. Less access to proper healthcare–many with the label are homeless and struggling with drugs or just at the mercy of their mind. Disability money doesn’t necessarily fund a heart healthy, body healthy diet either; most organic foods and healthy foods are hard to sustain for those on limited income and no support. Suicide rates are also higher. So this idea that some mythical chemical imbalance causes some mythical disease that slowly kills the brain like some kind of hallucinogenic Alzheimer’s is incorrect. Or rather, it’s never been proven.
This leads us into:
What Are The Confounding Variables?
These are factors which could muck up research findings.
For example, a lot of the papers I referenced in Is Schizophrenia a Brain Disease? had their confounding factors clearly listed. They said blatantly that much of their research has no definitive results. One paper even urged people not to take their results as proof that a chemical imbalance explains what people experience. So where does this idea that this label is a disease come from?
From people taking the one line of results and posting it in the New York Times or a CNN reporter giving a brief run-down of the study on national television. Belief is where this idea comes from. Science says otherwise.
I think one of the most famous incidents of this is the idea of the schizophrenogenic mother.
There was a time doctors and patients believed it was the mother’s bad parenting that created the experiences now labeled as schizophrenia. To test this, one researcher took a group of mothers who had children who matched the diagnosis and studied how they interacted.
She concluded a mother’s behavior can create ‘schizophrenic behavior’. Then she listed that in order to be an official and valid study, she would have to also study mothers who don’t have labeled children and their interactions; her confounding variable was no control group.
So what happened? The results were taken and spread to newspapers, to doctors, to psychologists and psychiatrists and for a while people really believed the mother was to blame for madness.
Because the general public, and even some doctors, didn’t understand how to read research papers. And also because it was an answer and they didn’t have any other answers. Rather than be scientific, they followed their intuition. It’s something that still plagues psychology today.
That’s why
Control Groups
Are super important.
Say you’re reading a paper on the medication Abilify and its relation to a reduction in psychotic symptoms at a dosage of 20mg. You read in the procedure section that thirty five people (an okay sample size) were recruited and given Abilify and you read in the Results section that 89% of the participants reported a reduction in symptoms within the four week time period. You also read in the discussion section that the researchers conclude Abilify has effectiveness at the dosage of 20mg on psychosis.
This sounds legitimate. And any average person who read that paper would probably feel that Abilify really does have an effectiveness on psychosis at 20mg.
What would be some problems with this, though?
One striking thing would be–where is the comparison? In order to see any level of effectiveness, we need to compare it to a group that doesn’t receive the treatment, a group that isn’t manipulated. This is a control group.
We’d also need to know how the participants were randomly chosen. We’d need to know the symptoms which were reduced and whether that was sustained. I’d criticize the four week trial period. Was there also a follow up? If so, what were the results? And, if you are like me and extra-critical, you’d wonder why they didn’t cite all the research in the introduction which says there is no efficacy for an Abilify dose over 10mg.
This can make certain things hard to study ethically and practically. There are many ways around that, but we won’t discuss all that here. If you are reading a paper and have trouble finding what they compared their manipulated variable to, or if their comparison group seems shoddy, search for some more research.
I’d like to point out had that research example I gave above actually been published, hell would have frozen over. There are some shoddy papers out there, but a lot are caught in peer review.
Last but not least:
Results are results. Interpretation is interpretation. Secondary sources are not results.
Media attempts to get things right. But often they don’t.
If you are on an advocacy site for mental health and you read a sentence that says “research shows”, and they don’t list a reference, ask for it. If they do list a reference, click on that reference. If that reference is another secondary source, and that secondary source doesn’t list a primary source, like the actual research paper, than you can’t be sure how accurate that “research shows” statistic is.
I’m not saying we should start mistrusting our fellow advocates. I’m saying as advocates the information we should put out must be backed by facts, not opinion or interpretation. Most of us who advocate are peers, meaning we have lived mental health experience. Great! We already have an advantage by having inside knowledge on experiences that most doctors don’t. Let’s boost that advantage with also having scientific knowledge and thinking critically.
Even I’m guilty of saying “research shows” in previous posts on my old blog and perhaps even this one already. It’s hard work finding these papers and reading them and citing them and analyzing them. But did anything come from soft work? No.
If you want to advocate, help empower us by criticizing research, reading research, and sharing research. Open a discussion about what’s being studied. Push clinical psychologists and psychiatrists to actually read what’s been coming out.
Together, we can change the perspective of mental health in society.
Would you like to continue the conversation, see silly (and beautiful) photos, or nonsensical two second videos? Great! Follow me:
Instagram: @written_in_the_photo
*New* Twitter: @philopsychotic
Snapchat: @FabulousIRL
If you liked this article, please share it, hit the like button, and follow ThePhilosophicalPsychotic on WordPress or with email notifications. I appreciate every reader and commentator. You give me more reason to continue this joyous hobby.
Please share! You won’t regret it.
The Philosophical Psychotic 