158 scientists used the same data, but their politics predicted the results. Study provides evidence that when experts act independently to answer the same question using the same dataset, their conclusions tend to align with their pre-existing ideological beliefs.
158 scientists used the same data, but their politics predicted the results
13 Comments
>They had the freedom to choose their own statistical methods and variables to test the hypothesis.
Expertise, skills and choice of hypothesis matter.
The study utilized data from 158 researchers organized into 71 separate teams. These teams had participated in an experiment where they were asked to determine whether immigration affects public support for social welfare programs. The researchers were provided with data from the International Social Survey Program, covering various countries and spanning the years 1985 to 2016.
Before the teams began their analysis, they completed a survey. One of the questions asked for their stance on immigration policy. Specifically, they were asked if laws on immigration should be relaxed or made tougher. Their responses were recorded on a scale ranging from zero to six.
The teams then proceeded to analyze the data. They were tasked with replicating a well-known previous study that found no link between immigration and welfare support. After replicating that study, the teams were instructed to extend the research using the new data provided. They had the freedom to choose their own statistical methods and variables to test the hypothesis.
Collectively, the 71 teams estimated 1,253 distinct statistical models. The results varied significantly. Some teams concluded that immigration strongly decreased public support for social programs. Other teams found that immigration strongly increased such support. Many others found no significant effect at all.
[deleted]
I fully accept that this is true for studies on highly politicised issues such as immigration and climate change. I wonder if it is as true for other hard-science issues such as meteorite composition or days of rainfall per year at a particular location.
Like Michaelangelo and a block of marble, they carve out what they see. The marble is just marble, the datasets just a dataset.
The only problem is that we assume these statistical analysis are, forgive the pun, set in stone.
And this is why peer reviews matter.
Seems that [George J. Borjas](https://en.wikipedia.org/wiki/George_J._Borjas) (One of the authors) may himself be susceptible.
*In 2017, an analysis of Borjas’ study on the effects of the Mariel boatlift concluded that Borjas’ findings “may simply be spurious” and that his theory of the economic impact of the boatlift “doesn’t fit the evidence.”[14] A number of other studies concluded the opposite of what Borjas’ study had found.*
And that is why sharing results and conclusions and engaging in peer review is important. Others will see things in your research you have missed.
As long as the debate stays centered in facts and researchers are open to well justified criticism then science can progress.
I wouldn’t rule out that certain pre-existing inclinations might lead to the researcher being able to create connections / infer conclusions that someone with different inclinations might not be able to, and vice-versa. For example, if I as a researcher have been exposed to some form of abuse and I am doing research into the psychology of abused victims I might be able to spot certain factors in my research data that someone who hasn’t been a victim of abuse might not be primed to, granted I am not an expert on research methodologies of large scale.
So perhaps it could be an issue that certain fields of research are almost entirely dominated by people ranging from the marx-curious to the pre-revolution feral…
so, my question is: what was the quality of those methods? did the groups show a marked difference in their degree of objectivity? in the public sphere, there are political groups with an ongoing willingness to ignore facts for the purpose of dehumanizing immigrants – did their scientists show this inclination as well?
if one group is fudging the numbers and the other is telling the truth, you would get this result.
“If you torture the data long enough, it will confess to anything” (c Ronald Coase)
Title seems to be misleading. Much respect for sociologists. They do a ton of work. But I’m given to understand that sociology, the subject of the research experiment, is not a hard science.