Should we trust papers published in top social science journals? (with Daniel Lakens)
Jul 24, 2024
auto_awesome
In this conversation with Daniel Lakens, an experimental psychologist at Eindhoven University of Technology, he tackles the trustworthiness of social science research. They discuss how p-hacking can distort findings and the need for rigorous methodologies. Daniel critiques psychology's fragmented theories, advocating for broader frameworks akin to physics. He also explores the limitations of peer review and suggests integrating diverse expertise to enhance research integrity. The episode finishes with insights on collaboration and the educational shifts needed in future research.
Constructive criticism culture in academia elevates research quality and integrity for impactful outcomes.
Deep dives
The Importance of Methodological Procedures in Trusting Academic Papers
Ensuring the trustworthiness of academic papers, especially in disciplines like psychology, relies heavily on methodological procedures. By setting clear falsifiable hypotheses, solid theoretical frameworks, and robust data analysis plans, researchers increase the reliability of their findings. Emphasizing these methodological components helps in forming a comprehensive judgment on the credibility of a new paper.
Significance of Large Sample Sizes in Research Reliability
Large sample sizes play a crucial role in enhancing the reliability of research results. Small sample sizes lead to high variability in estimates, making results less dependable. Conversely, large datasets provide higher accuracy due to reduced variability, making the findings more reliable and robust against potential biases like p-hacking, ensuring the trustworthiness of scientific studies.
The Challenge of P-Hacking Despite Large Data Sets
While large data sets can improve research reliability, they do not entirely prevent questionable practices like p-hacking. The flexibility researchers have in adjusting statistical models and analyses poses a risk of manipulating data to fit desired outcomes, even with substantial datasets. The presence of theoretical constraints helps mitigate p-hacking tendencies, highlighting the importance of strong theoretical frameworks in research.
The Need for Cumulative Theory Building in Psychology
Criticism towards psychology as a field highlights the lack of unified overarching theories compared to disciplines like physics. The emphasis on accumulating knowledge into comprehensive theories is essential for advancing the field and addressing criticisms of psychology's focus on individual effects rather than broader generalizable theories. Encouraging collaborative theory-building efforts and promoting rigorous theoretical frameworks can strengthen the cumulative progress of psychology.
Creating a Culture of Constructive Criticism in Science
Establishing a culture of constructive criticism within the scientific community can significantly enhance the quality and integrity of research. Encouraging researchers to actively seek and provide critical feedback, whether through formal red teaming exercises, adversarial collaborations, or open discussions, fosters a transparent and accountable research environment. By incorporating diverse perspectives and engaging in constructive critique, scientists can elevate the standards of academic inquiry and promote reliable and impactful research outcomes.
Implicit Association Test: Criticism and Ambiguity
The Implicit Association Test (IAT) is seen as methodologically intriguing but faces substantial critique. It is difficult to establish what the test actually measures, leading to questions about its validity. The IAT was initially seen as a measure of deep implicit biases, such as racism, but faced challenges regarding low test-retest reliability and interpretations surrounding its true meaning. Criticisms center on the task's ability to create associations based on task features rather than implicit associations, highlighting the importance of clearer communication when using or interpreting the IAT.
Historical Value of Terror Management Theory and Future Outlook
Terror Management Theory, popular in the '80s, explores the impact of mortality reminders on behaviors and attitudes. Recent criticisms and failed replications suggest a lack of conclusive evidence supporting its assertions. While the theory's historical significance is acknowledged, future research directions should address methodological concerns and clarify the theory's validity. The theory's development over the next decade will likely focus on addressing inconsistencies and refining its core principles.
Growth Mindset: Fundamental Learning Skill
The concept of a growth mindset, emphasizing the belief that abilities can improve with effort, presents a valuable skill to teach individuals. Advancing beyond one-off interventions, a growth mindset encourages individuals to view failures as opportunities for growth and learning. While its effect size might not be substantial, consistent reinforcement of a growth mindset can contribute to enhanced learning outcomes and resilience.
Collaborative Approach in Social Science Research
Future advancements in social science research should prioritize collaborative efforts among researchers to tackle complex societal challenges. Enhanced collaboration can drive innovative solutions and facilitate collective problem-solving. Promoting collaborative initiatives over individual ventures can lead to meaningful progress and comprehensive insights into societal dynamics and human behavior.
How much should we trust social science papers in top journals? How do we know a paper is trustworthy? Do large datasets mitigate p-hacking? Why doesn't psychology as a field seem to be working towards a grand unified theory? Why aren't more psychological theories written in math? Or are other scientific fields mathematicized to a fault? How do we make psychology cumulative? How can we create environments, especially in academia, that incentivize constructive criticism? Why isn't peer review pulling its weight in terms of catching errors and constructively criticizing papers? What kinds of problems simply can't be caught by peer review? Why is peer review saved for the very end of the publication process? What is "importance hacking"? On what bits of psychological knowledge is there consensus among researchers? When and why do adversarial collaborations fail? Is admission of error a skill that can be taught and learned? How can students be taught that p-hacking is problematic without causing them to over-correct into a failure to explore their problem space thoroughly and efficiently?
Daniel Lakens is an experimental psychologist working at the Human-Technology Interaction group at Eindhoven University of Technology. In addition to his empirical work in cognitive and social psychology, he works actively on improving research methods and statistical inferences, and has published on the importance of replication research, sequential analyses and equivalence testing, and frequentist statistics. Follow him on Twitter / X at @Lakens.
