The Nonlinear Library EA - Exploring Noise in Charity Evaluations by Malin Ploder
Jul 14, 2024
26:10
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Exploring Noise in Charity Evaluations, published by Malin Ploder on July 14, 2024 on The Effective Altruism Forum.
Hi! My name is Malin and I wrote my master's thesis in cognitive science in collaboration with Don Efficace, a young evaluator organization building their evaluation process to find the most effective charities in France. Together, we set out to explore the concept of noise (see below) in charity evaluations. Many researchers from other evaluator organizations contributed to this endeavor by responding to my survey or participating in interviews.
This post serves to summarize my research for them and anyone else who is interested in the topic - have a good read!
TL:DR
Noise, as defined by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein[1], refers to the unwanted variability in judgments caused by cognitive biases. In charity evaluations, this inconsistency can lead to unreliable recommendations, which can significantly affect the allocation of funds and erode donor trust. Given the complex nature of charity evaluations, noise is likely to occur, making it crucial to address in order to ensure consistent and effective decision-making.
Several strategies from other fields have been found effective in reducing noise and can be adapted for charity evaluations:
1. Implement Decision Guidelines and Scales:
Break down evaluations into clear criteria.
Use scales with anchors and descriptors for consistent assessments.
Consider comparative scales to reduce bias in subjective judgments.
2. Adopt Aggregation Strategies:
Encourage multiple independent estimates from researchers for cost-effectiveness analyses to improve accuracy.
Alternatively, use the options adapted to individuals, where two guesses from the same person are averaged.
3. Use the Mini-Delphi Method:
Structure discussions around initial independent estimates, followed by collective deliberation and revised judgments.
Future research should focus on measuring noise levels in charity evaluations and testing these strategies' effectiveness. Collaborating with other evaluator organizations can provide valuable insights and help design low-noise processes.
Introduction: Noise
In the context of my master's thesis, I explored the role of "noise" in charity evaluations. In the context of decision-making, the term noise was popularized by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein[1]. Their work has significantly advanced the application of cognitive sciences to real-life scenarios by demonstrating some of the tangible impacts cognitive biases can have on decision-making.
Specifically, they show how cognitive biases may lead to unwanted variability in judgments, which they call noise. I conducted three studies, a review of online information, a study, and interviews to investigate how noise-reduction strategies from the literature could apply to charity evaluations and which recommendations can be derived for Don Efficace. In this text, I summarize my findings as they may be relevant to charity evaluators.
If you want to know more, I invite you to read my thesis as well as "Noise: a flaw in human judgment" by Kahneman et al.[1].
The text will be structured as follows: First I introduce noise and why it matters in charity evaluations. Then I will present strategies that have been found to reduce noise in other fields that involve complex judgments, like judicial sentencing, medical diagnoses, or hiring decisions. For each of the strategies, I add the results from my research, setting them into the charity evaluation context. Lastly, I will give an outlook on what future research efforts in the field may look like.
Noise in Charity Evaluations
You may be familiar with cognitive biases like the confirmation bias, the halo effect, desirability bias, or the anchoring effect and how they can predi...
Hi! My name is Malin and I wrote my master's thesis in cognitive science in collaboration with Don Efficace, a young evaluator organization building their evaluation process to find the most effective charities in France. Together, we set out to explore the concept of noise (see below) in charity evaluations. Many researchers from other evaluator organizations contributed to this endeavor by responding to my survey or participating in interviews.
This post serves to summarize my research for them and anyone else who is interested in the topic - have a good read!
TL:DR
Noise, as defined by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein[1], refers to the unwanted variability in judgments caused by cognitive biases. In charity evaluations, this inconsistency can lead to unreliable recommendations, which can significantly affect the allocation of funds and erode donor trust. Given the complex nature of charity evaluations, noise is likely to occur, making it crucial to address in order to ensure consistent and effective decision-making.
Several strategies from other fields have been found effective in reducing noise and can be adapted for charity evaluations:
1. Implement Decision Guidelines and Scales:
Break down evaluations into clear criteria.
Use scales with anchors and descriptors for consistent assessments.
Consider comparative scales to reduce bias in subjective judgments.
2. Adopt Aggregation Strategies:
Encourage multiple independent estimates from researchers for cost-effectiveness analyses to improve accuracy.
Alternatively, use the options adapted to individuals, where two guesses from the same person are averaged.
3. Use the Mini-Delphi Method:
Structure discussions around initial independent estimates, followed by collective deliberation and revised judgments.
Future research should focus on measuring noise levels in charity evaluations and testing these strategies' effectiveness. Collaborating with other evaluator organizations can provide valuable insights and help design low-noise processes.
Introduction: Noise
In the context of my master's thesis, I explored the role of "noise" in charity evaluations. In the context of decision-making, the term noise was popularized by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein[1]. Their work has significantly advanced the application of cognitive sciences to real-life scenarios by demonstrating some of the tangible impacts cognitive biases can have on decision-making.
Specifically, they show how cognitive biases may lead to unwanted variability in judgments, which they call noise. I conducted three studies, a review of online information, a study, and interviews to investigate how noise-reduction strategies from the literature could apply to charity evaluations and which recommendations can be derived for Don Efficace. In this text, I summarize my findings as they may be relevant to charity evaluators.
If you want to know more, I invite you to read my thesis as well as "Noise: a flaw in human judgment" by Kahneman et al.[1].
The text will be structured as follows: First I introduce noise and why it matters in charity evaluations. Then I will present strategies that have been found to reduce noise in other fields that involve complex judgments, like judicial sentencing, medical diagnoses, or hiring decisions. For each of the strategies, I add the results from my research, setting them into the charity evaluation context. Lastly, I will give an outlook on what future research efforts in the field may look like.
Noise in Charity Evaluations
You may be familiar with cognitive biases like the confirmation bias, the halo effect, desirability bias, or the anchoring effect and how they can predi...