The Nonlinear Library EA - Data from the 2023 EA Forum user survey by Sarah Cheng
Jul 26, 2024
14:16
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Data from the 2023 EA Forum user survey, published by Sarah Cheng on July 26, 2024 on The Effective Altruism Forum.
The purpose of this post is to share data from a survey that the EA Forum team ran last year. Though we used this survey as one of many sources of information for internal analyses, I did not include any particular takeaways from this survey data in this post. I leave that as an exercise for the reader.
Overview
In August 2023, the EA Forum team ran a survey to learn more about how people use the Forum and how the Forum impacted them. We got 609 valid responses.
Thank you to everyone who responded - we really appreciate you taking the time. The results have been important for helping us understand the ways that the Forum creates value and disvalue that are otherwise hard for us to track. We've used it to evaluate the impact of the Forum and the marginal impact of our work, update our team's strategy, and prioritize the work we've done in the past 12 months.
The person who ran the survey and wrote up the analysis is no longer at CEA, but I figured people might be interested in the results of the survey, so I'm sharing some of the data in this post. Most of the information here comes from that internal analysis, but when I use "I" that is me (Sarah) editorializing.
This post is not comprehensive, and does not include all relevant data. I did not spend time double checking any of the information from that analysis. We plan to run another (updated) survey soon for 2024.
Some Forum usage data, for context
The Forum had 4.5k monthly active and 13.7k annually active logged in users in the 12 months ending on Sept 4 2023. We estimate that the total number of users was closer to 20-30k (since about 50% of traffic is logged out).
Here's a breakdown of usage data for logged-in users in those 12 months:
13.7k distinct logged in users
8.5k users with 3 distinct days of activity
5.7k users with 10 distinct days of activity
3.1k users who were active during of all months
1.7k users who were active during of all weeks
388 users who were active during on of all days
4.4k distinct commenters
171 distinct post authors
It's important to note that August 2022 - August 2023 was a fairly unusual time for EA, so while you can (and we have) used this survey data to estimate things like "the value the Forum generates per year", you might think that August 2023 - August 2024 is a more typical year, and so the data from the next survey may be more representative.
Demographic reweighting[1]
Rethink Priorities helped us with the data analysis, which included adjusting the raw data by weighting the responses to try to get a more representative view of the results. All charts below include both the raw and weighted[2] data.
The weighting factors were:
1. Whether the respondent had posted (relative to overall Forum usage)
2. Whether the respondent had commented (relative to overall Forum usage)
3. How frequently the respondent used the Forum (relative to overall Forum usage)
4. The respondent's EA engagement level (relative to the Forum statistics from the 2020 EA Survey)
5. The respondent's gender (relative to the Forum statistics from the 2020 EA Survey)
Some effects of the reweighting:
A significantly higher proportion of respondents have posted or commented on the Forum, relative to actual overall Forum usage, so reweighting decreases those percentages and the percentages of other actions (such as voting on karma).
A plurality (around 45%) of respondents said they visit the Forum about 1-2 times a week. This is more frequent than the overall Forum population, so reweighting decreases things like the percentage of users who applied for a job due to the Forum, and the mean rating of "significantly changed your thinking".
Overall, respondents tended to be more highly engaged than the ...
The purpose of this post is to share data from a survey that the EA Forum team ran last year. Though we used this survey as one of many sources of information for internal analyses, I did not include any particular takeaways from this survey data in this post. I leave that as an exercise for the reader.
Overview
In August 2023, the EA Forum team ran a survey to learn more about how people use the Forum and how the Forum impacted them. We got 609 valid responses.
Thank you to everyone who responded - we really appreciate you taking the time. The results have been important for helping us understand the ways that the Forum creates value and disvalue that are otherwise hard for us to track. We've used it to evaluate the impact of the Forum and the marginal impact of our work, update our team's strategy, and prioritize the work we've done in the past 12 months.
The person who ran the survey and wrote up the analysis is no longer at CEA, but I figured people might be interested in the results of the survey, so I'm sharing some of the data in this post. Most of the information here comes from that internal analysis, but when I use "I" that is me (Sarah) editorializing.
This post is not comprehensive, and does not include all relevant data. I did not spend time double checking any of the information from that analysis. We plan to run another (updated) survey soon for 2024.
Some Forum usage data, for context
The Forum had 4.5k monthly active and 13.7k annually active logged in users in the 12 months ending on Sept 4 2023. We estimate that the total number of users was closer to 20-30k (since about 50% of traffic is logged out).
Here's a breakdown of usage data for logged-in users in those 12 months:
13.7k distinct logged in users
8.5k users with 3 distinct days of activity
5.7k users with 10 distinct days of activity
3.1k users who were active during of all months
1.7k users who were active during of all weeks
388 users who were active during on of all days
4.4k distinct commenters
171 distinct post authors
It's important to note that August 2022 - August 2023 was a fairly unusual time for EA, so while you can (and we have) used this survey data to estimate things like "the value the Forum generates per year", you might think that August 2023 - August 2024 is a more typical year, and so the data from the next survey may be more representative.
Demographic reweighting[1]
Rethink Priorities helped us with the data analysis, which included adjusting the raw data by weighting the responses to try to get a more representative view of the results. All charts below include both the raw and weighted[2] data.
The weighting factors were:
1. Whether the respondent had posted (relative to overall Forum usage)
2. Whether the respondent had commented (relative to overall Forum usage)
3. How frequently the respondent used the Forum (relative to overall Forum usage)
4. The respondent's EA engagement level (relative to the Forum statistics from the 2020 EA Survey)
5. The respondent's gender (relative to the Forum statistics from the 2020 EA Survey)
Some effects of the reweighting:
A significantly higher proportion of respondents have posted or commented on the Forum, relative to actual overall Forum usage, so reweighting decreases those percentages and the percentages of other actions (such as voting on karma).
A plurality (around 45%) of respondents said they visit the Forum about 1-2 times a week. This is more frequent than the overall Forum population, so reweighting decreases things like the percentage of users who applied for a job due to the Forum, and the mean rating of "significantly changed your thinking".
Overall, respondents tended to be more highly engaged than the ...