The Nonlinear Library EA - AI Welfare Debate Week retrospective by Toby Tremlett
Sep 18, 2024
09:27
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Welfare Debate Week retrospective, published by Toby Tremlett on September 18, 2024 on The Effective Altruism Forum.
I wrote this retrospective to be shared internally in CEA - but in the spirit of more open communication, I'm sharing it here as well. Note that this is a review of the event considered as a product, not a summary or review of the posts from the week.
If you have any questions, or any additional feedback, that'd be appreciated! I'll be running another debate week soon, and feedback has already been very helpful in preparing for it.
Also, feedback on the retro itself is appreciated- I'd ideally like to pre-register my retros and just have to fill in the graphs and conclusions once the event actually happens, so suggesting data we should measure/ questions I should be asking would be very helpful for making better retro templates.
How successful was the event?
In my OKRs (Objectives and Key Results- AKA, my goals for the event), I wanted this event to:
Have 50 participants, with "participant" being anyone taking an event-related action such as voting, commenting, or posting.
We did an order of magnitude better than 50. Over 558 people voted during the week, and 27 authors wrote or co-wrote at least one post.
Change people's minds. I wanted the equivalent of 25 people changing their minds by 25% of the debate slider.
We did twice as well as I hoped here- 53 unique users made at least one mind change of 0.25 delta (representing 25% of the slider) or more.
Therefore, on our explicit goals, this event was successful . But how successful was it based on our other, non-KR goals and hopes?
Some other goals that we had for the event- either in the ideation phase, or while it was ongoing, were:
Create more good content on a particularly important issue to EAs.
Successful.
Increase engagement.
Seems unsuccessful.
Bring in some new users.
Not noticeably successful.
Increase messaging.
Not noticeably successful.
In the next four sections, I examine each of these goals in turn.
More good content
We had 28 posts with the debate week tag, with 7 being at or above 50 karma. Of the 7, all but one (JWS's thoughtful critique of the debate's framing) were from authors I had directly spoken to or messaged about the event.
Compared to Draft Amnesty Week (which led to posts from 42 authors, and 10 posts over 50 karma) this isn't that many- however, I think we should count these posts as ex ante more valuable because of their focus on a specific topic.
Ex-post, it's hard to assess how valuable the posts were. None of the posts had very high karma (i.e. the highest was 77). However, I did curate one of the posts, and a couple of others were considered for curation. I would be interested to hear takes from readers about how valuable the posts were - did any of them change your mind, lead to a collaboration, or cause you to think more about the topic?
Engagement
How much engagement did the event get?
In total, debate week posts got 127 hours of engagement during the debate week (or 11.6% of total engagement), and 181 hours from July 1-14 (debate week and the week after), 7.5% of that fortnight's engagement hours.
Did it increase total daily hours of engagement?
Note: Discussion of Manifest controversies happened in June, and led to higher engagement hours per day in the build up to the event. Important dates: June 17: 244 comments, June 18: 349 comments, June 20: 33 comments, June 25: 38 comments
It doesn't look as if the debate week meaningfully increased daily engagement. The average daily engagement for the week after the event is actually higher, although the 3rd day of the event (July 3rd- the day I mentioned that the event was ongoing in the EA Digest) remains the highest hours of engagement between July 1st and the date I'm writing this, August 21st.
Did it get us new us...
I wrote this retrospective to be shared internally in CEA - but in the spirit of more open communication, I'm sharing it here as well. Note that this is a review of the event considered as a product, not a summary or review of the posts from the week.
If you have any questions, or any additional feedback, that'd be appreciated! I'll be running another debate week soon, and feedback has already been very helpful in preparing for it.
Also, feedback on the retro itself is appreciated- I'd ideally like to pre-register my retros and just have to fill in the graphs and conclusions once the event actually happens, so suggesting data we should measure/ questions I should be asking would be very helpful for making better retro templates.
How successful was the event?
In my OKRs (Objectives and Key Results- AKA, my goals for the event), I wanted this event to:
Have 50 participants, with "participant" being anyone taking an event-related action such as voting, commenting, or posting.
We did an order of magnitude better than 50. Over 558 people voted during the week, and 27 authors wrote or co-wrote at least one post.
Change people's minds. I wanted the equivalent of 25 people changing their minds by 25% of the debate slider.
We did twice as well as I hoped here- 53 unique users made at least one mind change of 0.25 delta (representing 25% of the slider) or more.
Therefore, on our explicit goals, this event was successful . But how successful was it based on our other, non-KR goals and hopes?
Some other goals that we had for the event- either in the ideation phase, or while it was ongoing, were:
Create more good content on a particularly important issue to EAs.
Successful.
Increase engagement.
Seems unsuccessful.
Bring in some new users.
Not noticeably successful.
Increase messaging.
Not noticeably successful.
In the next four sections, I examine each of these goals in turn.
More good content
We had 28 posts with the debate week tag, with 7 being at or above 50 karma. Of the 7, all but one (JWS's thoughtful critique of the debate's framing) were from authors I had directly spoken to or messaged about the event.
Compared to Draft Amnesty Week (which led to posts from 42 authors, and 10 posts over 50 karma) this isn't that many- however, I think we should count these posts as ex ante more valuable because of their focus on a specific topic.
Ex-post, it's hard to assess how valuable the posts were. None of the posts had very high karma (i.e. the highest was 77). However, I did curate one of the posts, and a couple of others were considered for curation. I would be interested to hear takes from readers about how valuable the posts were - did any of them change your mind, lead to a collaboration, or cause you to think more about the topic?
Engagement
How much engagement did the event get?
In total, debate week posts got 127 hours of engagement during the debate week (or 11.6% of total engagement), and 181 hours from July 1-14 (debate week and the week after), 7.5% of that fortnight's engagement hours.
Did it increase total daily hours of engagement?
Note: Discussion of Manifest controversies happened in June, and led to higher engagement hours per day in the build up to the event. Important dates: June 17: 244 comments, June 18: 349 comments, June 20: 33 comments, June 25: 38 comments
It doesn't look as if the debate week meaningfully increased daily engagement. The average daily engagement for the week after the event is actually higher, although the 3rd day of the event (July 3rd- the day I mentioned that the event was ongoing in the EA Digest) remains the highest hours of engagement between July 1st and the date I'm writing this, August 21st.
Did it get us new us...