Edward Lee on the toxic culture of peer review
Changing Academic Life
The Problem With Peer Reviewing Conferences
I think part of the problem we face as a community is that there is no obvious way to reconcile all these conflicting goals, right? And I think that it would behoove us to look at how other communities work. The conferences are not, they're not even, you don't really even put your conference publications on your resume much. You know, the purpose of the reviewing in their conferences is just to keep the charlatans out.
In the last episode from my own reviewing hall of shame, I mentioned the coincidence of attending a webinar where Edward Lee talked about the ‘toxic culture of rejection’, based on a blog post he write in 2022. In this episode we hear from Edward directly and discuss the culture of rejection, in CS especially, the problems with peer reviewing, the nature of conferences, and how we might approach reviewing differently and start to change the culture around publications, acceptance rates and evaluations without losing quality standards.
His bio: Edward A. Lee has been working on embedded software systems for more than 40 years. After studying and working at Yale, MIT, and Bell Labs, he landed at Berkeley, where he is now Professor of the Graduate School in EECS. His research is focused on cyber-physical systems. He leads the open-source software project Lingua Franca and previously Ptolemy II, is a coauthor of textbooks on embedded systems, signals and systems, digital communications, and philosophical and social implications of technology. His current research is focused on a polyglot coordination language for distributed real-time systems called Lingua Franca that combines features of discrete-event modeling, synchronous languages, and actors.
“There's a classical view of the purpose of publication, which is essentially to add knowledge to the archive […] I feel…we should understand that the primary purpose of publication is to communicate with other humans.”
“A scientific discipline progresses in a very cultural way… it's really about a human culture of developing and evolving, and, and it tends to evolve in a very chaotic way.”
“Institutions should be prepared to do their own evaluation.”
“The criterion should be, what is informative, interesting, and potentially valuable and useful to the community.”
“Everyone involved in the [review] process knows that we're dealing with other human beings. And the phrase that I've tried to use, […] is to pretend that this paper was written by their sister. How would that change [how you assess the paper]?”
Overview (times approximate): [FULL TRANSCRIPT for download]
0:05 Welcome to Changing Academic Life.
0:30 Intro to the episode
03:03 Welcome Edward Lee starts to introduce himself
06:04 How the faculty position selection process has changed over time - hypercompetition.
07:31 The gradual change towards hypercompetition, the randomness of the review process and the role of luck in getting papers accepted
11:41 The problem of the conference peer review process in no real opportunity for dialogue compared to journal review processes
12:38 This has the effect of a certain amount of randomness and conservatism.
14:44 What are conferences for? The importance of informal communication orthogonal to the publications
17:02 The obsessive focus on novelty
18:25 The purpose of publication, how science progresses and the importance of dialogue and culture.
22:59 The challenge of publishing multidisciplinary and systems papers
26:44 Playing the game the right way
31:17 The randomness of reviews and factors around this in program committees
37:30 The tensions and conflicts of selective conferences for rankings
38:07 Learning from how other communities work re conferences and journals
40:21 The association of publications with funding to attend a conference
44:23 Institutions should be prepared to do their own evaluations not outsource them to reviewers
47:55 What we should be looking for when evaluating papers
51:16 The advantages and challenges of the double blind review process
55:38 Reminder that we are dealing with humans as reviewers
59:35 Arguing for getting rid of acceptance rate
01:04:25 Wrapping up
Related links:
[Blog post] Edward A. Lee, The Toxic Culture of Rejection in Computer Science, 22 Aug 2022, SIGBED https://sigbed.org/2022/08/22/the-toxic-culture-of-rejection-in-computer-science/
[Newsletter article – interview] Anna Kramer, How I decided to call out the ‘toxic’ culture’ of CS, 7 Sept 2022, Protocol. https://www.protocol.com/workplace/how-i-decided-edward-lee
Note: we talk about blind reviewing at some point. While this has been the standard terminology used for a long time about our anonymous review processes, I appreciate that this terminology can be experienced as ableist and perpetuating harmful stereotypes. See the following blog post:
[Blog article] Rachel Ades, An end to “Blind Review”, 20 Feb 2020, APA Online. https://blog.apaonline.org/2020/02/20/an-end-to-blind-review/
Acknowledgements:
Edward Lee photo: Photo credit by Rusi Mchedlishvili
The AI-powered Podcast Player
