The Problem With Peer Review in Computer Science?

In the last episode from my own reviewing hall of shame, I mentioned the coincidence of attending a webinar where Edward Lee talked about the ‘toxic culture of rejection’, based on a blog post he write in 2022. In this episode we hear from Edward directly and discuss the culture of rejection, in CS especially, the problems with peer reviewing, the nature of conferences, and how we might approach reviewing differently and start to change the culture around publications, acceptance rates and evaluations without losing quality standards.

His bio: Edward A. Lee has been working on embedded software systems for more than 40 years. After studying and working at Yale, MIT, and Bell Labs, he landed at Berkeley, where he is now Professor of the Graduate School in EECS. His research is focused on cyber-physical systems. He leads the open-source software project Lingua Franca and previously Ptolemy II, is a coauthor of textbooks on embedded systems, signals and systems, digital communications, and philosophical and social implications of technology. His current research is focused on a polyglot coordination language for distributed real-time systems called Lingua Franca that combines features of discrete-event modeling, synchronous languages, and actors.

“There's a classical view of the purpose of publication, which is essentially to add knowledge to the archive […] I feel…we should understand that the primary purpose of publication is to communicate with other humans.”

“A scientific discipline progresses in a very cultural way… it's really about a human culture of developing and evolving, and, and it tends to evolve in a very chaotic way.”

“Institutions should be prepared to do their own evaluation.”

“The criterion should be, what is informative, interesting, and potentially valuable and useful to the community.”

“Everyone involved in the [review] process knows that we're dealing with other human beings. And the phrase that I've tried to use, […]  is to pretend that this paper was written by their sister. How would that change [how you assess the paper]?”  

Overview (times approximate): [FULL TRANSCRIPT for download]

 0:05 Welcome to Changing Academic Life.

0:30 Intro to the episode

03:03 Welcome Edward Lee starts to introduce himself

06:04 How the faculty position selection process has changed over time  - hypercompetition.

07:31 The gradual change towards hypercompetition, the randomness of the review process and the role of luck in getting papers accepted

11:41 The problem of the conference peer review process in no real opportunity for dialogue compared to journal review processes

12:38 This has the effect of a certain amount of randomness and conservatism.

14:44 What are conferences for? The importance of informal communication orthogonal to the  publications

17:02 The obsessive focus on novelty

18:25 The purpose of publication, how science progresses and the importance of dialogue and culture.

22:59 The challenge of publishing multidisciplinary and systems papers

26:44 Playing the game the right way

31:17 The randomness of reviews and factors around this in program committees

37:30 The  tensions and conflicts of  selective conferences for rankings

38:07 Learning from how other communities work re conferences and journals

40:21 The association of publications with funding to attend a conference

44:23 Institutions should be prepared to do their own evaluations not outsource them to reviewers

47:55 What we should be looking for when evaluating papers

51:16 The advantages and challenges of the double blind review process

55:38 Reminder that we are dealing with humans as reviewers

59:35 Arguing for getting rid of acceptance rate

01:04:25 Wrapping up

Related links:

[Blog post] Edward A. Lee, The Toxic Culture of Rejection in Computer Science, 22 Aug 2022, SIGBED https://sigbed.org/2022/08/22/the-toxic-culture-of-rejection-in-computer-science/

[Newsletter article – interview] Anna Kramer, How I decided to call out the ‘toxic’ culture’ of CS, 7 Sept 2022, Protocol.  https://www.protocol.com/workplace/how-i-decided-edward-lee

Note: we talk about blind reviewing at some point. While this has  been the standard terminology used for a long time about our anonymous review processes, I appreciate that this terminology can be experienced as ableist and perpetuating harmful stereotypes. See the following blog post:

[Blog article] Rachel Ades, An end to “Blind Review”, 20 Feb 2020, APA Online. https://blog.apaonline.org/2020/02/20/an-end-to-blind-review/

Acknowledgements:

Edward Lee photo: Photo credit by Rusi Mchedlishvili