The Resolution of the Winograd Schema Challenge

In this chapter, they discuss the resolution of the Winograd Schema Challenge and the metric of success. They mention that originally, if a model could resolve 90 percent of the cases, it would be considered human level. However, there hasn't been rigorous human testing, so it's uncertain what the human level is. They also talk about alternative metrics that arose later on, such as investigating consistency in solving pairs of examples. They conclude that these alternative metrics don't significantly change the interpretation of whether the challenge was solved or not, as the bar for success was somewhat arbitrary.