ChinAI Newsletter

“ChinAI #237: Safety Benchmarks for Chinese Large Models” by Jeffrey Ding

Sep 18, 2023

Jeffrey Ding discusses the release of the SuperCLUE-Safety benchmark for Chinese large models, addressing the urgency of safe large models and China's AI landscape. The podcast explores safety rankings, AI safety benchmarks in China, industry exploitation, and reactions to Falcon 180B launch.

07:34

Creator website

Episode guests

Jeffrey Ding

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

SuperCLUE-Safety challenges Chinese large models with diverse test scenarios on responsible AI and instruction attacks.

Closed source models excel over open source in safety assessments, highlighting the need for improved AI safety measures in China.

Deep dives

Safety Benchmark for Chinese Large Models: SuperCLUE Safety

The SuperCLUE team released a safety benchmark for Chinese large models, challenging them with 2,456 pairs of questions across three safety categories: traditional safety, responsible AI, and instruction attacks. The examples provided in the podcast highlighted specific test scenarios like questioning a model's response to criminal behavior information disclosure, psychological manipulation techniques, and identifying illegal activities. The results revealed that closed source models outperformed open source ones in safety evaluations, with Chinese models narrowing the safety gap against international counterparts, contrary to prior assumptions.

Exploring Safety Benchmarks for Chinese Large Models

5min

Urgency of Safe Large Models and China's AI Landscape

3min

Subtitle: SuperCLUE-Safety, the first Chinese large-model multi-round adversarial safety benchmark, is released..

Greetings from a world where…

for the rest of the college football season, this status update will be devoted to tracking the Iowa Hawkeye offense's march to mediocrity

…As always, the searchable archive of all past issues is here. Please please subscribe here to support ChinAI under a Guardian/Wikipedia-style tipping model (everyone gets the same content but those who can pay support access for all AND compensation for awesome ChinAI contributors).

Feature Translation: SuperCLUE-Safety

Context: Every two months or so, we’ve been checking in with the SuperCLUE rankings, which aim to benchmark large language models from Chinese and international labs along different dimensions. In the previous update to the SuperCLUE benchmark, we saw Baidu's ErnieBot soar up the rankings, on the strength of its performance with Chinese-language particularities (e.g. idioms). This past week, the SuperCLUE team released a safety benchmark (link to [...]

---

First published:
September 18th, 2023

Source:
https://chinai.substack.com/p/chinai-237-safety-benchmarks-for

---

Narrated by TYPE III AUDIO.