Grammar Girl Quick and Dirty Tips for Better Writing What a ‘Science' magazine experiment says about the future of AI in journalism, with Abigail Eisenstadt
Nov 6, 2025
Abigail Eisenstadt, a senior science writer at Science magazine, discusses her team's year-long experiment assessing ChatGPT's ability to summarize research papers. She explains the difference between AI's transcription of studies and the essential contextual translation needed for accurate science communication. The findings reveal AI struggles with context while performing well on simpler tasks. Eisenstadt also shares insights on the potential benefits of AI tools for less-skilled writers and the interest in future experiments with evolving models.
AI Snips
Chapters
Transcript
Episode notes
Transcribing Vs Translating Scientific Papers
- Science's experiment found ChatGPT could accurately transcribe studies but failed to contextualize them for readers.
- Abigail Eisenstadt calls that missing step "translating," which is crucial for science journalism.
Year-Long, Consistent Test Run
- The team ran a year-long experiment summarizing roughly 64 papers across 40–50 weeks.
- They kept prompts consistent to observe model behavior over time rather than optimizing prompts.
Keep Prompts Simple To Test Realistic Use
- Use stable, human-style prompts only if you want to evaluate AI as a human proxy instead of optimizing outputs.
- Abigail's team deliberately avoided advanced prompt engineering to simulate non-expert use.