Generating coherent long stories requires outlining and iterative generation for narrative coherence.
Models like RE3 use detailed storytelling plans and control mechanisms to enhance narrative fidelity.
Deep dives
Challenges in Generating Coherent Long Stories
Generating long stories poses challenges as existing models like GPT-3 lose coherence in longer texts. Models struggle to maintain a coherent overarching plot in stories over 2,000 words, leading to random event walkthroughs. Previous models like GPT-2 and Chat-GPT attempt longer stories but often lack coherent plots and tend to 'tell rather than show.' Maintaining coherence becomes harder as stories lengthen, hindering progress towards novel-scale storytelling.
Historical Exploration and Approaches to Coherence in Story Generation
Prior work focused on shorter story generations, addressing problems like repetition and logical consistency. Structured approaches aimed at logical sentence flow were common. However, with the advent of models like GPT-3, problems shifted to maintaining coherence in longer narratives. Older models required manual coherence maintenance, contrasting with current models that rely on large-scale pre-training for longer story generation.
Evolution of Narrative Complexity and Long-Form Generation
The narrative complexity challenge remains, with long-range narrative coherence still a research frontier. The shift to novel-scale generation prompts a need for models to remember context over extensive texts. Current frameworks like RE3 aim to enhance planning and control to support generating lengthy, structured narratives with deep coherence and minimal repetition.
Enhancements in Planning and Control for Long-Form Narratives
The RE3 framework implements detailed storytelling plans and control mechanisms to improve narrative fidelity. By leveraging planning modules to structure stories and embedding control measures to guide generation, RE3 seeks to enhance storytelling coherence. Future research aims to advance the balance between narrative length, detail, and coherence in automated storytelling.
How can we generate coherent long stories from language models? Ensuring that the generated story has long range consistency and that it conforms to a high level plan is typically challenging. In this episode, Kevin Yang describes their system that prompts language models to first generate an outline, and iteratively generate the story while following the outline and reranking and editing the outputs for coherence. We also discussed the challenges involved in evaluating long generated texts.
Kevin Yang is a PhD student at UC Berkeley.
Kevin's webpage: https://people.eecs.berkeley.edu/~yangk/
Papers discussed in this episode:
1. Re3: Generating Longer Stories With Recursive Reprompting and Revision (https://www.semanticscholar.org/paper/Re3%3A-Generating-Longer-Stories-With-Recursive-and-Yang-Peng/2aab6ca1a8dae3f3db6d248231ac3fa4e222b30a)
2. DOC: Improving Long Story Coherence With Detailed Outline Control (https://www.semanticscholar.org/paper/DOC%3A-Improving-Long-Story-Coherence-With-Detailed-Yang-Klein/ef6c768f23f86c4aa59f7e859ca6ffc1392966ca)
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
