Back in 2018, researchers from Facebook AI developed a new method to improve story generation through hierarchical modeling. Their approach mimics how people plan out narratives. While significant developments have occurred in language generation, it is worth exploring this technique as several current projects leverage these techniques.
The key innovation is generating a short premise first, then expanding that premise into a full story. Take the premise "A knight goes on a quest to save the kingdom." From this high-level summary, a system can flesh out details - the specific characters, events, and dialogue - while staying focused on the overarching plot.
This technique helps in two ways. First, the premise acts like an outline, guiding the story generation process. Second, conditioning the story on the premise makes it easier for the AI to stay on topic. Without such grounding, AI systems tend to lose coherence as they generate text word-by-word.
To train and test hierarchical story generation, the researchers built a new dataset using the r/WritingPrompts subreddit. This online community shares story premises, or prompts, that inspire other users to write original tales. Drawing on over 300,000 prompt-story pairs, the dataset captures diverse genres and narrative styles.
The researchers' AI system first generates a short prompt, similar to a human providing a premise. It then passes this prompt to a second model that expands it into a full story. Both steps use sequence-to-sequence neural networks, which translate an input sequence into target text.
To improve story coherence, the researchers introduced two key innovations.
- First, they trained a standard sequence-to-sequence neural network model on the story generation task. This model learns to generate fluent stories, but often ignores the premise and fails to maintain consistency with it.
- Next, they trained a second sequence-to-sequence model, but this time provided the hidden state outputs of the first pre-trained model to the second model during training. In other words, the second model learns on top of the representations already learned by the first model. It has access to the pre-trained model's outputs. This encourages the second model to focus specifically on relating the story back to the premise, rather than just improving language modeling in general
By "fusing" the second model with the first pre-trained model in this way, the researchers aim to improve coherence between the premise and final story. The second model builds on top of the first to better maintain relevance.
Experiments found these advances substantially boosted performance. The AI's stories scored higher in human evaluations for coherence, relevance to the prompt, and overall quality compared to baseline systems. The gated self-attention enabled referring back to any previous part of the story. And model fusion encouraged tighter connections between the premise and story.
While far from perfect, these results demonstrate AI's increasing capacity for controllable, long-form text generation. The hierarchical approach mimics how people first conceptualize, then craft, narratives. Such human-inspired techniques will be key to teaching machines to tell truly compelling tales spanning paragraphs or pages.
The researchers highlight several directions for improvement. The premises generated by the AI tend to be generic, lacking the creativity of human prompts. Repetition remains an issue when expanding premises into stories. And problems like dropped pronouns persist.
Nonetheless, this work moves neural story writing systems in a promising direction. As models strengthen their understanding of narrative cause-and-effect, characters, and more, their power as digital storytellers will grow. Hierarchical modeling that mirrors human planning seems a fitting way to imbue AI with our innate gift for spinning both short yarns and epics.
Sources:
Hierarchical Neural Story Generation from ACL on Vimeo.
