Now Next Later AI - Blog ##AIStoryBrain

The Promise of Frozen Language Models

Fri, 11 Aug 2023 10:04:48 +1000

In recent years, artificial intelligence has taken great leaps forward thanks to large language models (LLMs) - AI systems trained on massive amounts of text data that can understand language and generate human-like text. Companies like Google, Microsoft, and startups like OpenAI and Anthropic have invested heavily in developing ever-larger LLMs with billions or even trillions of parameters.

However, once these giant LLMs are trained, companies face a dilemma - whether to "fine-tune" the model by further training it on specific tasks, or keep the model "frozen" without any changes. Fine-tuning allows the LLM to specialize and achieve state-of-the-art performance on specialized tasks. But it comes at a high cost - computationally expensive retraining, reduced versatility, and forgetting of previous capabilities.

In their research paper, AI21 Labs demonstrates that frozen LLMs have untapped potential that can match or exceed fine-tuning approaches, without these downsides. They present three new techniques to effectively "stand on the shoulders" of frozen giants:

1. Input-Dependent Prompt Tuning

Large language models are adept at understanding natural language, but they don't automatically know how to perform specific tasks like answering questions or summarizing text.However, their capabilities can be unlocked using prompt tuning.

The key idea behind prompt tuning is that providing the right prompt text before the input steers the language model towards the desired task.It's like giving the model instructions on how to process the upcoming input.

For example, if we want the language model to answer questions based on a passage of text, we can prepend the input with a prompt like:

"Answer the following question based only on the passage below:"

[Text Passage]

[Question]

This tunes the model to approach the upcoming input as a question answering task.The prompt acts like an adapter, steering the versatile model to useful behaviors without any training or fine-tuning.

So prompt tuning just means optimizing the wording of these instruction prompts for each task to get the best performance from the frozen language model.It's like learning how to most effectively communicate with and direct the model.

The key innovation from AI21 Labs was making prompt tuning input-dependent. Rather than using one static prompt per task, they trained a small neural network to generate custom prompts tailored to each specific input.

This input-dependent prompting allowed a single frozen language model to master over 100 diverse tasks, from question answering to summarization to sentiment analysis, matching extensive fine-tuning without degradation.

The prompts serve as lightweight yet powerful steering instructions that can specialize a frozen model on the fly based on the input.It's like having a dynamic adapter that configures the model differently for each unique situation.

2. Huge Frozen Readers for Question Answering

In open-domain question answering, the AI system must answer questions by finding relevant information from a massive collection of text passages, like Wikipedia.

Typically, these systems use a smaller "reader" model to read through the relevant passages and figure out the answer. That's because even the largest language models can only process a limited amount of text at once.

But smaller reader models have less knowledge and reasoning ability than giant language models with billions or trillions of parameters. So they don't fully unlock the potential of these frozen giants.

AI21 Labs tackled this by adding a "re-ranking" stage to condense the most important information from the passages into a condensed form that fits into the giant frozen language model.

This allowed their 17 billion parameter model to read enough of the relevant context to match specialized reader models that were extensively fine-tuned for question answering.

In essence, the smaller re-ranking model acts like a search engine, retrieving and condensing the most useful knowledge to fit the limitations of the frozen giant.

This gives the huge frozen model access to all the relevant information it needs to apply its powerful reasoning abilities. The giants' knowledge and capabilities can be tapped without fine-tuning that risks degrading other skills.

It demonstrates how frozen language models have untapped potential that can be unlocked with the right surrounding components, like the re-ranking stage here. Their true capabilities can be accessed without resorting to extensive fine-tuning.

3. Recursive Application of a Single LLM

Typically, large language models are used to process an input query just once before generating an output response. The model reads the input, does its internal reasoning, and returns a single output.

But AI21 Labs found that recursively applying the model on its own outputs can actually improve performance. Essentially, the model refines and enhances its initial output by processing it again.

It's like having the model double-check its own work and refine its initial response. Humans often re-read what we initially wrote to improve the wording and fix errors. Recursively applying language models does something similar, but in an automated way.

To implement this, AI21 built a small 2-layer neural network "connector" that feeds the language model's output back into its input.

So the model first processes the original query as normal. But then the connector passes the model's initial output back into it as the new input. This triggers it to refine and enhance that initial output.

In tests for question answering, just two recursive passes through a 7 billion parameter model allowed it to match the performance of a much larger 17 billion parameter model.

Essentially, it nearly doubled the capabilities of the smaller model by re-applying it recursively. This shows how recursive application unlocks additional performance without requiring even larger pretrained models.

The connector module creates a feedback loop, allowing the model to re-process its own output and correct errors or improve phrasing, much like a human would. This technique amplifies the capabilities of a given model without expensive retraining or fine-tuning.

Business Implications

These techniques enable building capable AI systems on top of a single, frozen pretrained LLM instead of an array of specialized fine-tuned models. This offers tangible business benefits:

Cost Savings - Avoiding expensive training of multiple large models cuts costs. Just maintaining and serving one frozen LLM backbone provides economies of scale.
Simplicity - Relying on prompting and other external components is far simpler than intricately fine-tuning models. Less specialized engineering effort is required.
Flexibility - New capabilities can be added without interfering with existing ones. Fine-tuning risks degradation on previous tasks.
Efficiency - Recursive passing allows improving performance on-demand by re-applying the LLM only when beneficial. Bigger pretrained models must be applied to all inputs.

While fine-tuning revolutionized AI, endless model growth is impractical. Frozen language models present an alluring path forward - unlocking their full potential with the right neural "plug-ins" provides a scalable approach to building production AI systems.

Source:

S TANDING ON THE S HOULDERS OF G IANT F ROZEN L ANGUAGE M ODELS

Making AI More Useful and Reliable with Modular Systems: MRKL

Thu, 10 Aug 2023 21:21:30 +1000

In recent years, large language models (LLMs) like GPT-4 have shown impressive capabilities in generating human-like text and engaging in natural language conversations. However, LLMs also have some serious limitations that constrain their usefulness for real-world applications. To overcome these limitations, AI researchers have proposed a new type of AI system architecture called Modular Reasoning, Knowledge and Language (MRKL).

The Limitations of Large Language Models

While LLMs can produce remarkably fluent text, they often generate factual errors, nonsensical statements, and inconsistent responses. This happens because LLMs do not actually have any real understanding of the world or ability to reason - they just recognize patterns in the massive datasets they are trained on. As a result, LLMs lack:

Access to current, real-time information that is constantly changing, like stock prices or weather data. The pre-trained models only know what was in their training data.
Access to proprietary data like customer records that exist in a company's databases. LLMs cannot connect to external databases.
Ability to perform symbolic reasoning and math. They struggle with simple arithmetic and logical deductions.
Ability to learn major new capabilities without catastrophic forgetting. Fine-tuning LLMs on new datasets leads to losing their original skills.

These problems severely limit the reliability and usefulness of LLMs for practical business applications. Companies cannot deploy unreliable AI assistants that generate false information or nonsensical outputs.

Introducing Modular AI Systems

To address the weaknesses of monolithic LLMs, AI researchers have proposed breaking AI systems into modules with different capabilities that can work together:

Neural modules based on LLMs that handle natural language
Symbolic modules that perform logical reasoning and math
Access to external knowledge bases like databases and APIs

This is the idea behind Modular Reasoning, Knowledge and Language (MRKL) architectures. MRKL systems have a router module that analyzes incoming questions and routes them to the most appropriate module - either the core LLM, a symbolic module, or an external database.

Benefits of the Modular Approach

Modular AI architectures provide important benefits compared to monolithic LLMs:

Reliability through redundancy. If the core LLM fails, questions can be routed to more reliable modules.
Easy extensibility by adding modules without retraining the whole system.
Explainability, since it's clear which module produced an answer.
Up-to-date real-time data from external APIs and databases.
Secure access to proprietary data sources.
Improved reasoning abilities by combining neural networks and symbolic modules.
Avoidance of catastrophic forgetting. New skills don't override old ones.

This modularity and hybrid approach allows AI systems to leverage the strengths of different techniques while minimizing their weaknesses.

Challenges of Integrating Neural and Symbolic AI

A key challenge in building modular AI systems is integrating the neural LLM components with the symbolic reasoning modules. These two types of AI rely on completely different processing techniques - neural nets versus discrete logical operations.

Researchers have found that even extracting basic math problems from text for input to a calculator module requires specialized training to reach high accuracy. For example, the query "I lost one ball" needs to be recognized as a subtraction problem: X - 1 = ?.

By using large datasets of mathematically annotated text, modular AI systems can be trained to extract appropriate reasoning tasks with over 99% accuracy. But significant research is still required to handle more complex reasoning across modules. Integrating neural networks and symbolic systems remains an active area of investigation.

The Future of Modular AI

Modular architectures represent an exciting evolution in AI that combines the strengths of different techniques. Companies like Anthropic and AI21 Labs are actively developing modular AI platforms to provide businesses with safer and more usable AI assistants. While challenges remain, the future appears bright for this hybrid approach to artificial intelligence.

Source:

MRKL Systems

Enhancing AI with Symbolic Thinking

Thu, 10 Aug 2023 18:34:26 +1000

The rapid progress of artificial intelligence over the past decade owes much to a class of algorithms called transformer neural networks. Transformers gave rise to large language models (LLMs) like GPT-4 or Claude 2 that display impressive natural language abilities.

But as AI becomes more integrated into business processes, sole reliance on data-driven machine learning approaches like transformers may prove limiting. Researchers are exploring how to combine these powerful statistical models with neurosymbolic methods that incorporate logical reasoning and structure.

The result could be AI systems that blend raw pattern recognition power with human-like compositional generalization and interpretability.

The Rise of Large Language Models

Much of the current excitement around AI stems from the advances of LLMs over the past few years. Models like GPT-3, PaLM, and Google's LaMDA have shown the ability to generate human-like text, answer questions, and accomplish tasks from basic prompts.

LLMs owe their abilities to a neural network architecture called transformers. Transformers process text more holistically than previous recurrent neural networks. They capture long-range dependencies in language by attending to all words in a context.

Training transformers on massive text corpora like the internet produces universal language models. With enough data and compute, these models learn statistical representations that prove surprisingly versatile for language tasks.

Finetuning techniques allow specializing LLMs to specific applications by updating the models on task data. For example, a finetuned GPT-3 model can be adapted into a conversational chatbot or a code completion tool.

The broad capabilities of LLMs along with their ease of use via prompting led to widespread adoption. Startups like Anthropic and Cohere are commercializing LLMs for business use cases. Apps built on LLMs range from automating customer support to generating content to synthesizing code.

Limits of Language Models

But for all their progress, LLMs still suffer from key limitations. Most notably, they display limited compositional generalization outside the distribution of their training data. For example, a LLM trained on English text will struggle with novel sentence structures or made-up words.

Humans seamlessly compose known concepts into new combinations thanks to our intuitive understanding of language syntax and meaning. Neural networks have no such innate symbolic reasoning capabilities.

LLMs are also black boxes. They can generate plausible and useful text or code but offer no interpretable justification for their outputs. Lack of interpretability makes it hard to audit models or identify causes of failures.

Finally, the massive scale of data and compute required to train LLMs makes them environmentally costly. Requiring less data and smaller models would allow much wider deployment of AI technology.

Integrating Symbolic Representations

To overcome the limits of language models, researchers are finding ways to incorporate more logical reasoning. The aim is to complement the statistical learning with capabilities closer to human understanding.

One approach injects structured knowledge representations into the training process. For example, some methods jointly train the language model with a knowledge graph. Knowledge graphs are data structures that represent facts as networks of entities and relationships. They encode real-world knowledge in a machine-readable graph format with nodes for entities like people and edges for relationships like "employed at". This allows computers to automatically reason over millions of interconnected facts. Knowledge graphs help power many AI applications today including search, recommendations, and question answering. The knowledge graph acts like a symbolic memory bank to improve reasoning.

Other techniques draw inspiration from classic logic programming languages like Prolog. These languages represent knowledge as human-readable rules. By integrating them into the training, the aim is to bake in more systematic symbolic thinking.

Researchers are also finding ways to refine and check language model outputs using logical constraints. For instance, one idea runs the text through separate logic rules as an extra plausibility filter beyond the statistical patterns.

In each case, the goal is to guide, restrict, and enhance the pattern-finding abilities of language models with more deliberate symbolic reasoning. Just like humans blend intuitive thinking with logic, the hope is to achieve AI systems that integrate learned statistical correlations with structured symbolic representations.

The end result could be models that display more generalized reasoning abilities, while also producing outputs we can audit, validate, and explain.

Towards Hybrid Intelligence

Ultimately, the aim is achieving hybrid systems that integrate the complementary strengths of neural and symbolic AI. Some researchers argue that intelligence emerges from the interplay of two mechanisms:

Correlation-based pattern recognition that is data-driven and associative.
Model-based compositional generalization relying on structured representations and explicit rules.

Large transformer networks excel at the former while neurosymbolic methods specialize in the latter. Combining these two modes of reasoning could thus give rise to more human-like artificial intelligence.

The business implications of such hybrid AI systems are far-reaching. Logical components would allow verifying conclusions, checking ethical compliance, and generating step-by-step explanations. Incorporating domain constraints would reduce data needs and may lead to safer and less environmentally costly systems.

At the same time, retaining differentiable components preserves versatility, allows critiquing and updating symbolic knowledge, and facilitates integrating with downstream machine learning tasks.

Realizing this vision of integrated reasoning poses research challenges. Tradeoffs exist between symbolic interpretability and neural flexibility. Multi-component systems risk bottlenecks limiting end-to-end learning. Architectures that blur gradients across reasoning layers may be needed.

Nonetheless, the potential payoff for deployable, ethical, and broadly capable AI merits investment into these hybrid systems. Given the enthusiasm around LLMs today, injecting connections to symbolic reasoning could be a crucial next step in fulfilling their promise while mitigating risks.

Blending logical rule-based reasoning with modern neural networks could create more capable and reliable AI systems. This combination of human-like symbolic thinking and data-driven pattern recognition represents an exciting path forward. The result may be AI that better aligns with human intelligence in terms of adaptability, efficiency, and trustworthiness. Integrating the strengths of both of these approaches could lead to more advanced and human-compatible AI.

Sources:

Constraining large language models with logic

Neurologic decoding improves logical consistency of text generated by large language models

Teaching transformers to systematically reason with differentiable logic

ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language

KG-BERT: BERT for knowledge graph completion

Neuro-symbolic concept learner: Discovering objects and their properties

Testing AI's Ability to Understand Language in Context

Thu, 10 Aug 2023 08:08:00 +1000

Artificial intelligence has made great strides in natural language processing in recent years. Systems can now translate text, answer questions, and generate coherent paragraphs on demand. However, most AI still struggles with true language understanding that requires integrating information across long texts.

Back in 2016, to address this limitation, researchers developed a benchmark called the LAMBADA dataset to rigorously test how well AI models can leverage broader discourse context when predicting an upcoming word.

LAMBADA contains over 10,000 passages extracted from fiction books, with the last word blanked out in each passage. When humans are given the full passage as context, they can easily guess the missing word. However, if humans only see the final sentence containing the blank, it becomes virtually impossible to predict the missing word.

For example, the sentence "Do you honestly think that I would want you to have a ?" on its own has many plausible words that could fill in the blank. But when given the full passage about a couple discussing pregnancy concerns beforehand, it becomes clear from the context that the missing word is "miscarriage."

The researchers tested a wide range of AI systems on LAMBADA, including statistical n-gram models as well as advanced neural network architectures like LSTMs. Back then, all the models performed extremely poorly, with 0% to 7% accuracy in predicting the missing word. The models often relied on simple techniques like picking a random proper noun from the passage. Even methods designed to track broader context failed to match human performance. LAMBADA continues to be used today too test new projects such as Novel AI, and this time Models are performing with over 70% accuracy.

Truly intelligent systems will need to integrate information across long passages and reason about that context to understand language the way people do.

While AI chatbots and virtual assistants are improving customer service and other applications, they cannot yet achieve the sophistication of human context processing. Benchmarks like LAMBADA push innovators to develop the next generation of AI that skillfully uses context instead of relying on surface-level statistical patterns.

Just as IQ tests expanded to gauge different types of intelligence beyond a single number, benchmarks like LAMBADA are important for building well-rounded language AI systems. Advancing contextual language understanding will enable more fluent, trustworthy interfaces between people and machines. Whether in customer service or product development, AI that masters using context could unlock new levels of human-computer interaction.

Sources:

The LAMBADA dataset: Word prediction requiring a broad discourse context

Filling in the Blanks: AI Learns to Suggest Missing Pieces of Stories

Thu, 10 Aug 2023 08:07:37 +1000

Stories unfold step-by-step, but writers sometimes get stuck on how to connect one part to the next. AI research from 2019 explored how to automatically generate reasonable suggestions for missing sections of text. This "story infilling" aimed to assist creative writing by proposing ideas that align with the existing story while still surprising the author.

The researchers found that standard AI language models at the time struggled to balance coherence with novelty when filling in gaps. The generated text ended up too boring or too random. To address this limitation, they designed a two-step hierarchical system:

First, the AI randomly selected a few rare, interesting words that could plausibly fit into the storyline based on the context. For a medieval fantasy passage, it might suggest words like "dragon," "princess," or "castle." The system focused on rare words since they provide more information to guide the rest of the text.
Second, the system generated full sentences conditioned on those interesting words, searching likely combinations that form coherent text. Leveraging the rare words prevented repetitive suggestions, while allowing the model to focus on fluency and coherence.

The researchers tested story infilling on passages from children's tales with missing sections of 15-30 words. Human evaluators preferred the hierarchical model's suggestions over non-hierarchical methods, which sacrificed diversity or quality.

While an early attempt, the study shows promise for AI-assisted writing tools. The approach mirrors a writer's workflow - first deciding on key ideas, then piecing together suitable wording. Similar techniques may enable more human-like narrative understanding and creativity.

The field has greatly advanced since 2019 with models like Claude and GPT-4. Yet even powerful AI still struggles with high-level plot and character consistency. Explicitly decomposing generation into steps of planning and drafting, as humans do, is one way to address these challenges. While AI cannot replace human creativity, structured models could soon provide useful brainstorming and revision tools for real authors.

Sources:

Unsupervised Hierarchical Story Infilling

Behind the Scenes of Storytelling: Using AI to Plan and Structure Narratives

Thu, 10 Aug 2023 08:07:19 +1000

Storytelling seems almost magical. Writers conjure up entire worlds from their imaginations. But even master storytellers rely on plans and outlines to craft complex, coherent narratives spanning hundreds of words. In 2019, researchers explored how artificial intelligence could similarly use hierarchical models to improve computer-generated stories.

Up until then, most AI systems created stories simply word-by-word from left to right. While fine for short texts, this method struggled with long-term plot and character consistency. The researchers proposed "coarse-to-fine" techniques to first generate story outlines, then build surface-level details conditioned on the outline.

Their approach involved three steps: modeling the sequence of actions using verbs and arguments, generating story sentences with placeholder entities like "ent0", and finally rewriting the placeholders with specific references. This mirrored how human writers first sketch a plot's arc, then go back to flesh out settings and characters.

By creating more structured drafts, the AI models improved event diversity and entity consistency compared to previous approaches. The placeholder entities also made it easier to track characters, replacing different mentions with the same token. The researchers found that human judges strongly preferred stories created with hierarchical planning versus direct generation.

While an early attempt, this work showed the promise of mimicking writing strategies like outlining and revising. The field has advanced rapidly since 2019 as models like GPT-4 or Claude 2 now generate amazingly fluent text. But behind the scenes, AI still struggles with plot and people - areas where hierarchical techniques could help. The research highlights the value of breaking narration into more human-like steps. A technique currently being explored by several AI-assisted writing startups such as Novel AI and Sudowrite.

Just as outlines aid human storytellers, explicit planning and revisions may allow AI to better learn from experience. More structured generation spaces let models focus on specific challenges like action sequences before full text. While AI has seen stunning progress, people remain the masters of storycraft. Studying the narrative strategies of writers may guide systems to become more helpful to writers.

Source:

Strategies for Structuring Story Generation

Reading Between the Lines: Using Math to Uncover Hidden Patterns in Books

Thu, 10 Aug 2023 08:06:53 +1000

Books may seem like straightforward stories, but researchers are finding mathematical patterns hidden in the text. By tracking how words are used over the course of a book in minute detail, they can reveal new insights into plot, emotion, and structure that are not visible to the naked eye.

The researchers started by scoring a large number of words based on their emotional meaning. For example, positive words like "love" scored higher while negative words like "war" scored lower. They used a framework called "ousiometrics" which boils down emotions to two key dimensions: power and danger. Power relates to agency, confidence, and positivity. Danger relates to emotional uncertainty, negativity, and aggression.

They then took thousands of books and broke them down into short segments of 50 words each. For each segment, they calculated the average power and danger scores based on the words present. This turned each book into a rolling wave of numbers, with peaks representing more emotional sections and valleys as more neutral parts.

Short books generally showed a steady wave pattern while long books had more fluctuations in emotion over the course of the text. Surprisingly, when they zoomed in on long books they found the fluctuating highs and lows had a consistent length of a few thousand words. This matches the typical length of chapters in published fiction.

To study the patterns further, the researchers used a technique called empirical mode decomposition that breaks down fluctuations in data into distinct components, much like musical notes make up chords. The text segments were also compared to "shuffled" versions of the books with random word order. The real books differed from the random versions after a certain decomposition level, indicating that the fluctuations were not random but reflected an underlying structure.

These findings suggest longer books have a wave-like shape that is closer to collections of short stories or chapters. The emotional ups and downs of the text cycle on a scale of thousands of words, perhaps reflecting how long the human brain can comfortably process a complex narrative before needing a reset. Shorter books lacked these larger fluctuations.

While we intuitively understand how passages evoke certain moods, the researchers were able to quantify the pacing of emotional highs and lows mathematically. Their work helps confirm the existence of nested patterns in writing - punctuation gives phrases, paragraphs offer local structure, chapters provide mid-level segments, and over the full book arcs emerge.

So the next time you open a book, think about the hidden rhythms inside that subtly influence your experience. The feelings evoked in the story may follow mathematical waves as you steadily progress from cover to cover. This emerging field opens up new ways of appreciating the art and science of expert storytelling.

Sources:

A decomposition of book structure through ousiometric fluctuations in cumulative word-time

Storywrangler: Tracking Culture and Events through Twitter's Lens

Thu, 10 Aug 2023 08:05:59 +1000

Social media platforms like Twitter offered an unprecedented window into the real-time thoughts, conversations, and interests of millions of people. Researchers developed a tool called Storywrangler that leveraged Twitter data to create an "instrument for understanding our world through the lens of social media."

Storywrangler analyzed over 100 billion tweets dating back to 2008 to detect trends in word usage over time. It broke down tweets into "n-grams" - sequences of one, two, or three words - and tracked how the usage frequencies of these n-grams changed on a daily basis across different languages.

This massive database allowed researchers to see how real-world events, from natural disasters to political movements, were reflected in the narratives that unfolded on Twitter. For example, Storywrangler revealed surging interest in climate-related terms during major storms and wildfires. And it captured the rapid rise and fall of hashtags associated with social justice protests. Beyond reacting to news, Twitter also mirrored more subtle cultural shifts, like the waxing and waning popularity of celebrities or diets.

Storywrangler went beyond tracking raw frequencies - it also quantified how widely information spread on social media through shares and reposts. This helped distinguish niche conversations from truly viral ideas. The researchers used "contagiograms" to visualize both the popularity and amplification of n-grams over time.

There were certainly limitations to the Twitter lens. The platform's user base skewed young, urban, and affluent compared to the general population. Bots and organized campaigns could artificially inflate interest in certain topics. And the meanings of words themselves evolved across the years.

But used carefully, Storywrangler offered an unparalleled window into the collective consciousness - recording not just major news events but also the mundane daily conversations of millions worldwide. It aimed to complement more traditional data sources like books and news archives. The researchers hoped Storywrangler would enable more data-driven computational social science to understand our fast-changing, digitally-connected world.

Source:

Storywrangler: A massive exploratorium for sociolinguistic, cultural, socioeconomic, and political timelines using Twitter

The Six Basic Emotional Story Arcs, According to Science

Thu, 10 Aug 2023 08:05:37 +1000

Stories have gripped humans for ages by evoking powerful emotions. Even years ago, researchers wondered - are there fundamental patterns underlying how tales tug our heartstrings? A 2016 study analyzed over a thousand stories to uncover the basic emotional arcs that form the building blocks of narratives.

Using digital methods, researchers at the University of Vermont quantified the moment-to-moment emotional trajectories of stories from the Project Gutenberg collection. They tracked sentiment throughout each book using a rolling average approach. This generated an "emotional arc" capturing how positivity rises and falls across the narrative.

Through data science techniques like matrix decomposition, clustering, and neural network mapping, six core emotional arcs emerged:

Rags to riches (rise)
Tragedy or riches to rags (fall)
Man in a hole (fall then rise)
Icarus (rise then fall)
Cinderella (rise, fall, then rise)
Oedipus (fall, rise, then fall again)

You can likely picture classic stories matching these shapes. Cinderella follows a rags-to-riches-to-rags-to-riches pattern. Oedipus the King exhibits a tragic fall, brief rise, then another fall. Each arc formally captures intuitions storytellers have traded on for ages.

Notably, the study found arcs with a fall-rise shape (man in a hole, Cinderella) were collectively most common, at around 30% of tales. The next most prevalent were tragedies at around 32%, then oedipal fall-rise-falls at 31%. Purely rising arcs were rarest at just 5% of stories.

This indicates emotional rollercoasters captivate us more than simple rises. Tragedy has proven perennially popular, despite leaving readers sad. Stories that plunge protagonists into despair before rising contain greater drama.

The analysis also uncovered that more complex arcs with multiple peaks and valleys enjoyed greater success by one metric: website downloads. Stories whose shape matched the Icarus, Oedipus, and double "man in a hole" arcs saw far more downloads than simpler arcs.

This hints the emotional journey impacts how narratives spread. The anguish of tragedy may make such tales powerfully shareable. Multi-phasic arcs may also hook readers through twists and turns.

Of course, downloads provide only a rough measure of success. Other factors like marketing and fame contribute. Still, the findings suggest crafting an evocative emotional trajectory helps stories resonate.

The study demonstrates how data science can unearth hidden patterns in the arts. Formalizing intuitions about entertainment with empirical evidence remains novel, intriguing territory.

Quantitative analysis today is even more powerful thanks to advances in machine learning and cultural analytics. New techniques promise insights unavailable to past generations of literary scholars.

Combining these digital humanities approaches with insight and traditional criticism will likely bear the richest fruits. As machines grow skilled at classifying sentiment and archetypes, what new discoveries await about the stories humans compulsively tell? What makes a narrative emotionally compelling transcends any one discipline.

Sources:

The emotional arcs of stories are dominated by six basic shapes

Teaching AI to Tell Better Tales by Integrating External Knowledge

Thu, 10 Aug 2023 08:05:08 +1000

Storytelling comes naturally to humans. But for machines, spinning an engaging narrative remains an elusive goal. While AI can generate remarkably fluent text, its tales often lack coherence or get repetitive. New research explores how integrating structured knowledge into AI systems can enhance storytelling abilities.

When reading a story, we draw on general knowledge about how events logically unfold and characters plausibly act. We track complex plot threads and fill gaps using common sense. Machines lack this innate understanding we take for granted. Their stories can become nonsensical or contradictory.

To tackle this, researchers are providing AI systems explicit knowledge in structured formats. This external knowledge acts like a guide, keeping machine-generated plots on track. It also helps avoid stale repetitions by expanding the ideas available to pull from.

Several common limitations plague today's AI storytellers:

Lack of long-term coherence. Without a sense of overall narrative arc, they ramble aimlessly.
Insufficient grounding in real-world facts. Stories come off vague rather than richly descriptive.
Repetition. They loop the same words and phrases like a broken record.
Hallucination. They fabricate events that don't logically follow.

Integrating knowledge resources like ConceptNet, which contains common sense facts about the world, alleviates these issues. The knowledge functions like an annotated outline, steering the plot. It also provides a memory bank of concepts to reference, varying the content.

But effectively harnessing external knowledge remains challenging. Two main strategies have emerged:

Injecting knowledge directly into the AI system's training process, like teaching a human author.
Using knowledge as an external guiding reference during story generation.

Each approach has trade-offs. Weighting structured resources too strongly can pollute the system's original language skills. But using knowledge merely as a loose guide can fail to correct nonsensical narration.

Striking the right balance is an active research problem. Scientists are also expanding the knowledge available to AI storytellers with new databases. Most systems today use generic common sense facts. But resources detailing specific people, places, and events could enable more detailed, vivid storytelling.

Automating evaluation also poses difficulties. No single "correct" story exists for a given prompt. Automatic metrics struggle to account for creativity and interest - aspects requiring human judgment. More robust evaluation is critical to gauge progress.

Despite hurdles, knowledge-infused narration clearly improves coherence, factual grounding, and variation. AI authors with a knowledge boost spin far more convincing yarns. The research provides a roadmap for machines to better mimic core elements of human storytelling.

Rather than viewing imagination and structure as at odds, they are complementary. Master storytellers combine free-flowing creativity with purposeful intent. By fusing extensive knowledge with unrestrained generation, machines inch closer toward unlocking that balancing act.

Sources:

Open-world Story Generation with Structured Knowledge Enhancement: A Comprehensive Survey