Plot identification

Introduction

A story in a manuscript comprises a set of overlapping plots. A plot is, fundamentally, a causal trajectory of events driven by characters that alters the state of the story’s world. This representation includes not only the physical condition of places and objects, but also the emotional and cognitive makeup of the characters themselves.

Identifying a plot is a task human readers perform instinctively, yet the outcome is deeply subjective. It depends heavily on the reader’s inherent biases, their background, and even their cognitive load at the time of reading—all of which dictate what information they find salient. Consequently, an event that one reader identifies as the crux of a major plotline might be dismissed by another as mere background, or attributed to an entirely different narrative arc.

This subjectivity complicates verifiable approaches to plot identification, historically pushing the field toward unsupervised methods like clustering. However, as explored below, these methods fall fundamentally short: they are structurally incapable of recognizing the causal nature of plots.

At Robo Data Lab, we investigate an alternative approach centered on world modeling: detecting salient state changes and inferring the causal relationships between them. A prerequisite for this is robust entity extraction—specifically, the ability to persistently track characters—a challenge we are addressing in a dedicated research pillar (see character identification).

What is Plot Identification - a worked example based on Jane Austen’s “Pride and Prejudice”

To illustrate how the extraction engine maps concurrent event trajectories within unstructured prose, we examine an excerpt from Chapter 26 of Jane Austen’s Pride and Prejudice. In this scene, a single continuous dialogue weaves two distinct, addressable narrative arcs together in real-time.

The Intersecting Plots

The engine must recognize that this conversation is not a sequence of alternating events, but a singular collision of two separate causal chains:

• [Plot 1 - (The Wickham Romance)]: The trajectory of Elizabeth’s romantic interest in George Wickham, which is complicated by his lack of fortune.
• [Plot 2 - (Family Duty & Guardianship)]: The trajectory of Mrs. Gardiner’s mentorship and protective oversight of Elizabeth, invoking the expectations of her family.

1. The Annotated Text

Instead of hallucinating alternating events at the paragraph level, a properly aligned engine parses the semantic crossover at the clause level. Mrs. Gardiner leverages the guardianship arc (Plot 2) to directly alter the parameters of the romance arc (Plot 1) within the exact same breath.

Plot 2: Mrs. Gardiner’s caution to Elizabeth was punctually and kindly given on the first favourable opportunity of speaking to her alone: after honestly telling her what she thought, she thus went on — ‘You are too sensible a girl, Lizzy, Plot 1: to fall in love merely because you are warned against it; Plot 2: and therefore I am not afraid of speaking openly. Seriously, I would have you be on your guard.

Plot 1: Do not involve yourself or endeavour to involve him in an affection which the want of fortune would make so very imprudent. I have nothing to say against him; he is a most interesting young man; and if he had the fortune he ought to have, I should think you could not do better. But as it is, you must not let your fancy run away with you.

Plot 2: You have sense, and we all expect you to use it. Your father would depend on your resolution and good conduct, I am sure. You must not disappoint your father.’

Plot 2: ‘My dear aunt, this is being serious indeed.’

Plot 2: ‘Yes, and I hope to engage you to be serious likewise.’

Plot 2: ‘Well, then, you need not be under any alarm. Plot 1: I will take care of myself, and of Mr. Wickham too. He shall not be in love with me, if I can prevent it.’

2. The Spatio-Temporal Graph

To process this collision for a search framework, the graph maps the extraction using a strict coordinate system:

• The Y-Axis (Frames): Divides the narrative into two horizontal swimlanes. Plot 2 (Family Duty) is the top frame, and Plot 1 (Romance) is the bottom frame.
• Node Sizing: Characters exclusive to one plot (Mrs. Gardiner, Father, Wickham) fit entirely inside their respective frames. Elizabeth, however, is a massive vertical node spanning both frames, visually proving she is the nexus entity binding the plots together.
• The X-Axis (Time): The nodes are arranged strictly from left to right in the exact order they are invoked in the text.

Related work

Let’s consider the skills one employs when attempting to dissect and identify a plot in a story:

• One needs to be able to identify the characters, events, and places.
• One needs to be able to tell the purpose for which they come together.
• One needs to understand how they relate to each other in space and time.
• One needs to manage the complexity of details, possibly strained throughout the entire manuscript, and isolate those entities out of them.

While these tasks resemble the objectives of attention mechanisms—including bi-directional and masked language modeling—no existing method produces quality high enough for production-grade software.

Existing methods often omit crucial plots, misclassify existing ones, break temporal and causal relationships within and between plots, or outright hallucinate narratives.

Furthermore, the complexity and vagueness of these tasks lie far beyond the realm of classical (non-neural) implementations, so we are omitting them altogether.

Narrative structure labelling

In Movie Plot Analysis via Turning Point Identification, Papalampidi, Keller and Lapata (2019) locate five turning points — Opportunity, Change of Plans, Point of No Return, Major Setback, Climax — in a document, trained on the TRIPOD corpus of synopses and screenplays. The output is five positions in the text. It holds no entities, no state, and no relations, so there is nothing in it to compare between two points in the story. Tellingly, trained annotators agree on only 35.48% of turning points in screenplays, which is a direct measurement of the reader-dependence described above.

Sentiment trajectories

Reagan et al. (2016) demonstrated in The emotional arcs of stories are dominated by six basic shapes that scoring a sliding window against a valence lexicon reduces resulting curves to recurring geometric shapes. Tools like syuzhet and Jon Chun’s SentimentArcs (2021) compute the same signal by other means. The output is one scalar per position. There are no entities to hold state and no events to attribute a movement to, so a change in the value cannot be traced back to a specific occurrence in the story.

Distributional and retrieval methods

To overcome the context limits of large texts, the field frequently relies on recursive chunking and summarization, as formalized in OpenAI’s Recursively Summarizing Books with Human Feedback (Wu et al., 2021) and evaluated heavily in the BookSum benchmark (Kryscinski et al., EMNLP 2021). In these retrieval and summarization pipelines, chunking the document and embedding the chunks produces a vector per span. However, a vector carries no identity, so it cannot assert that a character in Chapter 2 and a character in Chapter 21 are the same node. It also carries no addressability, so it cannot specify which attribute of which entity took a new value. Similarity between two vectors is also symmetric, and therefore carries no directional causality.

Traditional relation extraction pipelines

Before generative LLMs and TGNNs superseded them, the standard approach to building explicit relational graphs relied on supervised document-level relation extraction (DocRE). Zhou et al. (AAAI 2021) introduced ATLOP, which classified relations between entity pairs using a learnable threshold, reaching 63.4 F1 on the factual DocRED dataset.

We mention this legacy machinery to highlight a foundational failure mode that persists in structured extraction: these models assume entities are already perfectly located, typed, and normalized. When forced to rely on an upstream entity recognition pipeline rather than perfect gold-standard annotations—the reality of processing the ambiguous text of a novel—the exact same class of model suffers catastrophic error cascading, falling to 18.34 F1.

Entity state tracking

Work on state tracking originated in procedural texts to follow physical locations during step-by-step processes, pioneered by Dalvi et al. (2018) in Tracking State Changes in Procedural Text, which introduced the ProPara dataset. Datasets like OpenPI 2.0 (2024) expanded this to track physical entity changes across short texts.

However, translating this to narratives—where states are psychological, implicit, and stretched over hundreds of pages—has proven exceptionally difficult. Early work by Rashkin et al. (2018) attempted to track character motivations and emotional reactions, but only on simple, five-sentence commonsense stories. The gap for long-form fiction is so profound that the NarraBench survey (October 2025) analyzed 78 NLP benchmarks and found that narrative events and subjective character perspectives are nearly absent from current evaluations.

It was only recently that the field saw the introduction of STAGE (2026), a benchmark explicitly designed to test tracking of persistent characters and evolving states across full narratives. Yet, STAGE focuses on screenplays, which benefit from rigid structural formatting. To date, there is absolutely no established extraction framework or gold-standard dataset that successfully tracks persistent, evolving character states across the unstructured prose of full novels.

Entity graphs and knowledge graphs

State-of-the-art knowledge graph (KG) construction now leverages Large Language Models alongside Temporal Graph Neural Networks (TGNNs) to extract evolving relationships from unstructured text. Recent frameworks like LLM-S2KG (May 2026) and DynSciGraph (July 2026) dynamically model the architecture of information over time, while comprehensive surveys (such as the LLM-empowered knowledge graph construction survey, 2025) highlight the industry shift toward graphs optimized for Retrieval-Augmented Generation (GraphRAG).

While this produces nodes and edges — the right architectural form — three critical elements remain absent for narrative modeling:

• These frameworks are overwhelmingly optimized for factual domains (like finance or science) and do not natively represent subjective narrative events.
• Nodes typically serve as static semantic anchors rather than entities carrying addressable internal states (such as a character’s shifting emotional makeup).
• While TGNNs drastically improve the temporal understanding of a graph (successfully mapping when states mutate across discrete time steps), sequence does not equate to causation. They do not inherently capture the causal dependencies of those state mutations (the narrative why) at the granularity required to reconstruct a literary plot.

Event and causal graph extraction

To address the exact causality gap left by traditional knowledge graphs, a parallel track of NLP research focuses on extracting implicit causal relations by modeling temporal features. Where KGs model entities without causal events, frameworks in this domain do the inverse: they model causal events without persistent entities.

For example, Lin et al.’s framework, R² (March 2025), directly contrasts with KG architectures by extracting typed event records from a novel and linking them into causal plot lines for screenplay generation. Similarly, Wilmot’s Shadow-Loom (May 2026) builds a versioned causal graph over narratives utilizing Pearl’s ladder of causation, and a recent multi-agent system (March 2025) extracts episodic narrative arcs from television shows.

These are the closest approaches to our goal: events are nodes, and causation is an edge. However, because they act as the structural opposite of a knowledge graph, the nodes are only events. Characters, places, and objects are not treated as persistent nodes carrying state, so a change in the world appears in the graph only if a sentence states it directly. Furthermore, none of these three papers report strict extraction accuracy.

The limits of LLM inference

Despite operating in disjointed domains—from bibliometric forecasting to screenplay generation—the frameworks above share a unifying architectural consensus: Large Language Models are structurally incapable of maintaining causal or relational consistency on their own. Across the board, state-of-the-art graph construction now relegates LLMs exclusively to boundary extraction, relying on deterministic algorithms, counterfactual calculi, or Temporal Graph Neural Networks to act as the actual reasoning engines. Yet, while these hybrid systems successfully enforce logical consistency for overt events and factual relationships, they still lack the mechanisms to track the silent, psychological state mutations that drive a literary narrative.

Evaluation metrics in narrative AI

Currently, the evaluation of Large Language Models in the creative domain is overwhelmingly focused on text generation rather than extraction. The literature rigorously measures how well models write: Confederacy of Models (2023) utilized human evaluators to demonstrate that models like GPT-4 can surpass human baselines in end-to-end creative generation. Hemingway-bench (2026) relies on expert human writers who spend hours evaluating frontier models on generative tasks. Other recent frameworks, such as Creativity Index (2024), NoveltyBench (2025), and EQ-Bench Creative Writing (2026), similarly attempt to quantify the novelty or quality of generated stories. Even when evaluating strict formatting and mechanical constraints, as seen in WritingBench (2025), the focus remains on LLM-as-a-judge scoring of generated text.

While these metrics meticulously track generative capabilities, they offer zero utility for plot recovery. Conversely, on the structural extraction side, the field relies on traditional metrics like F1 for factual entity relations, or Structural Hamming Distance (SHD) for causal graphs. SHD assumes a rigid, mathematical ground truth that subjective fiction defies. Ultimately, the field currently lacks any benchmark or metric that evaluates the accurate extraction of causal chains and state changes from unstructured prose.

Formulation

We adopt the transport formalism developed in our alignment research and instantiate it for narrative. The notation is restated below in brief; it is derived in full under the reprojection hypothesis and the measurement operator.

Let $A \in \mathcal{S}$ denote the story world as held by the author, where $\mathcal{S}$ is the author’s latent state space; $B \in \Sigma^*$ the manuscript, a discrete sequence of symbols and the only artifact that exists; and $C \in \mathcal{L}$ the state a reader reconstructs, where $\mathcal{L}$ is that reader’s own latent space, carrying its own basis. Writing and reading are generative processes conditioned on the author’s priors $G$ and the reader’s priors $H$, the latter drawn from a population of readers $p(\mathcal{H})$:

$$B = t(A \mid G), \qquad\qquad C = s(B \mid H)$$

Let $\Pi$ denote a specific subclass of the salient-structure operator, strictly instantiated for the extraction of story plots. Applied to a story state it returns the set of plots that state contains:

$$\Pi(X) = \{\pi_1, \dots, \pi_k\}$$

Each $\pi_i$ is a plot in the sense given in the introduction: a causal trajectory of events, driven by characters, that alters the state of the story’s world — the physical condition of places and objects, and the emotional and cognitive makeup of the characters themselves. The members of $\Pi(X)$ are not disjoint. They share characters, and at points of crossover they share events. Neither $k$ nor the extent of that overlap is given in advance; both are part of what extraction must recover.

$\Pi(A)$, $\Pi(B)$ and $\Pi(C)$ stand for the plots the author held, the plots the manuscript carries, and the plots a given reader recovers, respectively.

Definition 1 Plot identification is the estimation of $\Pi(B)$ by a reconstruction process $s$ operating under machine priors $H_M$; writing $\Pi_M \equiv \Pi(s(B \mid H_M))$ for its output.

Properties of this formulation:

Approximating $\Pi(A)$ and $\Pi(C)$ through Preference Alignment. In the wild, author intent and individual reader interpretations are hidden variables. Our baseline target remains the invariant core across a population of reader priors:

$$\Pi^* = \text{Inv} \big\{ \, \Pi\big(s(B \mid H_i)\big) \, \big\}_{H_i \sim p(\mathcal{H})}$$

giving the base extraction objective:

$$\mathcal{L}_{core}(H_M) = \mathbb{E}_B \big[ \, d(\Pi_M, \Pi^*) + \beta \lvert \Pi_M \rvert \, \big]$$

where $d$ is semantic distortion defined in our alignment research.

However, during training, the preferences of specific readers ($\Pi(C)$) become observable via pairwise preference datasets. While the true author intent $\Pi(A)$ remains fundamentally unobservable, we can approximate a “silver standard” of it using author-generated metadata (e.g., outlines, synopses) as proxy preference data. Let $\mathcal{D}_{pref}$ consist of tuples $(B, \Pi_w, \Pi_l)$, where $\Pi_w$ is the preferred plot extraction (better reflecting the target prior) and $\Pi_l$ is the rejected extraction.

Assuming a Bradley-Terry model of preference, we can steer the machine prior $H_M$ using Direct Preference Optimization (DPO). The model’s implicit reward for an extraction is defined by the log-probability of generating that extraction relative to a reference model $H_{ref}$:

$$r_{H_M}(\Pi, B) = \gamma \log \frac{P(\Pi \mid B, H_M)}{P(\Pi \mid B, H_{ref})}$$

This allows us to formulate a preference loss that penalizes the model when it assigns higher likelihood to the dispreferred plot graph:

$$\mathcal{L}_{pref}(H_M) = - \mathbb{E}_{(B, \Pi_w, \Pi_l) \sim \mathcal{D}_{pref}} \left[ \log \sigma \Big( r_{H_M}(\Pi_w, B) - r_{H_M}(\Pi_l, B) \Big) \right]$$

The Joint Objective. The final plot identification objective is a joint optimization. It anchors the structural extraction to the invariant core to prevent hallucination, while shifting the salience boundary to align with the target human prior:

$$\min_{H_M} \Big( \mathcal{L}_{core}(H_M) + \lambda \mathcal{L}_{pref}(H_M) \Big)$$

Salience. The rate term $\beta \lvert \Pi_M \rvert$ supplies the operative definition. An element of $\Pi_M$ is salient if and only if its removal raises $d$ by more than $\beta$. Salience is measured by ablation of an element of $\Pi_M$ against a fixed query set, rather than asserted at extraction time.

Proposal for a viable implementation

To transition from mathematical formalism to a production-grade extraction engine, we propose a neuro-symbolic architecture where a Transformer acts as a perception layer that writes state-mutation commands to an external temporal graph. To teach the model both mechanical accuracy and human-like salience, we employ a two-stage training pipeline comprising Supervised Fine-Tuning (SFT) to anchor the model to the text’s reality, followed by Direct Preference Optimization (DPO) to teach it the cognitive compression of a human reader.

Why transformers?

To process plot extraction, the architecture cannot rely on traditional sequential or continuous generative paradigms. Recurrent Neural Networks (RNNs) suffer from catastrophic forgetting over long sequences, inevitably losing a character’s state between Chapter 1 and Chapter 20. Diffusion models, while state-of-the-art for continuous data, cannot operate on the discrete, topological nature of a causal graph.

The core reading engine must be a Transformer, leveraging its self-attention mechanism to resolve complex coreferences and localized semantics. However, because standard LLMs struggle to maintain long-term relational consistency implicitly, the Transformer does not act as the memory. It acts strictly as a perception layer, emitting discrete graph-update operations to an external Graph State Engine, which maintains the persistent nodes and causal edges over the narrative’s entire timeline.

Training process and loss function

The training loop satisfies our joint objective through two distinct phases:

Phase 1: Supervised Fine-Tuning (Anchoring to $\Pi(B)$) First, we perform causal language modeling on the extraction task. The model learns to read a chunk of text and emit explicit graph operations (e.g., UPDATE_STATE). The loss function here minimizes the semantic distortion $d$, teaching the model to track characters and extract explicit state changes without hallucinating relations.

Phase 2: Preference Optimization (Learning the drop to $\Pi(C)$) Next, we teach the model salience. We use Direct Preference Optimization (DPO) to model the cognitive compression of a human reader. The model’s implicit reward for an extraction is defined by the log-probability of generating that extraction relative to a reference model.

The final extraction objective is a joint optimization that anchors the structural extraction to the invariant core, while shifting the salience boundary to align with the target human prior:

$$\min_{H_M} \Big( \mathcal{L}_{core}(H_M) + \lambda \mathcal{L}_{pref}(H_M) \Big)$$

Dataset

The training data cannot merely be text mapped to summaries; it must be structured as serialized spatio-temporal graph commands.

For Phase 1 (SFT), the dataset pairs unstructured prose with invariant core graphs ($\Pi^*$) serialized as update operations. For Phase 2 (DPO), the dataset consists of pairs of operational sequences for the same chunk of text:

• Winning Sequence ($\Pi_w$): A sparse set of commands that only updates the graph for narrative-critical state changes.
• Losing Sequence ($\Pi_l$): An over-extracted set of commands that accurately reflects the text but includes micro-mutations (e.g., a character picking up a teacup) that distract from the causal plot chain.

How it comes together at inference

In production, the system streams the manuscript $B$ chunk-by-chunk into the trained Transformer. Instead of generating conversational text or summaries, the model continuously emits structured UPDATE_STATE commands. The external Graph State Engine applies these commands in real-time. The final output is not a document, but a queryable database: a strict spatio-temporal graph where every node is a persistent entity carrying its current state, and every edge is a verifiable causal link grounded in the text.

How it connects to the formulation

This architecture is a direct software implementation of our transport formalism. The SFT phase minimizes $\mathcal{L}_{core}$, successfully mapping the manuscript $B$ into the invariant core $\Pi^*$. The DPO phase operationalizes the rate term $\beta \lvert \Pi_M \rvert$. By penalizing the model for generating the over-extracted losing sequences ($\Pi_l$), the preference loss explicitly trains the model to apply that rate term itself—dropping mathematically valid but narratively useless events to successfully reconstruct the intended plot $\Pi(C)$.

How it differs from general-purpose LLMs

Modern instruct-tuned LLMs (like GPT-4 or Claude) are optimized for conversational alignment: they are trained to be helpful, harmless assistants that synthesize broad knowledge from their internal weights. When asked to identify a plot, they generate a highly smoothed, subjective text summary based on average patterns in their pretraining data.

Our model is trained strictly as a structural extractor and state mutator. It is completely disentangled from conversational helpfulness. Its alignment target is not conversational safety, but narrative salience and causal compression. By restricting its output space entirely to graph operations, and relying on an external deterministic engine to hold the state, we eliminate generative hallucination. It does not synthesize; it measures the text and updates a world model, precisely mirroring the operation of reading as a lossy codec.