Media, TTS, and export

Mental model

Media in Story is still Story time. Video sprites, speech, and export are not separate pipelines glued on afterward; they are prepared during bake, advanced through the shared playback model, and consumed by preview, still capture, cue capture, live presentation, or MP4 export. The same baked scene is the center of that system.

Responsibilities and boundaries

• playback advances a video sprite’s source time deterministically inside the Story timeline.
• speak resolves a compile-time voice facet, prepares audio before playback, and publishes waitable speech/audio markers.
• StoryPlayback owns seek plus render over baked snapshots and updates live source-video textures immediately before frame render.
• StoryExporter is the Story-layer bridge to media encoders; it does not bake scripts or own container-specific muxing policy.

Stable current behavior

• Preview, live, frame capture, cue capture, and export all reuse the same baked StoryPlayback seek/render core.
• StoryPlaybackMode.Exact is used for offline stepping such as export, frame, and cue capture, while RealtimePreview bounds decode work for preview and live.
• Speech labels become named marker facets such as $intro.drive, while bare speak and audio waits observe visible media sources lexically.
• StoryExportRequest requires the same engine that baked the scene so baked snapshot resource slots remain valid.
• When StoryExportRequest.AudioGraph is present, export adapts it through AudioBufferTrack. When it is null, the public exporter deliberately passes no audio track and produces video-only output.

Current limitations

• Realtime preview can intentionally show a best-effort, slightly stale video frame under load while Story time and audio time remain canonical.
• Platform voice facets such as male and female are strict host intents; missing matches diagnose rather than silently fall back.
• Story export is an orchestration layer over encoders and audio graphs, not a public container-assembly DSL.
• A caller that expects Story narration or media audio must build and pass the scene audio graph; a null graph explicitly requests video-only export.
• The current public speech surface is speak; broader clip-style authoring names remain outside the shipped contract.

How the pieces connect

1. Compilation resolves speech presets and video sprite authoring shapes.
2. Bake prepares speech/audio work, video schedules, and action-boundary snapshots.
3. Playback seeks those snapshots, updates source-video textures, and renders the requested Story time.
4. Preview/live use realtime decode policy, while frame/cues/export use exact offline stepping.
5. Export wraps playback as video and optional audio tracks for the selected encoder.

Go deeper

• Learn: Media and TTS, Preview, live, and export
• Examples: Video demo, TTS demo
• Reference: Story CLI, Story syntax
• Specs: RUNE-1011, RUNE-1012, RUNE-2002