RUNE-2000: Media FFmpeg Backend Proposed

Updated2026-08-23

Summary

Rune media I/O is defined by backend-neutral track contracts in Rune.Media. Rune.Media.FFmpeg currently remains a transitional audio/mux helper package beside that API, while Rune.Media.WMF is the concrete IMediaEncoder/IMediaDecoder backend that exercises the contract on Windows.

Design

Rune.Media defines:

IVideoTrack, IAudioTrack, VideoSample, and AudioSample as the public media transport contracts.
IMediaEncoder and IMediaDecoder as the encode/decode entry points.
MediaEncodeOptions, VideoStreamInfo, and AudioStreamInfo as the focused configuration and metadata types.
IAudioBuffer, effect buffers, and buffer graph composition (MixedAudioBuffer) for sample-domain audio preparation before export.
IMediaDecoder.Seek(TimeSpan) as the seek boundary for passive decode tracks; tracks themselves do not expose seek.

Rune.Media.FFmpeg currently provides:

FFmpegLibraryOptions and FFmpegLibrary.Configure(...) for native FFmpeg lookup.
FFmpegAudioEncoder as a legacy helper that opens an output stream with Initialize(), accepts canonical interleaved float32 PCM through Encode(...), and finalizes with Complete().
FFmpegAudioDecoder as a legacy helper that seeks and fills caller buffers with canonical interleaved float32 PCM.
FFmpegResampledAudioBuffer as a bridge from generic IAudioBuffer graphs into FFmpeg/libswresample format conversion.
FFmpegMuxer as a legacy remux shim for already encoded track files; it does not re-encode samples.

These FFmpeg types do not currently implement IMediaEncoder or IMediaDecoder.

Rune.Media.WMF currently provides the concrete track-based backend:

WmfMediaEncoder writes MP4 output only, using H.264 video and AAC audio.
The encoder pulls timestamped samples from IVideoTrack and IAudioTrack, tries a D3D11 hardware path first, and falls back to CPU staging-texture readback when needed.
WmfMediaDecoder exposes the first supported audio/video streams through passive tracks, returns float32 PCM audio, and returns BGRA video frames as Rune.Image uploads or DXVA-backed VideoSample values when that path is enabled.

Audio Mixing Boundary

Per-source shaping lives in the buffer/effect layer, and the mixer stays focused on summing prepared float PCM sources. Export code should wrap the prepared graph in an IAudioTrack before handing it to an IMediaEncoder.

Video Sample Boundary

Video transport is likewise track-based: callers feed IMediaEncoder with IVideoTrack, and VideoSample keeps timestamp, duration, and frame ownership together. StoryRenderTrack is the current Story-side adapter for this boundary.

Native Binaries

FFmpeg.AutoGen supplies bindings, not FFmpeg native DLLs. Applications may rely on the process search path or configure a native DLL directory with FFmpegLibrary.Configure(new FFmpegLibraryOptions(rootPath)). Tests that require native FFmpeg should remain capability-gated.

RUNE-2000: Media FFmpeg Backend ​Proposed

Summary ​

Design ​

Audio Mixing Boundary ​

Video Sample Boundary ​

Native Binaries ​