Automated Multi-Lingual Shorts Video Pipeline Architecture with FFmpeg & Gemini

English Version

Creating high-performing vertical short-form video content (YouTube Shorts, TikTok, Instagram Reels) requires seamless visual pacing, synchronized audio backtracks, and global audience reach. At Machina Muse, our automated video generation pipeline processes dozens of AI assets using FFmpeg, Gemini BGM synthesis, and multi-lingual subtitle burn-in.

1. Dedicated Off-Server Rendering Architecture

Heavy video encoding and frame interpolation using FFmpeg can consume significant CPU/RAM resources, causing server stalls on lightweight cloud instances (such as GCP e2-micro). To maintain sub-millisecond web API responses, our architecture offloads all FFmpeg video processing to dedicated background workers, which push finished video bundles to our cloud storage and web backend.

2. Audio & Visual Assembly Pipeline

• Image Batch Processing: Dozens of high-resolution 9:16 AI images are dynamically scaled, pan-zoomed (Ken Burns effect), and concatenated using FFmpeg filtergraphs.
• Gemini BGM Synthesis: Audio tracks matching the visual mood (e.g. cyber-tech synthwave, fantasy orchestral) are generated using Gemini audio models and seamlessly looped.
• Multi-Lingual Subtitle Burn-In (KO / EN / JA): Subtitles are dynamically synthesized from raw image prompts and visual descriptors, translated into Korean, English, and Japanese, and burned into the lower third of the video frame for maximum global engagement.