ffmpeg-usage

---
name: ffmpeg-usage
description: "Make polished videos with ffmpeg: a production pipeline plus vetted recipes for conversion, scaling, compositing, GIFs, captions, audio/voiceover, motion, and platform-ready export — for demos, ads, tutorials, launches, social clips, explainers, and pitches."
---

# ffmpeg Usage

## Overview

Two things in one skill: a **command library** of vetted ffmpeg recipes for everyday media jobs, and a **production pipeline** for making videos that actually look intentional. Treat ffmpeg as the final compositor and encoder — the renderer — and bring in helper tools for the parts ffmpeg is clumsy at (asset design, animation, captions, 3D). Use it whenever a request touches media: converting, resizing, building GIFs, pulling audio, editing, or producing a finished video for marketing, demos, tutorials, launches, social, explainers, app-store previews, or pitches.

**Version:** 2.0.0
**Needs:** ffmpeg ≥ 4.0; `ffprobe` recommended. Other tools (below) are optional and used only when they earn their place.

## When to apply

Trigger on requests to:

- Convert, rescale, or re-fit aspect ratio
- Build a GIF, extract or convert audio
- Edit clips — trim, merge, speed, rotate
- Work with subtitles or captions — burn-in, soft, extract, generate
- Compress, web-optimise, or format for a platform (YouTube, Instagram, TikTok, X)
- Grab thumbnails or frames; batch-process a folder
- **Plan and produce a finished video** — story beats, scene composition, animated text, framing, voiceover/music, and a repeatable render pipeline

## Prerequisites

```bash
brew install ffmpeg              # macOS
sudo apt-get install ffmpeg      # Debian/Ubuntu
choco install ffmpeg             # Windows
ffmpeg -version                  # confirm
```

---

# Command library

## 1. Format conversion

**MP4 → WebM:**
```bash
ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 30 -b:v 0 -c:a libopus output.webm
```

**MOV → MP4:**
```bash
ffmpeg -i input.mov -c:v libx264 -c:a aac -strict experimental output.mp4
```

**Anything → MP4 (broad compatibility):**
```bash
ffmpeg -i input.* -c:v libx264 -preset medium -crf 23 -c:a aac -b:a 128k output.mp4
```

## 2. Resolution

Rescale while preserving aspect ratio.

**To 720p / 1080p:**
```bash
ffmpeg -i input.mp4 -vf scale=-1:720 -c:a copy output_720p.mp4
ffmpeg -i input.mp4 -vf scale=-1:1080 -c:a copy output_1080p.mp4
```

**Fixed width, auto height:**
```bash
ffmpeg -i input.mp4 -vf scale=1280:-1 -c:a copy output.mp4
```

**Fit with letterbox padding:**
```bash
ffmpeg -i input.mp4 -vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" output.mp4
```

## 3. GIFs

Palette generation is what keeps GIFs sharp and small.

**Quick (10 fps):**
```bash
ffmpeg -i input.mp4 -vf "fps=10,scale=480:-1:flags=lanczos" output.gif
```

**Two-pass with palette:**
```bash
ffmpeg -i input.mp4 -vf "fps=10,scale=480:-1:flags=lanczos,palettegen" palette.png
ffmpeg -i input.mp4 -i palette.png -filter_complex "fps=10,scale=480:-1:flags=lanczos[x];[x][1:v]paletteuse" output.gif
```

**From a time range (single pass):**
```bash
ffmpeg -ss 00:00:10 -t 5 -i input.mp4 -vf "fps=10,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" output.gif
```

## 4. Audio

**Extract to MP3 / WAV:**
```bash
ffmpeg -i input.mp4 -vn -acodec libmp3lame -q:a 2 output.mp3
ffmpeg -i input.mp4 -vn -acodec pcm_s16le -ar 44100 -ac 2 output.wav
```

**Convert format:**
```bash
ffmpeg -i input.wav -c:a aac -b:a 192k output.m4a
```

**Replace audio with a track / mix two together:**
```bash
ffmpeg -i video.mp4 -i music.mp3 -c:v copy -c:a aac -map 0:v:0 -map 1:a:0 -shortest output.mp4
ffmpeg -i video.mp4 -i music.mp3 -filter_complex "[0:a][1:a]amix=inputs=2:duration=first" -c:v copy output.mp4
```

## 5. Editing

**Trim (stream-copy, no re-encode):**
```bash
ffmpeg -i input.mp4 -ss 00:00:10 -to 00:00:30 -c copy output.mp4   # 10s–30s
ffmpeg -i input.mp4 -ss 00:00:05 -t 10 -c copy output.mp4          # 10s from 5s
```

**Concatenate** — pick by container:

```bash
# Protocol — TS/MPEG/MP3/AAC, no list file needed:
ffmpeg -i "concat:file1.mp3|file2.mp3|file3.mp3" -c copy output.mp3

# Demuxer — MP4/MOV/MKV, via process substitution:
ffmpeg -f concat -safe 0 -i <(printf "file '%s'\n" video1.mp4 video2.mp4 video3.mp4) -c copy output.mp4
# Fallback if the shell lacks process substitution:
printf "file '%s'\n" video1.mp4 video2.mp4 video3.mp4 > list.txt
ffmpeg -f concat -safe 0 -i list.txt -c copy output.mp4 && rm list.txt

# Filter — mismatched codecs/resolutions, re-encodes:
ffmpeg -i video1.mp4 -i video2.mp4 -i video3.mp4 \
  -filter_complex "[0:v][0:a][1:v][1:a][2:v][2:a]concat=n=3:v=1:a=1[v][a]" \
  -map "[v]" -map "[a]" output.mp4
```

Decision: MP3/AAC/TS/MPEG → protocol; MP4/MOV/MKV → demuxer; differing codecs or sizes → filter.

**Speed / rotate:**
```bash
ffmpeg -i input.mp4 -filter:v "setpts=0.5*PTS" -an output.mp4   # 2x
ffmpeg -i input.mp4 -filter:v "setpts=2.0*PTS" output.mp4       # 0.5x
ffmpeg -i input.mp4 -vf "transpose=1" output.mp4               # 90° CW
ffmpeg -i input.mp4 -vf "transpose=2,transpose=2" output.mp4   # 180°
```

## 6. Subtitles

```bash
ffmpeg -i input.mp4 -vf subtitles=subtitles.srt output.mp4       # burn in
ffmpeg -i input.mp4 -i subtitles.srt -c copy -c:s mov_text output.mp4   # soft subs
ffmpeg -i input.mp4 -map 0:s:0 subtitles.srt                     # extract
```

## 7. Thumbnails

```bash
ffmpeg -i input.mp4 -ss 00:00:05 -vframes 1 thumbnail.jpg   # one frame
ffmpeg -i input.mp4 -vf fps=1/10 thumb%04d.jpg              # every 10s
ffmpeg -i input.mp4 -vframes 10 frame%04d.png              # first 10 frames
```

## 8. Compression & web

```bash
ffmpeg -i input.mp4 -c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k output.mp4    # balanced
ffmpeg -i input.mp4 -c:v libx264 -crf 28 -preset veryslow -c:a aac -b:a 96k output.mp4   # smaller
ffmpeg -i input.mp4 -c:v libx264 -preset medium -crf 23 -movflags +faststart -c:a aac -b:a 128k output.mp4   # web
```

## Platform presets

**YouTube:**
```bash
ffmpeg -i input.mp4 -c:v libx264 -preset slow -crf 18 -c:a aac -b:a 192k -pix_fmt yuv420p -movflags +faststart youtube.mp4
```

**Instagram Story (9:16, ≤15s):**
```bash
ffmpeg -i input.mp4 -vf "scale=1080:1920:force_original_aspect_ratio=decrease,pad=1080:1920:(ow-iw)/2:(oh-ih)/2" -c:v libx264 -preset medium -crf 23 -c:a aac -b:a 128k -t 15 instagram_story.mp4
```

**X / Twitter (16:9, ≤2:20):**
```bash
ffmpeg -i input.mp4 -vf scale=1280:720 -c:v libx264 -preset medium -crf 23 -c:a aac -b:a 128k -t 140 twitter.mp4
```

**TikTok (9:16, ≤60s):**
```bash
ffmpeg -i input.mp4 -vf "scale=1080:1920:force_original_aspect_ratio=decrease,pad=1080:1920:(ow-iw)/2:(oh-ih)/2" -c:v libx264 -preset medium -crf 23 -c:a aac -b:a 128k -t 60 tiktok.mp4
```

## Quick recipes

```bash
# Shrink a screen recording
ffmpeg -i screen_recording.mov -c:v libx264 -preset medium -crf 23 -vf "scale=1920:-1" -c:a aac -b:a 128k optimised.mp4

# Batch-convert a folder
for i in *.mov; do ffmpeg -i "$i" -c:v libx264 -crf 23 -c:a aac "${i%.mov}.mp4"; done

# Images → video
ffmpeg -framerate 30 -pattern_type glob -i '*.jpg' -c:v libx264 -pix_fmt yuv420p output.mp4   # sequence
ffmpeg -loop 1 -i image.jpg -c:v libx264 -t 5 -pix_fmt yuv420p output.mp4                      # 5s from one image
```

---

# Polished video production workflow

For good-looking videos, use ffmpeg as the final compositor/encoder, not as the only creative tool.

1. **Define the story beats.** One clear idea per beat: hook, problem, solution, proof, call to action. Write the through-line before touching a command.
2. **Gather source media** — screen recordings, product shots, images, AI/stock clips, or generated backgrounds. Capture clean footage: use a release build, hide debug overlays, cursors, and notifications, and record at the target resolution or higher.
3. **Normalise all media** to the target format — usually `1080x1920` vertical or `1920x1080` landscape, a consistent frame rate (30 fps is a safe default), and `yuv420p`.
4. **Create reusable assets** — transparent text overlays, frames, masks, backgrounds, captions, logos.
5. **Compose scene-by-scene** with ffmpeg, one short clip per beat.
6. **Add motion** — pan, zoom, scroll, parallax, easing, transitions.
7. **Add audio** — voiceover, music, sound effects, captions, and loudness normalisation.
8. **Render preview frames / contact sheets** before the final export.
9. **Verify with `ffprobe`.**

Prefer several short scenes over one long continuous recording — easier to time, replace, review, and reuse.

**Normalise clip timing** with `setpts=(target/actual)*PTS`, e.g. retime an 8s clip to 6s:
```bash
ffmpeg -y -i clip-raw.mp4 -vf "setpts=0.75*PTS,fps=30,format=yuv420p" -an clip-normalised.mp4
```

# Useful libraries and tools

ffmpeg is excellent for encoding, compositing, filters, audio mixing, scaling, and final export. Other tools are better at generating assets, animation, captions, and visual design. Reach for them only when ffmpeg gets awkward.

### Visual composition

- **ffmpeg** — final composition, encoding, A/V sync, transitions, overlays, scaling, cropping, filters.
- **ImageMagick** — quick image conversion, resizing, trimming, masks, contact sheets.
- **Pillow / PIL** — generate transparent PNG text overlays, caption cards, thumbnails, title frames, gradients, simple layouts.
- **OpenCV** — frame analysis, crop detection, motion tracking, masking, scene processing.
- **PySceneDetect** — auto-detect scene changes in long videos.
- **Blender** — 3D product/device renders, camera movement, lighting, shadows.
- **Three.js / React Three Fiber** — web-based 3D scenes and animated product mockups.
- **Remotion** — React-based programmatic video; reusable branded templates and dynamic videos.
- **Manim** — animated diagrams and technical/educational explainers.
- **Lottie / Bodymovin** — lightweight vector animations from After Effects or JSON.

**Selection rule:**

- ffmpeg for the final render and media processing.
- Pillow / HTML+CSS for designed text overlays.
- Remotion when the layout is mostly UI, components, and text.
- Blender or Three.js when the video needs real 3D depth.
- OpenCV when it needs detection, tracking, cropping, or frame analysis.

### Text, captions, and typography

For premium text, prefer generating transparent PNG overlays over relying only on ffmpeg `drawtext` (which is fragile around quoting and font discovery).

- **Pillow** — reliable text rendering, wrapping, shadows, gradients, tracking, caption cards.
- **HTML/CSS screenshots via Playwright** — best when text needs web-quality typography: flexbox/grid, custom fonts, gradients, responsive layout.
- **ASS subtitles** — styled and karaoke-style captions.
- **SRT/VTT** — simple subtitles with broad platform support.
- **Whisper / faster-whisper** — transcribe voiceover into captions.
- **aeneas / Gentle / WhisperX** — forced alignment for accurate word timing.

Guidelines: keep headlines short; keep captions clear of important content; respect mobile safe margins; avoid flashing text unless it's a deliberate, readable rhythm; animate text once, then hold steady; generate preview frames to check overlap and readability. Prefer a condensed bold display font for headlines.

### Audio, voiceover, and music

Good audio reads as more professional than extra visual effects.

Voiceover:

- **Kokoro TTS / Piper** — lightweight local open-source TTS; good for repeatable, private generation.
- **Coqui XTTS** — expressive TTS and voice cloning; check licensing and model availability before commercial use.
- **OpenAI TTS / ElevenLabs / PlayHT** — higher-quality hosted voices when commercial polish matters; need API keys and cost control.

Music and sound: use open-licensed music when publishing commercially and keep the licence/attribution file beside the asset; prefer instrumental tracks under voiceover; duck music under speech; add subtle effects for transitions, taps, reveals, and success states; normalise final loudness.

```bash
# Normalise loudness
-af "loudnorm=I=-16:TP=-1.5:LRA=11"

# Fade music in and out
-af "afade=t=in:ss=0:d=1,afade=t=out:st=27:d=1"

# Mix voice and music
-filter_complex "[voice][music]amix=inputs=2:duration=first"

# Duck (sidechain) music under voice
-filter_complex "[music][voice]sidechaincompress=threshold=0.04:ratio=8:attack=20:release=250[ducked];[voice][ducked]amix=inputs=2"

# Loop music to fit the video without retiming the picture
ffmpeg -y -i silent.mp4 -stream_loop -1 -i music.mp3 -map 0:v:0 -map 1:a:0 \
  -shortest -c:v copy -c:a aac -b:a 192k -movflags +faststart final.mp4
```

Voiceover guidance: one short line per scene, matched to what's on screen; use punctuation to control rhythm; nudge TTS speed up slightly if it sounds flat; leave pauses between beats; generate captions from the **final** voiceover, not the draft script.

### Motion and animation

Static compositions feel like slideshows. Add subtle motion to every scene: slow push-in, gentle pan, device parallax, background drift, screen scroll, text ease-in then hold, fast-but-readable transitions on the beat, light zoom for emphasis. Prefer smooth easing over linear movement; animate once at a beat's start, then hold.

```bash
# Ease-out slide-in for a text overlay PNG (smooth, no flicker)
ffmpeg -y -i base.mp4 -i title.png \
  -filter_complex "[0:v][1:v]overlay=x='(W-w)/2':y='if(lt(t,0.35),-h+(220+h)*sin(t/0.35*PI/2),220)':enable='between(t,0,3)'[v]" \
  -map "[v]" -map 0:a? -c:v libx264 -crf 18 -preset medium -pix_fmt yuv420p \
  -c:a aac -b:a 192k -movflags +faststart out.mp4

# Ken Burns push-in
zoompan=z='min(zoom+0.0015,1.08)':d=150:x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)'
```

# Framing products, devices, and mockups

When footage shows an app, website, or product, frame it so it reads as a real object rather than a flat rectangle.

- Compose on the target canvas (e.g. `1080x1920`) and keep text in top/bottom bands, off the footage.
- Fit footage into the screen area, preserving aspect ratio:
  ```bash
  ffmpeg -y -i clip.mp4 -vf "scale=564:1284:force_original_aspect_ratio=decrease,pad=564:1284:(ow-iw)/2:(oh-ih)/2:black,fps=30,format=rgba" screen.mp4
  ```
- Make the device smaller than the canvas, with rounded body, bezel, notch/island, side buttons, and shadow; keep the content in a transparent screen cutout.
- Give the device a **stage** — warm rim light, soft shadow, gentle glow, faint outline. Never sit a black device on a black background.
- For consumer work, a cinematic dark background with warm white/gold type usually beats neon accent bars. Pick **one** accent colour and keep it.
- Accent key words within a line (white text, final keyword in the accent) instead of one flat colour.
- Write beat-specific support copy that explains the moment — never one generic line under every scene.
- Keep texture subtle; a little grain adds depth, visible noise cheapens it.

A solid vertical layout to start from:
```text
canvas: 1080x1920
media stage: x=40, y=235, w=1000, h=1420
device body: w=600, h=1298
device stage: dark rounded rectangle behind the device, low-opacity warm outline
headline: y=72, condensed uppercase, wide tracking
bottom tagline: y=1708, condensed uppercase, final keyword in accent
support copy: y=1814, smaller light text
accent: one warm colour, used consistently
```

# Video QA checklist

Before final delivery:

- `ffprobe` to confirm dimensions, duration, codecs, pixel format, and audio:
  ```bash
  ffprobe -v error -select_streams v:0 \
    -show_entries stream=width,height,r_frame_rate,duration \
    -show_entries format=duration,size -of default=nw=1 final.mp4
  ```
- Export preview frames at key timestamps and build a contact sheet for review:
  ```bash
  ffmpeg -y -ss 00:00:03 -i final.mp4 -frames:v 1 -update 1 preview-03.png
  ```
- Check text readability on mobile and that captions don't overlap important content.
- Check audio starts and ends cleanly and that voiceover matches what's on screen.
- Check for debug overlays, reload flashes, cursor artifacts, or stale screens.
- Confirm the export uses `yuv420p` and `+faststart`.

Recommended final export:
```bash
ffmpeg -i input.mp4 \
  -c:v libx264 -crf 18 -preset medium -pix_fmt yuv420p \
  -c:a aac -b:a 192k \
  -movflags +faststart \
  output.mp4
```

---

## Best practices

- **Inspect first:** `ffprobe -v quiet -print_format json -show_format -show_streams input.mp4`
- **`-c copy` whenever you can** to skip re-encoding.
- **Test on a slice** with `-t 10` before committing to a long render.
- **CRF guide:** 18 ≈ visually lossless, 23 = good default, 28 = smaller and acceptable (range 0–51).
- **`-movflags +faststart`** on web video for progressive playback.

## Guidelines for the agent

On a media request: identify the task, pick the matching recipe, confirm prerequisites (ffmpeg present, input exists), explain the command, run it with error handling, verify the output, and suggest optimisations where they help. For a finished video, work the pipeline above — beats first, then source, compose, motion, audio, QA — and break complex jobs into explained steps. Reach for a helper tool only when ffmpeg alone is awkward, and say why.

For concatenation, prefer process substitution with `printf` over a temporary `list.txt`; fall back to the file only when the shell can't do it.

Always: confirm the input exists and is readable, check ffmpeg is installed, ensure the output path is writable, fail with a clear message, and show progress on long renders.

## References

- Docs: https://ffmpeg.org/documentation.html
- Wiki: https://trac.ffmpeg.org/wiki
- Codecs: https://ffmpeg.org/ffmpeg-codecs.html
- Filters: https://ffmpeg.org/ffmpeg-filters.html