FFmpeg + OpenCV Integration Guide

references/opencv-pipelines-and-libraries.md

When to Use

• Decode with FFmpeg and process frames with OpenCV
• Encode OpenCV-generated/processed frames with FFmpeg
• Compare

  cv2.VideoCapture

  , subprocess pipes, PyAV, ffmpegcv, VidGear, and Decord
• Fix RGB/BGR color bugs or

  (height, width)

  dimension-order bugs
• Preserve audio while replacing processed video frames
• Build GPU-assisted video I/O around CPU OpenCV processing

Library Selection

cv2.VideoCapture

Critical Gotchas

1. OpenCV is BGR. FFmpeg/PyAV/PIL/Decord often produce RGB. Convert explicitly.
2. NumPy dimensions are

   (height, width, channels)

   . Pixel access is

   img[y, x]

   , not

   img[x, y]

   .
3. Video filters can drop audio. Preserve or remux original audio intentionally.
4. Release resources. Always close

   VideoCapture

   , pipes, writers, and PyAV containers.
5. Rawvideo pipes require exact frame size.

   width * height * channels

   must match

   -pix_fmt

   .

Minimal Pipe Patterns

cmd = [
    "ffmpeg", "-i", input_path,
    "-f", "rawvideo", "-pix_fmt", "bgr24", "-"
]

cmd = [
    "ffmpeg", "-y",
    "-f", "rawvideo", "-vcodec", "rawvideo",
    "-s", f"{width}x{height}", "-pix_fmt", "bgr24",
    "-r", str(fps), "-i", "-",
    "-c:v", "libx264", "-preset", "fast", "-crf", "23",
    "-pix_fmt", "yuv420p", output_path
]

Core Workflow

1. Probe input dimensions, fps, duration, and audio streams.
2. Pick the I/O library based on the selection table.
3. Lock pixel format at boundaries (

   bgr24

   for OpenCV pipes;

   yuv420p

   for final H.264 output).
4. Process frames in generators/batches; avoid loading full videos unless they are small.
5. Preserve or re-encode audio explicitly.
6. Verify output duration, fps, resolution, codec, and A/V sync.

Reference Map

• references/opencv-pipelines-and-libraries.md

  - Full preserved reference: color/dimension gotchas, cleanup patterns, FFmpeg-to-OpenCV and OpenCV-to-FFmpeg pipes, bidirectional pipeline, ffmpegcv, VidGear, Decord, PyAV, Modal.com examples, GPU pipeline, cheat sheets, sources.

Related Skills

• ffmpeg-python-integration-reference

  - Type-safe parameters, colors, time units

• ffmpeg-pyav-integration

  - PyAV API details

• ffmpeg-hardware-acceleration

  - GPU decode/encode and filter pipelines