multimodal-llm

Original：🇺🇸 English
Translated

Vision, audio, and multimodal LLM integration patterns. Use when processing images, transcribing audio, generating speech, or building multimodal AI pipelines.

22installs
Sourceyonatangross/orchestkit
Added on

NPX Install

npx skill4agent add yonatangross/orchestkit multimodal-llm

Tags

Translated version includes tags in frontmatter
multimodal-llmvision-integrationaudio-processingai-pipelinesmodel-selection

SKILL.md Content

View Translation Comparison →

Multimodal LLM Patterns

Integrate vision and audio capabilities from leading multimodal models. Covers image analysis, document understanding, real-time voice agents, speech-to-text, and text-to-speech.

Quick Reference

CategoryRulesImpactWhen to Use
Vision: Image Analysis1HIGHImage captioning, VQA, multi-image comparison, object detection
Vision: Document Understanding1HIGHOCR, chart/diagram analysis, PDF processing, table extraction
Vision: Model Selection1MEDIUMChoosing provider, cost optimization, image size limits
Audio: Speech-to-Text1HIGHTranscription, speaker diarization, long-form audio
Audio: Text-to-Speech1MEDIUMVoice synthesis, expressive TTS, multi-speaker dialogue
Audio: Model Selection1MEDIUMReal-time voice agents, provider comparison, pricing
Total: 6 rules across 2 categories (Vision, Audio)

Vision: Image Analysis

Send images to multimodal LLMs for captioning, visual QA, and object detection. Always set 
max_tokens
and resize images before encoding.
RuleFileKey Pattern
Image Analysis
rules/vision-image-analysis.md
Base64 encoding, multi-image, bounding boxes

Vision: Document Understanding

Extract structured data from documents, charts, and PDFs using vision models.
RuleFileKey Pattern
Document Vision
rules/vision-document.md
PDF page ranges, detail levels, OCR strategies

Vision: Model Selection

Choose the right vision provider based on accuracy, cost, and context window needs.
RuleFileKey Pattern
Vision Models
rules/vision-models.md
Provider comparison, token costs, image limits

Audio: Speech-to-Text

Convert audio to text with speaker diarization, timestamps, and sentiment analysis.
RuleFileKey Pattern
Speech-to-Text
rules/audio-speech-to-text.md
Gemini long-form, GPT-4o-Transcribe, AssemblyAI features

Audio: Text-to-Speech

Generate natural speech from text with voice selection and expressive cues.
RuleFileKey Pattern
Text-to-Speech
rules/audio-text-to-speech.md
Gemini TTS, voice config, auditory cues

Audio: Model Selection

Select the right audio/voice provider for real-time, transcription, or TTS use cases.
RuleFileKey Pattern
Audio Models
rules/audio-models.md
Real-time voice comparison, STT benchmarks, pricing

Key Decisions

DecisionRecommendation
High accuracy visionClaude Opus 4.6 or GPT-5
Long documentsGemini 2.5 Pro (1M context)
Cost-efficient visionGemini 2.5 Flash ($0.15/M tokens)
Video analysisGemini 2.5/3 Pro (native video)
Voice assistantGrok Voice Agent (fastest, <1s)
Emotional voice AIGemini Live API
Long audio transcriptionGemini 2.5 Pro (9.5hr)
Speaker diarizationAssemblyAI or Gemini
Self-hosted STTWhisper Large V3

Example

python
import anthropic, base64

client = anthropic.Anthropic()
with open("image.png", "rb") as f:
    b64 = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": [
        {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
        {"type": "text", "text": "Describe this image"}
    ]}]
)

Common Mistakes

  1. Not setting 
    max_tokens
    on vision requests (responses truncated)
  2. Sending oversized images without resizing (>2048px)
  3. Using 
    high
    detail level for simple yes/no classification
  4. Using STT+LLM+TTS pipeline instead of native speech-to-speech
  5. Not leveraging barge-in support for natural voice conversations
  6. Using deprecated models (GPT-4V, Whisper-1)
  7. Ignoring rate limits on vision and audio endpoints

Related Skills

  • rag-retrieval
    - Multimodal RAG with image + text retrieval
  • llm-integration
    - General LLM function calling patterns
  • streaming-api-patterns
    - WebSocket patterns for real-time audio