Video Transcriber

This skill enables AI agents to transcribe audio from video files using Whisper (local processing), OpenAI Whisper API, Google Speech-to-Text, or Gemini API (cloud processing with advanced features).

When to Use

• User wants to transcribe a video or audio file
• User needs subtitles/captions for a video
• User wants to analyze video content through transcription
• User needs to identify viral-worthy segments
• User wants speaker diarization or emotion detection

Model Selection

Whisper (Local)

Pros:

• Free to use
• 100% privacy (no cloud upload)
• Good for sensitive content
• Lower cost for high volume

Cons:

• Requires local processing power
• No built-in speaker diarization
• No emotion detection
• Limited to 99 languages

Models:

• tiny - Fastest, lower accuracy (~32MB)
• base - Fast, good accuracy (~74MB)
• small - Balanced speed/accuracy (~244MB)
• medium - Good accuracy, slower (~769MB)
• large-v3 - Highest accuracy, slowest (~1550MB)

Local-first testing: Use tiny when you want the fastest local run for validation.

Gemini API (Cloud)

Pros:

• High accuracy with gemini-flash-lite-latest
• Built-in speaker diarization
• Emotion detection from speech
• Context understanding
• Can identify viral segments
• 125+ language support
• Faster processing (cloud-based)

Cons:

• Requires API key
• Cloud upload (privacy consideration)
• Cost per usage
• Internet required

OpenAI Whisper API (Cloud)

Pros:

• High accuracy with word-level timestamps
• No local GPU/CPU needed
• Consistent results

Cons:

• Requires API key
• Cloud upload (privacy consideration)
• Cost per usage
• Internet required

Google Speech-to-Text (Cloud)

Pros:

• High accuracy with word-level timestamps
• Speaker diarization support
• Scales well for long audio

Cons:

• Requires Google Cloud credentials
• Cloud upload (privacy consideration)
• Cost per usage
• Internet required

Available Scripts

scripts/transcribe.py

Transcribe audio from video file.

Usage:

python skills/video-transcriber/scripts/transcribe.py <video_path> [options]

Options:

• --model, -m: Model to use (whisper, gemini, openai, google) - default: auto
• --whisper-model: Whisper model size (tiny, base, small, medium, large-v3) - default: medium
• --openai-model: OpenAI Whisper model (default: whisper-1)
• --google-model: Google Speech model (default: latest_long)
• --use-faster: Use faster-whisper for speed - default: True
• --output, -o: Output file path (default: <video_path>.srt)
• --format: Output format (srt, vtt, json) - default: srt
• --language: Language code (e.g., en, id) - default: auto
• --speaker-diarization: Enable speaker labels (Gemini only)
• --emotion-detection: Enable emotion detection (Gemini only)
• --device: Device for Whisper (auto, cpu, cuda) - default: auto

Examples:

Transcribe with Whisper (default):

python skills/video-transcriber/scripts/transcribe.py video.mp4

Transcribe with Gemini API:

python skills/video-transcriber/scripts/transcribe.py video.mp4 --model gemini

Transcribe with OpenAI Whisper API:

python skills/video-transcriber/scripts/transcribe.py video.mp4 --model openai --format json

Note: When using --model openai, the system will try Google Speech-to-Text as a fallback if OpenAI fails and Google credentials are available.

Transcribe with Google Speech-to-Text:

python skills/video-transcriber/scripts/transcribe.py video.mp4 --model google --format json

Transcribe with speaker diarization and emotion detection (Gemini):

python skills/video-transcriber/scripts/transcribe.py video.mp4 --model gemini --speaker-diarization --emotion-detection

Transcribe with large Whisper model:

python skills/video-transcriber/scripts/transcribe.py video.mp4 --whisper-model large-v3

Output to JSON:

python skills/video-transcriber/scripts/transcribe.py video.mp4 --format json

scripts/analyze.py

Analyze audio content using Gemini API for viral segments, summary, or emotions.

Usage:

python skills/video-transcriber/scripts/analyze.py <video_path> [options]

Options:

• --analysis-type: Type of analysis (viral, summary, emotions, questions) - default: viral
• --num-segments: Number of segments to identify (for viral analysis) - default: 5
• --model: Model to use (default: gemini)

Examples:

Detect viral segments:

python skills/video-transcriber/scripts/analyze.py video.mp4 --analysis-type viral

Get summary:

python skills/video-transcriber/scripts/analyze.py video.mp4 --analysis-type summary

Analyze emotions:

python skills/video-transcriber/scripts/analyze.py video.mp4 --analysis-type emotions

Output Format

SRT Format

1
00:00:00,000 --> 00:00:05,000
This is the first subtitle.

2
00:00:05,500 --> 00:00:10,000
This is the second subtitle.

JSON Format

[
  {
    "index": 1,
    "start": 0.0,
    "end": 5.0,
    "text": "This is the first subtitle.",
    "speaker": "Speaker A",
    "emotion": "neutral"
  }
]

Auto Selection Logic

When --model auto, the system selects based on:

1. Privacy priority: Always use Whisper
2. Quality needed: Use gemini for highest quality
3. Content length: Use faster-whisper for long content (> 1 hour)
4. Feature requirements: Use gemini if speaker diarization or emotion detection needed
5. Default: Use gemini-flash-lite-latest

Note: openai and google models must be selected explicitly. Auto does not switch to them.

Environment Variables

# For Gemini API
export GEMINI_API_KEY="your-api-key"

# For OpenAI Whisper API
export OPENAI_API_KEY="your-api-key"

# For Google Speech-to-Text
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

# Optional: For Vertex AI
export GOOGLE_PROJECT_ID="your-project-id"
export GOOGLE_LOCATION="us-central1"

Integration with Other Skills

After transcription, you can use these skills:

• highlight-scanner: Analyze transcript for viral moments
• subtitle-overlay: Add captions to video
• autocut-shorts: Full workflow for creating short clips

Common Workflow

1. User provides video file or URL
2. Download if needed (youtube-downloader)
3. Transcribe using this skill
4. Analyze transcript for highlights (highlight-scanner)
5. Create short clips (autocut-shorts)

Tips

• Use --use-faster with Whisper for faster processing
• Use Gemini when you need speaker diarization
• Use --format json for programmatic processing
• For long videos, consider splitting into segments
• Use --analysis-type viral to identify best segments for short-form content

References

• Whisper documentation: https://github.com/openai/whisper
• Gemini API: https://ai.google.dev/gemini-api/docs/audio
• Language codes: ISO 639-1 codes (en, id, es, etc.)