tribe-v2-bci-applied

Applied BCI research and neuro-informed content optimization using Meta's TRIBE v2 brain encoder. Predicts neural responses to media, UI, and content without brain scanners — enabling stimulus optimiz

Files5

View on GitHub

SKILL.md

GitHub

TRIBE v2 Applied BCI Skill

Agentic skill for applied BCI research and neuro-informed content optimization — from predicting fMRI cortical responses to media without brain scanners, through stimulus optimization and attention ranking, to generating cortical priors for non-invasive BCI decoding research.

License constraint: TRIBE v2 is CC BY-NC 4.0. This skill is for non-commercial research only. Commercial neuromarketing, advertising optimization, or audience profiling for profit requires a separate license from Meta. Read references/ethics-privacy.md before any applied use.

Quick Start

1. Install TRIBE v2

# Python 3.11+ required
git clone https://github.com/facebookresearch/tribev2
cd tribev2
pip install -e .

2. Load Model and Run First Prediction

from tribev2 import TribeModel

# Load model — downloads weights on first run (~several GB)
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")

# Build events dataframe from your stimulus
df = model.get_events_dataframe(video_path="path/to/video.mp4")

# Predict cortical responses
preds, segments = model.predict(events=df)

# preds.shape = (n_timesteps, n_vertices)
# n_vertices ~20,000 on fsaverage5 surface mesh
print(f"Predicted response shape: {preds.shape}")
print(f"Mean activation across all cortex: {preds.mean():.4f}")

3. Supported Input Modalities

# Video (extracts visual + auditory + motion features)
df = model.get_events_dataframe(video_path="clip.mp4")

# Text only (activates language network)
df = model.get_events_dataframe(text_path="script.txt")

# Audio only (activates auditory + language regions)
df = model.get_events_dataframe(audio_path="voiceover.wav")

4. Extract Region Activation

import numpy as np

# Visual cortex — approximate fsaverage5 vertex range
visual_vertices = list(range(1000, 7000))
visual_activation = preds[:, visual_vertices].mean()
print(f"Visual cortex mean activation: {visual_activation:.4f}")

Workflow A: Content Engagement Ranking

Trigger: You have N media files (videos, audio clips, or text variants) and want to rank them by predicted neural engagement — which one will drive more visual attention, emotional resonance, or language processing.

When to use: A/B testing ad creative before production, ranking tutorial formats, comparing voiceover styles, testing UI motion animations.

Tool: scripts/content_tester.py

Input: folder of files (all same modality), target regions

1. Load TRIBE v2 once
   → Single model load amortized across all files

2. For each file:
   → model.get_events_dataframe(...)
   → model.predict(events=df)
   → Compute mean activation per requested region
   → Record per-file scores

3. Compute overall_engagement_score = mean(all region scores)

4. Rank files descending by overall_engagement_score

Output: CSV with per-region scores + overall rank, top-3 printed to console

Run it:

python scripts/content_tester.py \
  --input-dir ./ad_variants/ \
  --modality video \
  --regions visual,auditory,language \
  --output engagement_rankings.csv

Interpreting results:

Score range Interpretation
> 0.6 High predicted engagement — stimulus strongly activates target networks
0.3 – 0.6 Moderate engagement — typical for well-produced content
< 0.3 Low engagement — consider redesigning stimulus elements

Design considerations:

  • Scores are relative within your batch, not absolute fMRI values
  • Compare within modality for best results (don't rank videos vs audio directly)
  • High visual + low language = visually engaging but not verbally memorable
  • High language + low visual = good for information retention tasks

Workflow B: Stimulus Optimization

Trigger: You have a base stimulus and want to find variants that maximize predicted activation in a specific cortical region — e.g., maximize visual cortex response for a display ad, or maximize language network response for a tutorial narration.

When to use: Iterative content refinement, creative optimization loops, accessibility improvements (maximize auditory processing for hearing-impaired content), BCI stimulus design.

Tool: scripts/optimize_stimulus.py

Input: base stimulus file, target region, modality, number of variants

1. Predict baseline activation on original file
   → Establish baseline score for target region

2. Generate N perturbation variants
   Video: brightness (0.7x–1.4x), contrast (0.8x–1.3x), saturation, playback speed
   Audio: speed (0.85x–1.15x), pitch shift, volume normalization variants
   Text: model prints paraphrase suggestions for human review (cannot auto-perturb text)

3. Predict activation on each variant
   → model.predict(events=variant_df)
   → Score target_region_vertices.mean()

4. Rank variants by target region mean activation
   → Output CSV: variant_file, target_region_mean, rank
   → Print top variant's delta vs baseline

Output: Ranked CSV, best variant path, improvement percentage

Run it:

python scripts/optimize_stimulus.py \
  --input ./base_ad.mp4 \
  --target-region visual \
  --modality video \
  --n-variants 10 \
  --output-dir ./optimized/

Target region options:

Region flag Cortical target Applied goal
visual V1–V4, MT (vertices 1000–7000) Visual attention, saliency
auditory A1 + belt + STS (vertices 8000–13000) Voice quality, audio engagement
language Broca's + Wernicke's (vertices 15000–18500, LH) Comprehension, verbally memorable
motion MT/V5 (vertices 5500–7000) Motion perception, dynamic content
default_mode mPFC/PCC/AG (vertices 19000–20000) Mind-wandering, narrative immersion

Greedy optimization loop (advanced — run in a shell loop):

BEST="base_ad.mp4"
for ITER in 1 2 3 4 5; do
  python scripts/optimize_stimulus.py \
    --input "$BEST" \
    --target-region visual \
    --modality video \
    --n-variants 8 \
    --output-dir ./iter_${ITER}/
  BEST=$(python -c "
import csv
with open('iter_${ITER}/rankings.csv') as f:
    rows = list(csv.DictReader(f))
print(rows[0]['variant_file'])
")
  echo "Iter $ITER best: $BEST"
done

Workflow C: Attention Proxy Analysis

Trigger: You want to quantify predicted attentional engagement — not just one region, but a composite proxy combining visual and social/multisensory processing.

Rationale: True attentional engagement in fMRI correlates with simultaneous activation of:

  • Early visual cortex (V1–V4): processing visual input at all
  • MT/V5: tracking motion — moving stimuli attract attention
  • STS (superior temporal sulcus): social signals, face motion, voice prosody

High combined score = stimulus is predicted to capture and hold attention.

import numpy as np
from tribev2 import TribeModel

model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")

def attention_proxy(model, file_path, modality="video"):
    """Compute attention proxy score from TRIBE v2 predictions."""
    if modality == "video":
        df = model.get_events_dataframe(video_path=file_path)
    elif modality == "audio":
        df = model.get_events_dataframe(audio_path=file_path)
    else:
        df = model.get_events_dataframe(text_path=file_path)

    preds, segments = model.predict(events=df)

    # Component regions
    early_visual = preds[:, 1000:5500].mean(axis=1)   # V1–V4
    motion_region = preds[:, 5500:7000].mean(axis=1)   # MT/V5
    sts_region = preds[:, 11500:13000].mean(axis=1)    # STS

    # Attention proxy: weighted combination
    attention_ts = 0.4 * early_visual + 0.3 * motion_region + 0.3 * sts_region

    return {
        "attention_proxy_mean": float(attention_ts.mean()),
        "attention_proxy_peak": float(attention_ts.max()),
        "attention_proxy_std": float(attention_ts.std()),
        "peak_timestep": int(attention_ts.argmax()),
        "early_visual_mean": float(early_visual.mean()),
        "motion_mean": float(motion_region.mean()),
        "sts_mean": float(sts_region.mean()),
    }

# Usage
result = attention_proxy(model, "campaign_video.mp4", modality="video")
print(f"Attention proxy: {result['attention_proxy_mean']:.4f}")
print(f"Peak engagement at timestep: {result['peak_timestep']}")

Interpreting the attention proxy timeseries:

import matplotlib.pyplot as plt

# Visualize attention dynamics over time
preds, segments = model.predict(events=df)
early_v = preds[:, 1000:5500].mean(axis=1)
motion = preds[:, 5500:7000].mean(axis=1)
sts = preds[:, 11500:13000].mean(axis=1)
proxy = 0.4 * early_v + 0.3 * motion + 0.3 * sts

plt.figure(figsize=(12, 4))
plt.plot(proxy, label="Attention proxy", linewidth=2)
plt.plot(early_v, alpha=0.5, label="Visual (V1-V4)")
plt.plot(motion, alpha=0.5, label="Motion (MT)")
plt.plot(sts, alpha=0.5, label="STS (social)")
plt.xlabel("Timestep")
plt.ylabel("Predicted activation")
plt.title("Attention proxy across stimulus duration")
plt.legend()
plt.tight_layout()
plt.savefig("attention_dynamics.png", dpi=150)

Use this to:

  • Find the moment in a video where attention is predicted to drop (cut or restructure that segment)
  • Compare opening hooks: which 10-second intro scores highest on attention proxy?
  • Identify which audio/visual elements drive the peaks

Workflow D: BCI Prior Generation

Trigger: You are working on a non-invasive BCI (EEG, fMEG, or fNIRS) decoding project and need population-average cortical activation priors — e.g., to localize imagined speech, visual imagery, or auditory perception without running a full fMRI study.

Rationale: TRIBE v2 was trained on large-scale fMRI data. Its predictions represent population-average expected activations for a stimulus class. These priors can seed:

  • Spatial filters for EEG source localization (beamforming, eLORETA)
  • Region-of-interest masks for constrained decoding
  • Expected activation patterns for cross-modal transfer learning
import numpy as np
from tribev2 import TribeModel

model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")

def generate_cortical_prior(model, stimulus_class_files: list, modality: str = "audio") -> np.ndarray:
    """
    Generate a population-average cortical activation prior for a stimulus class.
    
    Args:
        stimulus_class_files: list of file paths for stimuli in this class
        modality: 'video', 'audio', or 'text'
    
    Returns:
        prior: (n_vertices,) mean activation map across stimuli and time
    """
    all_preds = []

    for fpath in stimulus_class_files:
        if modality == "audio":
            df = model.get_events_dataframe(audio_path=fpath)
        elif modality == "video":
            df = model.get_events_dataframe(video_path=fpath)
        else:
            df = model.get_events_dataframe(text_path=fpath)

        preds, _ = model.predict(events=df)
        # Average over time for this stimulus
        all_preds.append(preds.mean(axis=0))

    # Average over stimulus class
    prior = np.stack(all_preds).mean(axis=0)
    return prior


def save_prior_as_nifti_compatible(prior: np.ndarray, output_path: str):
    """
    Save prior as numpy array for downstream BCI toolchain use.
    Compatible with MNE-Python, nibabel, and FSL workflows.
    """
    np.save(output_path, prior)
    print(f"Saved prior shape {prior.shape} to {output_path}")
    print("Load with: import numpy as np; prior = np.load('prior.npy')")


# Example: generate speech vs. non-speech priors for EEG decoding
speech_files = ["speech_1.wav", "speech_2.wav", "speech_3.wav"]
non_speech_files = ["music_1.wav", "noise_1.wav", "tone_1.wav"]

speech_prior = generate_cortical_prior(model, speech_files, modality="audio")
non_speech_prior = generate_cortical_prior(model, non_speech_files, modality="audio")

# Differential contrast prior (speech - non-speech)
contrast_prior = speech_prior - non_speech_prior
save_prior_as_nifti_compatible(contrast_prior, "speech_contrast_prior.npy")

# Identify top vertices (most discriminative regions)
top_vertices = np.argsort(np.abs(contrast_prior))[-500:]
print(f"Top 500 discriminative vertices: {top_vertices}")
print(f"Language network (Broca's ~15000-17000): {sum(15000 <= v <= 17000 for v in top_vertices)} vertices in range")

Integrating with MNE-Python (EEG source modeling):

import mne
import numpy as np

# Load your TRIBE v2 prior
prior = np.load("speech_contrast_prior.npy")  # shape: (n_vertices_fsaverage5,)

# Use as initial weights for minimum norm estimate (MNE)
# Prior needs to be projected to source space matching your EEG setup
# See MNE docs: mne.minimum_norm.make_inverse_operator with depth weighting

# The prior defines which regions you expect to be active —
# feeds into beamformer spatial filter initialization or
# constrains the solution space for sparse inverse methods

Important disclosures for BCI use: TRIBE v2 priors are population-average predictions. Individual subject brains differ in activation patterns. When using these priors in real BCI pipelines, disclose this assumption to end users. Do not present population-average predictions as personalized neural decoding. See references/ethics-privacy.md for full BCI-specific risk disclosure.

Ethical Guardrails

This skill operates under CC BY-NC 4.0 restrictions and ethical norms for brain simulation research.

This skill MUST NOT be used for:

  • Commercial advertising optimization or neuromarketing for profit
  • Building psychological profiles for commercial audience targeting
  • Generating "neural dark patterns" — stimuli designed to bypass conscious decision-making
  • Any profiling of individuals without explicit informed consent
  • Any use involving minors without a guardian consent framework
  • Surveillance or monitoring applications

This skill MAY be used for:

  • Academic research and publication (non-commercial)
  • Accessibility improvement research
  • Clinical hypothesis generation (not diagnosis)
  • UX research with full participant disclosure
  • Non-invasive BCI research with appropriate consent frameworks

Before any applied use, read the full ethics and licensing reference: references/ethics-privacy.md

For commercial licensing, contact Meta Research: https://research.facebook.com

Tool Reference

TRIBE v2 API

Method Input Output Notes
TribeModel.from_pretrained(model_id) HuggingFace model ID TribeModel instance Downloads ~GB of weights on first call
model.get_events_dataframe(video_path=) Video file path pd.DataFrame Extracts visual, auditory, motion features
model.get_events_dataframe(audio_path=) Audio file path pd.DataFrame Extracts auditory + language features
model.get_events_dataframe(text_path=) Text file path pd.DataFrame Extracts language + semantic features
model.predict(events=df) Events DataFrame (preds, segments) preds.shape = (timesteps, vertices)

Key Output Properties

Property Value Description
preds.shape[0] Varies with stimulus duration Number of predicted timepoints
preds.shape[1] ~20,000 Vertices on fsaverage5 surface mesh
preds.mean() Float Overall cortical activation mean
preds[:, v_start:v_end].mean() Float Region mean activation

Supported Formats

Modality Formats Notes
Video .mp4, .avi, .mov Extracts visual + audio features jointly
Audio .wav, .mp3, .flac Pure auditory feature extraction
Text .txt Language model feature extraction

Cortical Region Reference

For full region atlas with vertex ranges, activation profiles, and applied BCI use cases, see references/cortical-region-atlas.md.

Quick vertex range cheatsheet:

Region Vertices (approx) Key activators
V1/V2 (primary visual) 1000–4000 Edges, contrast, spatial frequency
V4 (color/form) 4000–5500 Color, shape, object form
MT/V5 (motion) 5500–7000 Optical flow, motion direction
IPS/FEF (attention) 7000–8500 Top-down attention, gaze control
A1 (primary auditory) 8000–10000 Tone, pitch, onset
Belt regions (auditory) 10000–11500 Voice, timbre, melody
STS (social/voice) 11500–13000 Speaker identity, prosody, face motion
FFA (faces) 12000–14000 Face identity, expression
PPA (places/scenes) 14000–16000 Spatial layout, architecture
Broca's area (LH) 15000–17000 Syntax, speech production
Wernicke's area (LH) 17000–18500 Speech comprehension
VWFA (reading) 18500–19500 Visual words, orthography
vmPFC (reward/value) 18000–19000 Emotional valence, reward expectation
mPFC/PCC/AG (DMN) 19000–20000 Narrative, self-referential, mind-wandering

References

Related in design

add-uint-support

Add unsigned integer (uint) type support to PyTorch operators by updating AT_DISPATCH macros. Use when adding support fo...

add
uint
support
by pytorch
95,362

skill-writer

Guide users through creating Agent Skills for Claude Code. Use when the user wants to create, write, author, or design a...

skill
writer
by pytorch
95,362

frontend-design

Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to...

frontend
design
by anthropics
47,860