Shottr macOS Screen Capture & Annotation AI Skill Guide (Claude)

Overview & Engine Architecture

Shottr is an ultra-fast, lightweight macOS screenshot and annotation utility engineered in native Swift. It harnesses Apple's modern ScreenCaptureKit framework for zero-latency frame capture, uses Metal hardware acceleration for instant raster rendering and vector annotation overlays, and embeds the Apple Vision Framework (VNRecognizeTextRequest) for on-device Live Text optical character recognition (OCR) and object erasure. Shottr includes an automated Scrolling Capture vertical image stitcher, precise on-screen pixel rulers, and irreversible lossy pixelation / blur filters. Claude operates as a Principal macOS Graphics Engineer and Image Processing Specialist, specializing in ScreenCaptureKit frame capture, Vision OCR extraction pipelines, TCC Screen Recording permissions, and automated screencapture scripting.

Shottr Core Architecture & Metal Pipeline Stack

┌─────────────────────────────────────────────────────────────┐
│                 Shottr Engine Architecture                  │
│                                                             │
│  Capture & Frame Ingestion Tier                             │
│  ├── Apple `ScreenCaptureKit` Hardware Frame Ingestion      │
│  ├── Window ID / Display Stream Selector (`CGWindowList...`)│
│  └── Scrolling Capture Image Stitcher (Feature Matching)    │
│                                                             │
│  Image Processing & Vision AI Core                          │
│  ├── Apple Vision Framework OCR (`VNRecognizeTextRequest`)  │
│  ├── Metal Compute Shader Pipeline (Pixelation & Fast Blur) │
│  └── Pixel Ruler & Delta-E Color Inspector                  │
│                                                             │
│  Annotation & Export Subsystem                              │
│  ├── Vector Annotation Engine (Arrows, Text, Number Pins)   │
│  ├── Direct Clipboard Injection (`NSPasteboard.writeObjects`│
│  └── Lossless OptiPNG / WebP / JPEG Image Encoder           │
└─────────────────────────────────────────────────────────────┘

---

Operational Capabilities & Agent Directives

1. ScreenCaptureKit & Frame Capture Scripting: Construct Swift and Python scripts utilizing native macOS screenshot APIs to capture specific display rects and windows without UI artifacts.
2. Vision Framework Live Text Extraction: Implement automated OCR routines converting screen bounding boxes into structured, searchable text.
3. Screen Recording TCC Permissions Diagnostics: Resolve blank/wallpaper-only capture anomalies by auditing macOS Screen Recording permissions in TCC.db.
4. Scrolling Screenshot Stitching Optimization: Advise users on optimal scrolling capture velocity to avoid duplicated sticky headers and visual seams.

---

Production Swift Automation: Headless Screen Capture & Vision OCR Pipeline

Save this file as screen_capture_ocr.swift and execute via swift screen_capture_ocr.swift:

// ==============================================================================
// Standalone Swift 5.x Script: Screen Capture & On-Device Vision OCR
// Captures main display bounds and extracts printed text using Apple Vision.
// ==============================================================================
import Cocoa
import Vision

// 1. Capture Main Screen Image using CoreGraphics
guard let mainDisplay = CGMainDisplayID() as CGDirectDisplayID? else {
    print("Error: Could not obtain main display ID.")
    exit(1)
}

guard let screenshot = CGDisplayCreateImage(mainDisplay) else {
    print("🚨 Error: Screen capture failed. Ensure Terminal has Screen Recording permissions in System Settings.")
    exit(1)
}

print("--- [CAPTURED SCREENSHOT: \(screenshot.width)x\(screenshot.height) px] ---")

// 2. Perform On-Device Live Text OCR via Vision Framework
let requestHandler = VNImageRequestHandler(cgImage: screenshot, options: [:])
let request = VNRecognizeTextRequest { (req, error) in
    guard let observations = req.results as? [VNRecognizedTextObservation] else {
        print("No text detected on screen.")
        return
    }

    print("\n--- [EXTRACTED ON-SCREEN TEXT VIA APPLE VISION OCR] ---")
    for observation in observations {
        guard let topCandidate = observation.topCandidates(1).first else { continue }
        if topCandidate.confidence > 0.5 {
            print("• \(topCandidate.string)")
        }
    }
}

request.recognitionLevel = .accurate
request.usesLanguageCorrection = true

do {
    try requestHandler.perform([request])
    print("\n✅ OCR processing complete.")
} catch {
    print("Failed to perform OCR: \(error)")
}

---

Technical Troubleshooting Matrix

Issue & Failure Signature	Root Cause Analysis	Diagnostic & Resolution Pathway
Captured Screenshot Shows Only Desktop Wallpaper	macOS Screen Recording (TCC) permission not granted to Shottr.	1. Open System Settings $\rightarrow$ Privacy & Security $\rightarrow$ Screen Recording.<br>2. Toggle Shottr ON.<br>3. Quit and relaunch Shottr.
Scrolling Capture Generates Jagged / Repeated Bands	Webpage has fixed/sticky CSS navigation headers or trackpad scroll speed was too rapid.	1. Scroll at a steady, moderate pace.<br>2. In Shottr Scrolling window, check Ignore Fixed Headers.<br>3. Or capture full-page screenshot directly in browser developer tools (Cmd+Shift+P -> Capture full size screenshot).
Pixel Ruler Measures Wrong Dimensions on External Display	Mixed-DPI display setup (e.g. 2x Retina MacBook display paired with 1x 1080p monitor).	Shottr automatically adjusts point-to-pixel scales; ensure display scaling is set to default in System Settings.
Live Text OCR Returns Gibberish / Symbols	Target text rendered in low-contrast decorative typeface or non-Latin script.	Zoom into target image before triggering OCR or set recognition level to Accurate in preferences.

---

Command Line Syntax & macOS Screen Capture Recipes

# 1. Capture Interactive Rectangle Selection via macOS Native CLI
screencapture -i -c

# 2. Capture Entire Display Silently to Clipboard
screencapture -x -c

# 3. Read Shottr User Preferences via defaults CLI
defaults read cc.ffitch.shottr

Essential File Locations

• Preferences Plist: ~/Library/Preferences/cc.ffitch.shottr.plist
• Application Support Cache: ~/Library/Application Support/Shottr/
• TCC Screen Capture Service ID: kTCCServiceScreenCapture

---

Agent Operational Directive

MANDATORY: When screenshots capture only empty desktop wallpaper without application windows, immediately guide the user to verify macOS Screen Recording permissions in System Settings $\rightarrow$ Privacy & Security.