SIEM & SOAR

This skill covers all SIEM (Security Information and Event Management) and SOAR (Security Orchestration, Automation, and Response) technologies. It provides cross-platform expertise in log management, event correlation, detection engineering, normalization standards, and security automation. Read the relevant sibling skill for deep implementation details.

When to Use This Skill vs. a Sibling Skill

Use this skill when the question is cross-platform or conceptual:

• "Which SIEM should we choose for our environment?"
• "How do we build a detection engineering program?"
• "Compare Splunk vs. Sentinel vs. Elastic for our use case"
• "What is the best normalization standard?"
• "How do SIGMA rules work across platforms?"
• "Design our SOC architecture"
• "SIEM data onboarding strategy"
• "How much log storage do we need?"
• "SOAR vs. SIEM automation -- where do we draw the line?"

Read a sibling skill when the question is platform-specific:

• "Write an SPL correlation search" --> splunk
• "Splunk Enterprise Security risk-based alerting" --> splunk-es
• "KQL query for Sentinel analytics rule" --> sentinel
• "Elastic EQL sequence detection" --> elastic-security
• "QRadar AQL offense query" --> qradar
• "Chronicle YARA-L detection rule" --> chronicle
• "XSIAM XQL correlation" --> xsiam
• "LogScale LQL streaming query" --> logscale
• "Build an XSOAR playbook" --> xsoar
• "Splunk SOAR automation" --> splunk-soar
• "Sentinel playbook with Logic Apps" --> sentinel-playbooks

How to Approach Tasks

When you receive a request:

1. Classify the request:

   • Platform selection -- Compare SIEM/SOAR platforms against requirements
   • Architecture / Strategy -- Load references/concepts.md for foundational SIEM concepts
   • Detection engineering -- Cross-platform detection methodology, SIGMA rules, MITRE ATT&CK mapping
   • Data onboarding -- Log source strategy, normalization, parsing, enrichment
   • SOC operations -- Triage workflows, alert management, metrics (MTTD, MTTR)
   • Cost optimization -- Ingestion volume, tiering, filtering, retention policies
   • Platform-specific -- Read the appropriate sibling skill

2. Gather context -- Environment (cloud/on-prem/hybrid), team size, budget, existing tooling, compliance requirements, log volume (GB/day), retention needs

3. Analyze -- Apply SIEM-specific reasoning. Consider data volume, query performance, detection coverage, operational maturity, and total cost of ownership.

4. Recommend -- Provide prioritized recommendations with trade-offs. SIEM selection is never one-size-fits-all.

5. Qualify -- State assumptions about scale, team skill, and budget constraints

SIEM Fundamentals

Core SIEM Functions

1. Log Collection -- Aggregate logs from endpoints, network devices, cloud services, applications, and identity providers
2. Normalization -- Transform raw logs into a consistent schema (CIM, ECS, ASIM, UDM) for cross-source correlation
3. Indexing / Storage -- Store normalized events for real-time and historical search
4. Correlation -- Match patterns across multiple log sources to detect threats (correlation rules, analytics rules)
5. Alerting -- Generate alerts when correlation rules trigger, with severity and context
6. Investigation -- Provide search, drill-down, and visualization for analyst triage
7. Reporting -- Compliance reporting, SOC metrics, executive dashboards

Detection Engineering Lifecycle

1. Threat Intelligence    -->  What threats target our environment?
        |
2. Data Source Mapping    -->  Do we have visibility? (MITRE ATT&CK data sources)
        |
3. Detection Logic        -->  Write detection rules (platform-native or SIGMA)
        |
4. Testing & Validation   -->  Atomic Red Team, Caldera, manual simulation
        |
5. Tuning                 -->  Reduce false positives, add exceptions, refine thresholds
        |
6. Deployment             -->  Promote to production with severity and response actions
        |
7. Metrics & Maintenance  -->  Track detection coverage, MTTD, alert fidelity, rule decay

SIGMA Rules

SIGMA is a vendor-agnostic detection rule format that compiles to platform-specific queries:

title: Suspicious PowerShell Download Cradle
id: 3b6ab547-8ec2-4991-b9d2-2b06702a48d7
status: stable
description: Detects PowerShell download cradles commonly used by attackers
logsource:
    category: process_creation
    product: windows
detection:
    selection:
        CommandLine|contains|all:
            - 'powershell'
            - 'Net.WebClient'
        CommandLine|contains:
            - 'DownloadString'
            - 'DownloadFile'
    condition: selection
level: high
tags:
    - attack.execution
    - attack.t1059.001

SIGMA compiles to:

• Splunk SPL: index=windows sourcetype=WinEventLog:Security ... | search CommandLine="*powershell*Net.WebClient*DownloadString*"
• Sentinel KQL: SecurityEvent | where CommandLine has_all ("powershell", "Net.WebClient") and CommandLine has_any ("DownloadString", "DownloadFile")
• Elastic EQL: Process creation event with matching command line patterns
• QRadar AQL: SQL-like query against normalized fields
• Chronicle YARA-L: Event match with target.process.command_line conditions

Normalization Standards

Standard	Platform	Key Concept
CIM (Common Information Model)	Splunk	Data models with standardized field names; acceleration for dashboards
ECS (Elastic Common Schema)	Elastic	Hierarchical field naming (e.g., process.name, source.ip)
ASIM (Advanced Security Information Model)	Sentinel	Unifying parsers that normalize at query time; schema-based
UDM (Unified Data Model)	Chronicle	Google's schema for security telemetry; entity-centric
OCSF (Open Cybersecurity Schema Framework)	Cross-platform	AWS-originated open standard; growing adoption

Data Onboarding Strategy

Prioritize log sources by detection value:

Priority	Log Sources	Detection Value
P1 -- Critical	EDR, identity (AD/Entra), email gateway, firewall/proxy	Core visibility for 80% of attack techniques
P2 -- High	Cloud audit logs (AWS CloudTrail, Azure Activity, GCP Audit), DNS, DHCP	Lateral movement, cloud compromise, C2 detection
P3 -- Medium	Application logs, VPN, DLP, vulnerability scanners	Insider threat, data exfiltration, vulnerability correlation
P4 -- Low	Network flow (NetFlow/IPFIX), PCAP metadata, printer logs	Forensic enrichment, compliance, niche detections

Alert Triage Framework

Effective triage reduces Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR):

1. Tier 0 -- Automated -- SOAR handles enrichment, deduplication, known-false-positive suppression, auto-close of informational alerts
2. Tier 1 -- Triage -- Analyst validates alert, checks context (asset criticality, user risk score, related alerts), escalates or closes
3. Tier 2 -- Investigation -- Deep-dive analysis, timeline reconstruction, scope assessment, containment decisions
4. Tier 3 -- Hunt -- Proactive hypothesis-driven threat hunting using SIEM search and detection gaps

SIEM Platform Comparison

Capability	Splunk	Sentinel	Elastic Security	QRadar	Chronicle	XSIAM	LogScale
Query Language	SPL / SPL2	KQL	EQL, ES|QL, KQL, Lucene	AQL (SQL-like)	YARA-L 2.0	XQL	LQL
Normalization	CIM	ASIM	ECS	QID + DSM	UDM	XDM	Custom
Deployment	On-prem, Cloud, Hybrid	Cloud-native (Azure)	On-prem, Cloud, Hybrid	On-prem (SaaS divested)	Cloud-native (GCP)	Cloud-native	Cloud, Self-hosted
Pricing Model	Ingestion (GB/day) or workload	Ingestion (GB/day) + retention	Node-based or ingestion	EPS (events/sec)	Flat (per user)	Ingestion + compute	Ingestion (GB/day)
SOAR Built-in	Via Splunk SOAR (separate)	Playbooks (Logic Apps)	Response actions (limited)	QRadar SOAR (separate)	SOAR module	Automation Center	Via Falcon Fusion
ML / AI	MLTK, predictive analytics	Fusion ML, UEBA, Copilot	ML anomaly detection jobs	Anomaly detection	Duet AI, Mandiant TI	XSIAM Copilot, ML clustering	Statistical functions
Strengths	Mature ecosystem, SPL power, Splunkbase	Azure integration, cost tiers, Defender XDR	Open source core, EQL sequences, flexible	Automatic offense grouping, AQL familiarity	Unlimited retention, retroactive rules, Mandiant TI	Converged platform (SIEM+SOAR+XDR), AI-first	High-volume streaming, index-free, real-time
Weaknesses	Cost at scale, complexity	Azure-centric, KQL learning curve	Operational overhead (self-managed), complexity	Aging platform, SaaS discontinued	GCP-centric, limited customization	Vendor lock-in, emerging maturity	Smaller ecosystem, limited SOAR

Platform Selection Decision Tree

Start: What is your primary cloud?
  |
  ├── Azure-heavy  -->  Microsoft Sentinel (native integration with Defender XDR, Entra, M365)
  |
  ├── GCP-heavy    -->  Chronicle/Google SecOps (native GCP integration, Mandiant TI)
  |
  ├── AWS-heavy    -->  Consider Splunk Cloud, Elastic, or XSIAM (no dominant AWS-native SIEM)
  |
  └── Multi-cloud / On-prem
       |
       ├── Budget priority         -->  Elastic Security (open source core) or LogScale (competitive pricing)
       ├── Mature SOC, complex needs -->  Splunk (deepest ecosystem, SPL power)
       ├── Converged SOC platform   -->  XSIAM (SIEM + SOAR + XDR in one)
       └── High-volume, real-time   -->  LogScale (streaming architecture, index-free)

SOC Metrics

Track these metrics to measure SIEM/SOAR effectiveness:

Metric	Definition	Target
MTTD	Mean Time to Detect -- time from event to alert	< 1 hour for critical threats
MTTR	Mean Time to Respond -- time from alert to containment	< 4 hours for critical incidents
Alert Fidelity	True positives / total alerts	> 80% (below 50% = alert fatigue)
Detection Coverage	ATT&CK techniques with active detections / total relevant techniques	> 60% for top tactics
Automation Rate	Alerts handled by SOAR without human intervention	> 40% for mature SOCs
Dwell Time	Attacker presence before detection	Reduce quarter over quarter
EPS / GB per Day	Ingestion volume	Monitor for budget forecasting

Cost Optimization Strategies

SIEM costs are driven primarily by ingestion volume. Common optimization tactics:

1. Tiered storage -- Use hot/warm/cold/frozen tiers. Not all data needs fast search (Splunk SmartStore, Sentinel basic logs, Elastic frozen tier).
2. Filtering at source -- Drop noisy, low-value events before ingestion (verbose debug logs, health checks, success-only auth events).
3. Summary indexing -- Pre-aggregate statistics for reporting; keep raw data shorter.
4. Log routing -- Send compliance-only logs to cheap storage (S3, blob); send security-relevant logs to SIEM.
5. Data model acceleration -- Pre-compute common searches to avoid expensive full-index scans.
6. Commitment tiers -- Most vendors offer discounts for committed ingestion volumes (Sentinel commitment tiers, Splunk workload pricing).
7. Event sampling -- For extremely high-volume, low-fidelity sources (e.g., NetFlow), sample instead of ingesting 100%.

Anti-Patterns to Watch For

1. "Collect everything, detect later" -- Ingesting every log without a detection plan leads to massive costs and no security value. Map data sources to specific detections.
2. "One alert per threat" -- Single-event detections produce noise. Use correlation (multi-event, multi-source) and risk-based scoring.
3. "SIEM as a log archive" -- A SIEM without active detection rules is an expensive log store. Invest in detection engineering.
4. "Copy-paste vendor rules" -- Default rules without tuning generate alert fatigue. Every rule needs environment-specific tuning.
5. "Ignoring the data pipeline" -- Normalization, parsing, and enrichment quality determine detection quality. Garbage in, garbage out.
6. "SOAR without mature processes" -- Automating bad processes makes them faster, not better. Define playbooks manually before automating.
7. "Single-platform lock-in" -- Over-reliance on one vendor's ecosystem makes migration painful. Use SIGMA for portable detections where possible.

Technology Routing

Read these sibling skills for platform-specific expertise:

Request Pattern	Route To
SIEM Platforms
Splunk, SPL, search heads, indexers, forwarders, SmartStore	splunk (see its Version-Specific Guidance)
Splunk Enterprise Security, ES, notable events, RBA	splunk-es
Microsoft Sentinel, KQL, ASIM, Fusion, analytics rules	sentinel
Elastic Security, EQL, ES|QL, detection engine, Fleet	elastic-security (see its Version-Specific Guidance)
IBM QRadar, AQL, offenses, DSMs, Ariel	qradar
Google Chronicle, YARA-L, UDM, SecOps	chronicle
Palo Alto XSIAM, XQL, AI-driven SOC	xsiam
CrowdStrike LogScale, LQL, streaming SIEM	logscale
SOAR Platforms
SOAR platform comparison, playbook strategy	soar
Cortex XSOAR, playbook IDE, war rooms	xsoar
Splunk SOAR, Phantom, visual playbooks	splunk-soar
Sentinel Playbooks, Logic Apps automation	sentinel-playbooks
Tines, no-code security automation	tines
Torq, hyperautomation, SOC copilot	torq

Reference Files

Load these for deep foundational knowledge:

• references/concepts.md -- SIEM/SOAR foundational concepts: log management lifecycle, event correlation theory, normalization standards, SIGMA rule language, detection engineering methodology, SOC maturity model. Read for "how does SIEM work" or cross-platform architecture questions.