wiztelemetry-tracing

Use when working with WizTelemetry Tracing extension for KubeSphere, including installation, configuration, and tracing query API

Files1

View on GitHub

SKILL.md

GitHub

WizTelemetry Tracing

Overview

WizTelemetry Tracing is an extension component in the KubeSphere Observability Platform that provides distributed tracing functionality based on the OpenTelemetry standard.

When to Use

  • Installing or configuring the WizTelemetry Tracing extension
  • Understanding tracing architecture (Generator + Operator + Collector + Agent)
  • Using the tracing query API to query traces, spans, service graphs
  • Configuring OpenTelemetry auto-instrumentation for applications

Components

Component Description Default Enabled
generator WizTelemetry Tracing Generator: generates service graphs from tracing data (StatefulSet) true
operator OpenTelemetry Operator: manages OpenTelemetry Collector and auto-instrumentation true
collector OpenTelemetry Collector for WizTelemetry: receives traces, exports to Vector and OpenSearch true
agent WizTelemetry Tracing Agent: collects tracing data from local log files (DaemonSet) false
demo OpenTelemetry Demo: generates sample tracing data for demonstration false

Dependencies

  • WizTelemetry Platform Service (whizard-telemetry): Required
  • OpenSearch (opensearch): Required

Installation

Prerequisites

REQUIRED: Complete all steps in order before generating InstallPlan.

Step 1: Get Available Clusters and Confirm Target

CRITICAL: DO NOT proceed until target clusters are determined.

Step 1.1: Get available clusters

kubectl get clusters -o jsonpath='{.items[*].metadata.name}'

Step 1.2: Determine target clusters

  • If user explicitly specified target clusters in the request -> Use those clusters directly, proceed to Step 2
  • If user did NOT specify target clusters -> Ask user to confirm which clusters to deploy to, then proceed to Step 2

Ask user (if not specified):

Available clusters: host, dev
Which clusters do you want to deploy WizTelemetry Tracing to?

Step 2: Get Latest Version (if not provided by user)

MUST do this to get the latest version:

kubectl get extensionversions -n kubesphere-system -l kubesphere.io/extension-ref=wiztelemetry-tracing -o jsonpath='{range .items[*]}{.spec.version}{"\n"}{end}' | sort -V | tail -1

This outputs the latest version (e.g., 1.0.6). Note this down - you'll use it in the InstallPlan.

Step 3: Get Generator Endpoint

Get a node IP from the target cluster:

kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="ExternalIP")].address}' 2>/dev/null || \
kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}'

This outputs a node IP (e.g., 192.168.1.100). Use this as the generator endpoint.

If multiple shards (e.g., generator.shardCount: 3), generate all endpoints:

NODE_IP="<NODE_IP>"
for i in 0 1 2; do
  PORT=$((32318 + i))
  echo "  - http://${NODE_IP}:${PORT}"
done

Replace <NODE_IP> with the actual node IP from the previous command.

Step 4: Confirm Configuration with User

Before creating the InstallPlan, confirm the following with the user:

I'll install WizTelemetry Tracing with the following configuration:
- Version: <VERSION>
- Target clusters: <TARGET_CLUSTERS>
- Generator endpoint: http://<NODE_IP>:32318
- OpenSearch endpoint: <OPENSEARCH_ENDPOINT>

Do you want to proceed? (yes/no)

If user confirms, proceed to create the InstallPlan. If not, adjust the configuration based on user feedback.

Step 5: Create InstallPlan

CRITICAL: InstallPlan metadata.name MUST be wiztelemetry-tracing. DO NOT use any other name.

CRITICAL: config field is YAML format. You MUST:

  • Use the config structure exactly as shown in the template
  • DO NOT add configuration fields that are not shown in the template
  • DO NOT modify the structure or hierarchy

CRITICAL: All placeholders MUST be replaced with actual values. DO NOT leave them as placeholders.

Based on your selections:

  • Target clusters: User-confirmed cluster names
  • OpenSearch endpoint: User-provided (default: https://opensearch-cluster-data.kubesphere-logging-system.svc:9200)
  • OpenSearch credentials: User-provided (default user: admin)

Template

apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: wiztelemetry-tracing
  namespace: kubesphere-system
spec:
  extension:
    name: wiztelemetry-tracing
    version: <VERSION>  # From Step 2
  enabled: true
  upgradeStrategy: Manual
  config: |
    global:
      generator:
        endpoints:
          - http://<HOST_NODE_IP>:32318
      storage:
        opensearch:
          auth:
            strategy: basic
            user: <OPENSEARCH_USER>
            password: <OPENSEARCH_PASSWORD>
          endpoints:
            - <OPENSEARCH_ENDPOINT>
    generator:
      shardCount: 1
      service:
        nodePort: 32318
  clusterScheduling:
    placement:
      clusters:
        - <TARGET_CLUSTERS>

Replace placeholders:

  • <VERSION>: From Step 2 (e.g., 1.0.6)
  • <HOST_NODE_IP>: Node IP from Step 3 (e.g., 192.168.1.100)
  • <OPENSEARCH_USER>: OpenSearch username
  • <OPENSEARCH_PASSWORD>: OpenSearch password
  • <OPENSEARCH_ENDPOINT>: OpenSearch endpoint (e.g., https://opensearch-cluster-data.kubesphere-logging-system.svc:9200)
  • <TARGET_CLUSTERS>: User-confirmed cluster names

Enable Agent (Log File Collection)

To enable tracing data collection from local log files, set agent.enabled: true in the config. This deploys a DaemonSet that scans log files and forwards traces to the generator.

Enable Demo

To enable the OpenTelemetry Demo (generates sample trace data), set demo.enabled: true in the config.

Send Traces to Tempo

To forward traces to Tempo instead of OpenSearch, modify the collector.collector.config:

collector:
  enabled: true
  collector:
    config: |
      exporters:
        otlp:
          endpoint: <TEMPO_DISTRIBUTOR_GRPC_ENDPOINT>
          headers:
            x-scope-orgid: wiztelemetry-tracing-ks
          tls:
            insecure: true
      service:
        pipelines:
          traces:
            exporters:
              - otlp
              - otlphttp

Multiple Generator Instances

To scale the generator for higher throughput, increase generator.shardCount and update global.generator.endpoints:

global:
  generator:
    endpoints:
      - http://<HOST_NODE_IP>:32318
      - http://<HOST_NODE_IP>:32319
      - http://<HOST_NODE_IP>:32320

generator:
  shardCount: 3
  service:
    nodePort: 32318

shardCount must match the number of endpoints in global.generator.endpoints. The nodePort of each shard increases in sequence starting from the configured service.nodePort.

Configuration Parameters

Global Parameters

DNS Service

Parameter Type Default Description
global.dnsService string coredns DNS service name

Generator Global Parameters

Parameter Type Default Description
global.generator.endpoints list ["http://<ip>:32318"] Endpoints of all generator shards. Tracing data is routed to different shards by traceId.

OpenSearch Storage Parameters

Parameter Type Default Description
global.storage.opensearch.auth.strategy string basic Auth strategy
global.storage.opensearch.auth.user string admin OpenSearch username
global.storage.opensearch.auth.password string admin OpenSearch password
global.storage.opensearch.endpoints list OpenSearch endpoint URLs
global.storage.opensearch.index.prefix string wiz-tracing-span Index prefix
global.storage.opensearch.index.timestring string %Y.%m.%d Index time format (strftime pattern)

Generator Parameters

Parameter Type Default Description
generator.shardCount int 1 Number of generator shards. Must equal the number of entries in global.generator.endpoints.
generator.image.tag string v1.0.2 Generator image tag
generator.image.imagePullPolicy string IfNotPresent Image pull policy
generator.service.nodePort int 32318 NodePort of the first shard. Subsequent shards use consecutive ports.
generator.resources.limits.cpu string 2 CPU limit
generator.resources.limits.memory string 2000Mi Memory limit
generator.resources.requests.cpu string 100m CPU request
generator.resources.requests.memory string 100Mi Memory request

Operator Parameters

Parameter Type Default Description
operator.enabled bool true Enable OpenTelemetry Operator
operator.fullnameOverride string wiztelemetry-tracing-operator Override name for the operator deployment

Collector Parameters

ISM Policy Parameters

Parameter Type Default Description
collector.ism_policy.enable bool true Enable OpenSearch Index State Management policy
collector.ism_policy.span_index_pattern string *wiz-tracing-span* Index pattern for span indices
collector.ism_policy.service_index_pattern string *wiz-tracing-service* Index pattern for service indices
collector.ism_policy.min_index_age string 7d Minimum index retention period
collector.ism_policy.span_index_priority int 9950 Index priority for span indices

Collector Deployment Parameters

Parameter Type Default Description
collector.enabled bool true Enable OpenTelemetry Collector
collector.collector.mode string deployment Deployment mode (deployment or daemonset)
collector.collector.replicas int 2 Number of collector replicas
collector.collector.upgradeStrategy string automatic Upgrade strategy

Agent Parameters

Parameter Type Default Description
agent.enabled bool false Enable Tracing Agent (DaemonSet)
agent.dockerRootDir string /var/lib/docker Docker root directory
agent.config.scanInterval string 1m Log file scan interval
agent.config.deletionDelay string 5m Delay before deleting processed files
agent.config.logPath string /app/logs/*/*trace*.log Path pattern for trace log files
agent.config.podLabelSelector map {} Filter pods by label selector
agent.config.namespaceLabelSelector map {} Filter namespaces by label selector
agent.config.includeNamespaces list [] List of namespaces to include
agent.config.excludeNamespaces list [] List of namespaces to exclude
agent.resources.limits.cpu string 500m CPU limit
agent.resources.limits.memory string 500Mi Memory limit
agent.resources.requests.cpu string 10m CPU request
agent.resources.requests.memory string 10Mi Memory request
agent.vector.resources.limits.cpu string 2 Vector sidecar CPU limit
agent.vector.resources.limits.memory string 2000Mi Vector sidecar memory limit
agent.vector.resources.requests.cpu string 100m Vector sidecar CPU request
agent.vector.resources.requests.memory string 100Mi Vector sidecar memory request

Demo Parameters

Parameter Type Default Description
demo.enabled bool false Enable OpenTelemetry Demo

Tracing Query API

Query Traces

curl -X POST "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/tracing.wiztelemetry.io/v1alpha1/traces" \
  -H "X-Remote-User: admin" \
  -H "Content-Type: application/json" \
  -d '{"condition": {}, "limit": 20}'

Query Spans

curl -X POST "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/tracing.wiztelemetry.io/v1alpha1/spans" \
  -H "X-Remote-User: admin" \
  -H "Content-Type: application/json" \
  -d '{"condition": {}, "limit": 20}'

Get Service Graph

curl -X POST "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/tracing.wiztelemetry.io/v1alpha1/servicegraphs" \
  -H "X-Remote-User: admin" \
  -H "Content-Type: application/json" \
  -d '{"condition": {"start": "<START_TIME>", "end": "<END_TIME>"}}'

Get Services

curl -X GET "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/tracing.wiztelemetry.io/v1alpha1/services" \
  -H "X-Remote-User: admin"

Get Tags

curl -X GET "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/tracing.wiztelemetry.io/v1alpha1/tags" \
  -H "X-Remote-User: admin"

Get Values by Tag

curl -X GET "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/tracing.wiztelemetry.io/v1alpha1/values?tags=span.kind&limit=100" \
  -H "X-Remote-User: admin"

Get Histogram

curl -X GET "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/tracing.wiztelemetry.io/v1alpha1/histogram?key=<KEY>&kind=1&interval=15m&startTime=<START>&endTime=<END>" \
  -H "X-Remote-User: admin"

Query Parameters:

Parameter Type Description
key string The key of node or edge
kind int Histogram kind: 1=RequestTotal, 2=RequestTimeAverage, 3=FailedRequestTotal, 4=ResponseTotal, 5=ResponseTimeAverage, 6=FailedResponseTotal
interval string Time interval (e.g., 15m, 1h, 1d)
startTime string Start time in seconds since epoch
endTime string End time in seconds since epoch

Get Associated Workloads

curl -X GET "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/tracing.wiztelemetry.io/v1alpha1/workloads?keys=<KEYS>&startTime=<START>&endTime=<END>" \
  -H "X-Remote-User: admin"

Query Parameters:

Parameter Type Description
keys string Comma-separated list of node keys
startTime string Start time in seconds since epoch
endTime string End time in seconds since epoch

Get/Set Apdex Thresholds

# Get apdex thresholds
curl -X GET "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/tracing.wiztelemetry.io/v1alpha1/apdex/thresholds?keys=<KEYS>" \
  -H "X-Remote-User: admin"

# Set apdex thresholds
curl -X PUT "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/tracing.wiztelemetry.io/v1alpha1/apdex/thresholds" \
  -H "X-Remote-User: admin" \
  -H "Content-Type: application/json" \
  -d '{"serviceA:serviceB": 0.5}'

OpenTelemetry Auto-Instrumentation

Create Instrumentation Resource

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://wiztelemetry-tracing-collector.wiz-telemetry-tracing:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  python:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://wiztelemetry-tracing-collector.wiz-telemetry-tracing:4318
  dotnet:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://wiztelemetry-tracing-collector.wiz-telemetry-tracing:4318
  go:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: http://wiztelemetry-tracing-collector.wiz-telemetry-tracing:4318
EOF

Enable Instrumentation via Annotations

Add the following annotation to a pod or namespace:

Language Annotation
Java instrumentation.opentelemetry.io/inject-java: "true"
NodeJS instrumentation.opentelemetry.io/inject-nodejs: "true"
Python instrumentation.opentelemetry.io/inject-python: "true"
.NET instrumentation.opentelemetry.io/inject-dotnet: "true"
Go instrumentation.opentelemetry.io/inject-go: "true"
Apache HTTPD instrumentation.opentelemetry.io/inject-apache-httpd: "true"
Nginx instrumentation.opentelemetry.io/inject-nginx: "true"
SDK only instrumentation.opentelemetry.io/inject-sdk: "true"

Annotation values:

  • "true" - inject from the namespace's Instrumentation resource
  • "my-instrumentation" - use a specific Instrumentation CR in the current namespace
  • "my-ns/my-instrumentation" - use an Instrumentation CR in a different namespace
  • "false" - do not inject

Extension Operations

Check Extension Status

kubectl get installplan wiztelemetry-tracing
kubectl get extensions wiztelemetry-tracing

Check Component Status

kubectl get statefulset -n wiz-telemetry-tracing
kubectl get deployment -n wiz-telemetry-tracing
kubectl get daemonset -n wiz-telemetry-tracing
kubectl get pods -n wiz-telemetry-tracing

Uninstall Extension

Uninstall from all clusters:

kubectl delete installplan wiztelemetry-tracing

Uninstall from specific cluster:

To remove WizTelemetry Tracing from a specific cluster, update the InstallPlan by removing that cluster from clusterScheduling.placement.clusters:

apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: wiztelemetry-tracing
spec:
  extension:
    name: wiztelemetry-tracing
    version: <VERSION>
  enabled: true
  upgradeStrategy: Manual
  clusterScheduling:
    placement:
      clusters:
        - <REMAINING_CLUSTERS>  # Remove the cluster you want to uninstall from

Related in integration

bluebubbles

Your own personal AI assistant. Any OS. Any Platform. The lobster way. 🦞

by clawdbot
7,675

camsnap

Your own personal AI assistant. Any OS. Any Platform. The lobster way. 🦞

by clawdbot
7,675

eightctl

Your own personal AI assistant. Any OS. Any Platform. The lobster way. 🦞

by clawdbot
7,675