walkeros-using-logger

Use when working with walkerOS sources/destinations to understand standard logging patterns, replace console.log, or add logging to external API calls. Covers DRY principles, when to log, and migratio

Files1

View on GitHub

SKILL.md

GitHub

Using the walkerOS Logger

Overview

The logger is walkerOS's standard logging system, available in all sources and destinations via env.logger or logger parameter. It provides scoped, level-aware logging that replaces console.log.

Core principle: Don't log what the collector already logs, and never log a failure yourself. A step reports a failure by throwing a StepError; the collector writes the one ERROR line per cause. Log only meaningful operations like external API calls and transformations, at debug.

Logger Access

In Sources

export const sourceFetch = async (
  config: PartialConfig,
  env: Types['env'], // env.logger is available here
): Promise<FetchSource> => {
  // Logger is scoped automatically by collector: [type:sourceId]
  env.logger.info('Server listening on port 3000');
};

In Destinations

export const destinationDataManager: DestinationInterface = {
  async init({ config, env, logger }) {
    // logger parameter is scoped automatically: [datamanager]
    logger.debug('Auth client created');
  },

  async push(event, { config, data, env, logger }) {
    // logger parameter is scoped: [datamanager]
    logger.debug('API response', { status: 200 });
  },
};

Note: You don't need to create or configure the logger—it's provided automatically with proper scoping.

Logger Methods

interface Logger.Instance {
  error(message: string | Error, context?: unknown | Error): void;
  warn(message: string | Error, context?: unknown | Error): void;
  info(message: string | Error, context?: unknown | Error): void;
  debug(message: string | Error, context?: unknown | Error): void;
  throw(message: string | Error, context?: unknown): never;
  json(data: unknown): void;
  scope(name: string): Logger.Instance;
}

Log Levels

  • ERROR (0): Always visible—use for fatal/critical errors only
  • WARN (1): Degraded state, config issues, transient failures
  • INFO (2): High-level operations (server startup, event processed)
  • DEBUG (3): Low-level details (API calls, transformations)

Default: ERROR only (must configure to see WARN/INFO/DEBUG)

Context Parameter

All methods accept optional structured context:

logger.debug('Sending to API', {
  endpoint: '/events',
  method: 'POST',
  eventCount: 5,
});
// Output: DEBUG [datamanager] Sending to API { endpoint: '/events', method: 'POST', eventCount: 5 }

When to Log (and When NOT to)

❌ DON'T Log These (Collector Handles)

  • Init status: "Initializing...", "Init started", "Init complete"
  • Push status: "Processing event...", "Event received"
  • Generic status: "Settings validated", "Config loaded"
  • Duplicate scoping: Don't add source/dest name (already in scope)

Why: Collector can log these automatically since it calls init/push and has scoped logger.

✅ DO Log These (Meaningful Operations)

  • External API calls: Before/after with request/response details
  • Auth operations: Token refresh, client creation/failures
  • Transformations: Complex mappings or data processing
  • Notes that are not failures: a deliberate decision, an advisory about configuration (warn: walkerOS acted, the intent is still met)
  • Never failures: a missing setting, a failed request or an unsendable event is a thrown StepError, not a log line

Usage Patterns

Pattern 1: Validation Errors (Throw a StepError)

import { StepError } from '@walkeros/core';

// ✅ GOOD - A missing setting is a failure: throw, never return or log
async init({ config }) {
  const { apiKey, projectId } = config.settings || {};

  if (!apiKey)
    throw new StepError('Config settings apiKey missing', { code: 'config' });

  if (!projectId)
    throw new StepError('Config settings projectId missing', {
      code: 'config',
    });
}

Why a thrown StepError:

  • The collector turns the throw into a failed push (or a failed start), counts it, and writes one ERROR line per cause and minute with the code
  • A step that also logged the failure would write a second line for the same cause
  • code and retryable tell a server source whether to ask the producer to resend (config, missing-input and a 4xx are not retryable)
  • logger.throw only throws now. It writes no log line, so it is not a way to report a failure

Never return early or return false from init instead of throwing: a step that returns normally counts as delivered.

Pattern 2: External API Calls

import { StepError } from '@walkeros/core';

// ✅ GOOD - Trace the call at debug, throw on a failure
async push(event, { config, logger }) {
  const endpoint = 'https://api.vendor.com/events';

  logger.debug('Calling API', { endpoint, method: 'POST', eventId: event.id });

  const response = await fetch(endpoint, {
    method: 'POST',
    body: JSON.stringify(event),
  });

  logger.debug('API response', { status: response.status, ok: response.ok });

  if (!response.ok)
    // The status sets retryable (429 and 5xx retry, other 4xx do not). The
    // response body goes on `body`, never into the message, and never into a
    // log line.
    throw new StepError(`HTTP ${response.status}`, {
      code: 'transport',
      status: response.status,
      body: await response.text(),
    });
}

Prefer sendWeb / sendServer: they already throw a StepError with the status and body.

Pattern 3: Auth Operations

import { StepError } from '@walkeros/core';

// ✅ GOOD - Trace the client creation, throw on failure
async init({ config, logger }) {
  try {
    const authClient = await createAuthClient(config.settings);
    logger.debug('Auth client created');
    return {
      env: { authClient },
    };
  } catch (error) {
    throw new StepError('Authentication failed', {
      code: 'config',
      cause: error,
    });
  }
}

The SDK's own message stays on cause, so a vendor error that carries personal data never reaches a log line.

Pattern 4: Server Startup (Sources)

// ✅ GOOD - Log server listening (high-level info)
if (settings.port !== undefined) {
  server = app.listen(settings.port, () => {
    env.logger.info(
      `Express source listening on port ${settings.port}\n` +
        `   POST ${settings.path} - Event collection (JSON body)\n` +
        `   GET ${settings.path} - Pixel tracking (query params)\n` +
        `   OPTIONS ${settings.path} - CORS preflight`,
    );
  });
}

Anti-Patterns (What NOT to Do)

❌ BAD: Verbose Init Logging

async init({ logger }) {
  logger.debug('Data Manager init started'); // Redundant
  logger.info('Data Manager initializing...'); // Redundant
  logger.debug('Settings validated'); // Redundant

  const authClient = await createAuthClient();
  logger.debug('Auth client created'); // OK

  logger.info('Data Manager ready'); // Redundant
}

Problem: Collector knows when init is called. Only log meaningful operations (auth client creation).

❌ BAD: Redundant Push Logging

async push(event, { logger }) {
  logger.debug('Processing event', {
    // Redundant
    name: event.name,
    id: event.id,
  });

  // Do work...

  logger.info('Event processed'); // Redundant
}

Problem: Collector knows when push is called and can log automatically.

❌ BAD: Using console.log

// ❌ NEVER use console.log in sources/destinations
console.log('Processing event:', event.name);

// ✅ Use logger instead
logger.debug('API call', { endpoint });

Migration Checklist

When updating a source/destination to use the logger:

  • Remove all console.log, console.warn, console.error statements
  • Remove verbose init/push status logging (let collector handle)
  • Throw a StepError with a code for every failure (apiKey missing, failed request, unsendable event); never log a failure and return
  • Add logger.debug before external API calls (with endpoint, method)
  • Add logger.debug after external API calls (with response status)
  • Add logger.debug for auth operations (client creation, token refresh)
  • Use structured context (objects) instead of string concatenation
  • Verify tests still pass with logger mocked

Testing with Logger

Use createMockLogger from @walkeros/core to check debug traces. To prove that a failure is reported, start a real collector (startFlow), mock only the network or the vendor global, and read the push result; a mocked collector cannot see loss:

import { startFlow } from '@walkeros/collector';

test('a missing apiKey fails the push', async () => {
  const { elb } = await startFlow({
    destinations: { vendor: { code: destination, config: { settings: {} } } },
  });

  const result = await elb({ name: 'page view' });

  expect(result.ok).toBe(false);
  expect(result.failed?.vendor.reason?.code).toBe('config');
});

Log Level Configuration

Default log level is ERROR. To see INFO/DEBUG logs:

import { startFlow } from '@walkeros/collector';

const { elb } = await startFlow({
  logger: {
    level: 'DEBUG', // Show all logs
  },
  destinations: {
    /* ... */
  },
});

Levels:

  • 'ERROR': Only errors (default)
  • 'WARN': Errors + warnings
  • 'INFO': Errors + warnings + info
  • 'DEBUG': Everything

Level contract: ERROR means the user's intent is not met (an event was not delivered, a setting has no effect). WARN means walkerOS acted and the intent is still met. INFO is lifecycle and self-healing. DEBUG is the per-event trace and deliberate decisions: unmet consent, a skipped event and a consent wait are debug, never a failure.

Format: on a server flow, "collector": { "logger": { "format": "json" } } prints one JSON object per line (severity, message, time, scope, context), which platforms such as Cloud Run read as structured logs. A web flow rejects format. The flow's own handler and the runtime's handler both see every line, so forwarding to a monitoring service needs no parsing.

Related Skills

Key Files:

Choosing a level on public endpoints

A server source endpoint is exposed to the open internet. Level choice decides whether ambient scanner noise turns into pages.

Situation Level Also
Unparseable body, unknown path, oversized payload debug count on collector.status.sources.<id>.rejected
Event the collector rejects as invalid warn with the reason counted as rejected, answered 400
Settled ok: false arriving after the HTTP ack warn nothing to answer, the ack already went out
Handler fault, rejected push promise error answered 500

Rules that follow from the table:

  • Never error for anything a stranger can trigger. On Cloud Run an error-level stack trace on stderr opens an Error Reporting issue, so scanner traffic becomes an alert.
  • Do not duplicate the collector's error logs. The collector already logs thrown pipeline failures at error. A source that logs them again doubles every real fault.
  • A rejected push and a settled ok: false are different things. If a helper collapses both into one return shape, log the thrown case inside that helper at error, because the call site can no longer tell them apart.

Reference pattern: packages/server/sources/express/src/index.ts, the error-boundary middleware for body-parser failures and the 400/500 split in the synchronous POST branch.

Best Practice Examples:

Needs Improvement:

Related in development

update-docs-vercel-next-js

This skill should be used when the user asks to "update documentation for my changes", "check docs for this PR", "what d...

by vercel
137,316

docstring-pytorch-pytorch

Write docstrings for PyTorch functions and methods following PyTorch conventions. Use when writing or updating docstring...

by pytorch
96,875

at-dispatch-v2

Convert PyTorch AT_DISPATCH macros to AT_DISPATCH_V2 format in ATen C++ code. Use when porting AT_DISPATCH_ALL_TYPES_AND...

dispatch
by pytorch
95,362