Shipping Gateway Logs to the OpenTelemetry Collector

Once a gateway emits well-structured JSON access logs, the next problem is getting them off the node, cleaned, and into a store — reliably, without dropping the lines that matter, and without shipping a bearer token to a third party by accident. The OpenTelemetry Collector is the standard tool for this: a vendor-neutral pipeline of receivers, processors, and exporters that decouples how the gateway writes logs from where they end up. This guide wires Kong 3.x and Envoy 1.32+ through a Collector, parses the JSON so trace_id becomes a first-class field correlated to the distributed trace, redacts secrets as a second layer of defence, batches for throughput, and exports to a store. The operational stakes are concrete: get the pipeline wrong and you either lose logs silently under load, or you leak credentials into long-retained storage.

Prerequisite concepts

This is a hands-on extension of structured access logging, which defines the JSON schema and the required fields — route, upstream, status, the two latency fields, trace_id, consumer, and ratelimit_remaining — that the pipeline below assumes are already present on each line. It also sits under gateway observability and operations, the broader treatment of how metrics, traces, and logs fit together. The trace_id correlation this guide preserves originates from the traceparent header handled in distributed tracing and context propagation; if that header is not being generated and propagated consistently, the log-to-trace join here will have nothing to join on.

The pipeline shape

A Collector configuration is three ordered stages: receivers ingest data, processors transform it, and exporters ship it. For gateway logs, the receiver is either filelog (tail a file or stdout) or otlp (accept a push from the gateway); the processors parse JSON, promote trace_id, redact, and batch; the exporter writes to your store. The diagram shows the two ingestion paths converging on a shared processor chain.

Step 1 — Emit logs the collector can read

The gateway side is covered in depth in the parent guide; the pipeline needs two things from it. For Envoy 1.32+, write the JSON access log to stdout (the StdoutAccessLog logger from the parent page) so a container runtime writes it to a file the filelog receiver can tail. For Kong 3.x, either write file-log to /dev/stdout for the same file-tailing path, or point http-log at the Collector’s OTLP HTTP endpoint:

# Kong 3.x — http-log pushing batched JSON to the collector's HTTP receiver
plugins:
  - name: http-log
    config:
      http_endpoint: http://otel-collector.internal:4318/v1/logs
      method: POST
      content_type: application/json
      queue_size: 200
      flush_timeout: 2

The file-tailing path is generally more resilient: the file buffers if the Collector restarts, and nothing sits on the request hot path. The push path suits environments where you cannot share a log file.

Step 2 — Receive, parse, and promote trace_id

The Collector config below defines both receivers and the processor chain. The critical steps are parsing the JSON body so trace_id becomes a real attribute, then promoting it onto the OpenTelemetry log record’s TraceID field so OTLP-native backends automatically link the log to its trace.

# otel-collector 0.100+ — receivers and processors for gateway logs
receivers:
  # Push path from Kong http-log / OTLP
  otlp:
    protocols:
      http: { endpoint: 0.0.0.0:4318 }
  # Tail path for Envoy (and Kong file-log) stdout
  filelog:
    include: [ /var/log/pods/*gateway*/*/*.log ]
    operators:
      # Container runtime wraps each line; extract the JSON payload first,
      # then parse it so every schema field becomes an attribute.
      - type: json_parser
        parse_from: body
        parse_to: attributes

processors:
  # Promote the parsed trace_id onto the log record so backends join to traces
  transform:
    log_statements:
      - context: log
        statements:
          - set(trace_id.string, attributes["trace_id"]) where attributes["trace_id"] != nil and attributes["trace_id"] != "-"
  # Defence-in-depth redaction: blank secret-shaped fields the gateway should
  # never have emitted, so a mistake upstream cannot leak to the store.
  redaction:
    allow_all_keys: true
    blocked_values:
      - "(?i)bearer\\s+[a-z0-9._-]+"      # any stray bearer token
      - "[0-9]{13,19}"                       # PAN-shaped numbers
    blocked_key_patterns:
      - "(?i)authorization"
      - "(?i)cookie"
      - "(?i)set-cookie"
      - "(?i)api[_-]?key"
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    send_batch_size: 8192
    timeout: 5s

Ordering matters: memory_limiter sits before batch so the Collector sheds load before it accumulates a large batch and risks an out-of-memory kill. The redaction processor runs before batch so nothing sensitive is ever buffered.

Step 3 — Export and wire the pipeline

Exporters ship the processed logs. Enable a persistent sending queue so an outage at the destination does not lose data, and assemble the ordered pipeline in the service block:

# otel-collector 0.100+ — exporter and pipeline wiring
exporters:
  otlphttp/logs:
    logs_endpoint: https://log-store.internal:4318/v1/logs
    sending_queue:
      enabled: true
      storage: file_storage        # survive collector restarts
      num_consumers: 4
      queue_size: 10000
    retry_on_failure:
      enabled: true

extensions:
  file_storage:
    directory: /var/lib/otelcol/queue

service:
  extensions: [ file_storage ]
  pipelines:
    logs:
      receivers: [ otlp, filelog ]
      # memory_limiter first, batch last — a standard, deliberate ordering
      processors: [ memory_limiter, transform, redaction, batch ]
      exporters: [ otlphttp/logs ]

With trace_id promoted onto the log record and the store indexing it, you now have a bidirectional pivot: from a log line to its trace, and from a trace span back to the raw gateway logs for that request.

Decision matrix: filelog vs OTLP push

Gotchas and failure signals

Double-encoded JSON. Container runtimes wrap each stdout line in their own JSON envelope ({"log":"{...}\n","stream":"stdout"}). If you parse only the outer layer, every gateway field stays trapped in a string. Parse the runtime envelope first, then parse the inner log payload — or use the operator chain to extract and re-parse before mapping attributes.

trace_id lands as a body field but does not join traces. Searching works, but the OTLP-native trace-to-logs link is missing. The cause is skipping the transform step that promotes the attribute onto the record’s TraceID. Setting it as an attribute is not the same as setting it on the log record itself.

Redaction regex too greedy. An over-broad blocked_values pattern can blank legitimate fields — for example a numeric pattern matching an order ID as if it were a card number. Test redaction against a sample of real lines before production, and prefer key-pattern blocking over value-pattern blocking where the field name is known.

Silent drops under load. If the Collector’s otelcol_exporter_send_failed_log_records or otelcol_processor_dropped_log_records climbs, you are losing data. Alert on both, size the sending_queue for your expected outage window, and remember the filelog path degrades more gracefully because the file retains unread lines.

Validation

• A test request’s log line arrives in the store with all schema fields intact as structured attributes.
• trace_id on the stored log record matches the traceparent trace ID of the same request’s trace, and the trace-to-logs pivot works in both directions.
• A request carrying an Authorization: Bearer header produces a stored log with no token present, confirming both gateway and collector redaction.
• Restarting the Collector loses no lines — the filelog offset resumes and the sending queue drains from disk.
• otelcol_processor_dropped_log_records and otelcol_exporter_send_failed_log_records are scraped and alerted on.
• memory_limiter precedes batch in the pipeline, and batch is the final processor before export.

FAQ

Should I use the filelog receiver or push OTLP logs directly from the gateway?

Use the filelog receiver when the gateway writes JSON to stdout or a file: the file buffers if the Collector restarts and no plugin sits on the request path. Push OTLP or HTTP directly when you cannot mount a shared log file — for example in a fully managed environment — accepting that in-gateway batching can drop lines under backpressure. Many teams run filelog on a node agent and OTLP on the gateway for different fleets.

How does the collector preserve trace correlation for logs?

Parse the JSON body so trace_id becomes a real field, then promote it to the log record’s TraceID with the transform processor. When trace_id is set on the OpenTelemetry log record itself, OTLP-native backends link the log to its trace automatically, and a store that indexes trace_id lets you pivot both ways. Skip the promotion and the ID is still a searchable body field, but you lose the automatic join.

Where should redaction happen — the gateway or the collector?

Both, in layers. The gateway must never emit secrets, because anything in the log file is already exposed on that node. The Collector then runs a redaction processor as defence in depth, masking secret-shaped fields so an upstream mistake cannot ship a token to your store. Gateway-side redaction is the primary control; the Collector processor is the safety net.

How do I keep the collector from dropping logs under load?

Add the batch processor to amortize exports, put memory_limiter ahead of it so the Collector sheds load before it OOMs, and enable a persistent sending_queue so data survives a downstream outage. With filelog, the log file is your buffer, so a restart resumes from the last read offset. Monitor otelcol_processor_dropped_log_records and the exporter’s failure metric and alert on them.

Parent: Structured Access Logging

Related

• Structured Access Logging — the JSON schema and required fields this pipeline assumes are already emitted.
• Distributed Tracing & Context Propagation — where the traceparent and trace_id the pipeline correlates on come from.
• Gateway Observability & Operations — how logs, metrics, and traces combine into an operational picture.