Integrations / Observability stack

Follow one request through every hop

Argent sends a trace of every request from its gateway, controls and control plane to an OpenTelemetry Collector it ships with. Traces land in Tempo, Grafana Alloy ships Argent's logs to Loki, and a request's trace shares the ID of its signed audit record.

OpenTelemetry logo beside a trace waterfall, one span highlighted
01 / INTEGRATIONS / OBSERVABILITY STACK

What Argent does with OpenTelemetry, Tempo and Loki

When a request is slow or refused, the question is where. A trace answers it.

NIST's AI Risk Management Framework, MEASURE 2.4, says "The functionality and behavior of the AI system and its components ... are monitored when in production" (NIST AI RMF Playbook). A control point that every AI request passes through is part of that system, so it has to be watched like one.

  • The OpenTelemetry Collector receives Argent's traces over OTLP, the OpenTelemetry protocol (OpenTelemetry docs: Collector, OTLP specification).
  • Tempo keeps the traces (Grafana docs: Tempo). Each request's filters and each control it passes through are recorded as spans, with their timing.
  • Loki keeps the logs (Grafana docs: Loki). Grafana Alloy (Grafana docs: Alloy) ships every Argent container's logs to Loki, labeled by service.
  • Grafana shows all of it. Tempo and Loki are provisioned as Grafana data sources next to Prometheus, so traces, logs and metrics are in one place.

All of it runs inside your deployment.

02 / INTEGRATIONS / OBSERVABILITY STACK

One ID from trace to signed record

A trace and an audit record that describe the same request should be easy to match. In Argent they share an ID.

  • Your trace, continued. If the caller already sends a W3C traceparent header (W3C Trace Context), Argent's work appears inside the caller's own distributed trace, as a child of the caller's span.
  • Otherwise, the audit ID. With no incoming trace, the span's trace ID is derived from the request's audit trace ID, so the Tempo trace and the signed audit row carry the same sixteen bytes.
  • Back to the access log. Envoy's x-request-id is kept on the span, so a trace can still be joined to the access-log line.
  • Timing that cannot be changed unnoticed. Separately from the trace, each signed audit row records when each control ran and how long it took. The replay view also splits a request's time into provider wait, upload and the time Argent itself added. See Session Replay.
03 / INTEGRATIONS / OBSERVABILITY STACK

Telemetry never slows a request

Telemetry is never load-bearing. A dead or wedged trace backend cannot slow, block or fail an AI request.

  • Bounded and non-blocking. Spans wait in a fixed queue of 4,096. When it is full, new spans are dropped rather than holding up the request.
  • No hanging exports. Each export call has a time limit, so a collector that accepts the connection and never answers cannot pin Argent's exporter.
  • Failures are visible. Export errors are logged and counted in a metric, so empty Grafana panels can be told apart from "no traffic".
  • Boots without it. On Compose, the core stack starts healthy without the observability overlay; traces are simply dropped until a collector is there.
  • The collector protects itself. In the Helm chart, the first step in every collector pipeline starts refusing new data well before the container's memory limit, so it sheds load instead of being killed (OpenTelemetry Collector: memory limiter).
04 / INTEGRATIONS / OBSERVABILITY STACK

How it connects

Argent's services push traces to the collector, which writes them to Tempo. Logs reach Loki through Grafana Alloy.

ProtocolOTLP over gRPC (port 4317) or HTTP (port 4318)
Who sendsThe control plane, the gateway filter (ext-proc) and the offline verifier, through OTLP_ENDPOINT, which the Helm chart sets to otel-collector:4317 when observability is on
TracesCollector to Tempo
MetricsArgent's own metrics are scraped by Prometheus directly (see the Prometheus page); the collector exposes its own metrics, including the Presidio health probe on Compose, on an endpoint Prometheus scrapes
LogsGrafana Alloy to Loki. Compose: every container, labeled by service. Kubernetes: Argent's own namespace only, read through the Kubernetes API
Health probeIn the Compose stack, the collector also checks the Presidio redaction engine's health and reports it as a metric
Where it runsThe Compose observability overlay, or the Helm chart release (on by default)
05 / INTEGRATIONS / OBSERVABILITY STACK

What you configure

Tracing is on by default. These are the settings that exist.

  1. Choose how much to trace. ARGENT_TRACING is on (every request, the default, unless the caller's own trace header marks it not sampled), off, or a sampling ratio between 0 and 1. The decision is a pure function of the trace ID, so a request is sampled the same way on every replica.
  2. Set retention and storage. The same observability.retention value (default 168h) sets how long Tempo keeps traces and Loki keeps logs. Tempo and Loki each get 10 GiB of storage by default.
  3. Size the collector and Tempo. Both have memory limits in the chart, and their Go memory limit is set to 90% of it automatically, so one change is enough.
  4. Keep or drop log shipping. observability.alloy.enabled turns Kubernetes log shipping to Loki on or off.
  5. Open Grafana's Explore view. Search Tempo by trace ID, the same ID as on the signed audit row, or filter Loki by service.
06 / INTEGRATIONS / OBSERVABILITY STACK

Two OpenTelemetry paths

This page is about Argent's own traces and logs, for the team that runs it. There is a second, separate path.

  • Argent's own telemetry (this page): spans from Argent's services, sent to the bundled collector and Tempo, plus their logs in Loki.
  • Your coding tools' telemetry: Claude Code, Codex CLI and Gemini CLI can send their own usage telemetry to Argent over OTLP. Argent keeps the counts per agent, drops the content and writes a signed record. See OpenTelemetry from AI coding tools.

To register and test a Splunk HTTP Event Collector as the destination for Argent's signed record, see Splunk HTTP Event Collector.

07 / INTEGRATIONS / OBSERVABILITY STACK

Where this fits