
Follow one request through every hop
Argent sends a trace of every request from its gateway, controls and control plane to an OpenTelemetry Collector it ships with. Traces land in Tempo, Grafana Alloy ships Argent's logs to Loki, and a request's trace shares the ID of its signed audit record.
What Argent does with OpenTelemetry, Tempo and Loki
When a request is slow or refused, the question is where. A trace answers it.
NIST's AI Risk Management Framework, MEASURE 2.4, says "The functionality and behavior of the AI system and its components ... are monitored when in production" (NIST AI RMF Playbook). A control point that every AI request passes through is part of that system, so it has to be watched like one.
- The OpenTelemetry Collector receives Argent's traces over OTLP, the OpenTelemetry protocol (OpenTelemetry docs: Collector, OTLP specification).
- Tempo keeps the traces (Grafana docs: Tempo). Each request's filters and each control it passes through are recorded as spans, with their timing.
- Loki keeps the logs (Grafana docs: Loki). Grafana Alloy (Grafana docs: Alloy) ships every Argent container's logs to Loki, labeled by service.
- Grafana shows all of it. Tempo and Loki are provisioned as Grafana data sources next to Prometheus, so traces, logs and metrics are in one place.
All of it runs inside your deployment.
One ID from trace to signed record
A trace and an audit record that describe the same request should be easy to match. In Argent they share an ID.
- Your trace, continued. If the caller already sends a W3C
traceparentheader (W3C Trace Context), Argent's work appears inside the caller's own distributed trace, as a child of the caller's span. - Otherwise, the audit ID. With no incoming trace, the span's trace ID is derived from the request's audit trace ID, so the Tempo trace and the signed audit row carry the same sixteen bytes.
- Back to the access log. Envoy's
x-request-idis kept on the span, so a trace can still be joined to the access-log line. - Timing that cannot be changed unnoticed. Separately from the trace, each signed audit row records when each control ran and how long it took. The replay view also splits a request's time into provider wait, upload and the time Argent itself added. See Session Replay.
Telemetry never slows a request
Telemetry is never load-bearing. A dead or wedged trace backend cannot slow, block or fail an AI request.
- Bounded and non-blocking. Spans wait in a fixed queue of 4,096. When it is full, new spans are dropped rather than holding up the request.
- No hanging exports. Each export call has a time limit, so a collector that accepts the connection and never answers cannot pin Argent's exporter.
- Failures are visible. Export errors are logged and counted in a metric, so empty Grafana panels can be told apart from "no traffic".
- Boots without it. On Compose, the core stack starts healthy without the observability overlay; traces are simply dropped until a collector is there.
- The collector protects itself. In the Helm chart, the first step in every collector pipeline starts refusing new data well before the container's memory limit, so it sheds load instead of being killed (OpenTelemetry Collector: memory limiter).
How it connects
Argent's services push traces to the collector, which writes them to Tempo. Logs reach Loki through Grafana Alloy.
| Protocol | OTLP over gRPC (port 4317) or HTTP (port 4318) |
| Who sends | The control plane, the gateway filter (ext-proc) and the offline verifier, through OTLP_ENDPOINT, which the Helm chart sets to otel-collector:4317 when observability is on |
| Traces | Collector to Tempo |
| Metrics | Argent's own metrics are scraped by Prometheus directly (see the Prometheus page); the collector exposes its own metrics, including the Presidio health probe on Compose, on an endpoint Prometheus scrapes |
| Logs | Grafana Alloy to Loki. Compose: every container, labeled by service. Kubernetes: Argent's own namespace only, read through the Kubernetes API |
| Health probe | In the Compose stack, the collector also checks the Presidio redaction engine's health and reports it as a metric |
| Where it runs | The Compose observability overlay, or the Helm chart release (on by default) |
What you configure
Tracing is on by default. These are the settings that exist.
- Choose how much to trace.
ARGENT_TRACINGison(every request, the default, unless the caller's own trace header marks it not sampled),off, or a sampling ratio between 0 and 1. The decision is a pure function of the trace ID, so a request is sampled the same way on every replica. - Set retention and storage. The same
observability.retentionvalue (default168h) sets how long Tempo keeps traces and Loki keeps logs. Tempo and Loki each get 10 GiB of storage by default. - Size the collector and Tempo. Both have memory limits in the chart, and their Go memory limit is set to 90% of it automatically, so one change is enough.
- Keep or drop log shipping.
observability.alloy.enabledturns Kubernetes log shipping to Loki on or off. - Open Grafana's Explore view. Search Tempo by trace ID, the same ID as on the signed audit row, or filter Loki by service.
Two OpenTelemetry paths
This page is about Argent's own traces and logs, for the team that runs it. There is a second, separate path.
- Argent's own telemetry (this page): spans from Argent's services, sent to the bundled collector and Tempo, plus their logs in Loki.
- Your coding tools' telemetry: Claude Code, Codex CLI and Gemini CLI can send their own usage telemetry to Argent over OTLP. Argent keeps the counts per agent, drops the content and writes a signed record. See OpenTelemetry from AI coding tools.
To register and test a Splunk HTTP Event Collector as the destination for Argent's signed record, see Splunk HTTP Event Collector.
Where this fits
Pages that go deeper.
- Same stack, other half: metrics, alert rules and dashboards on the Prometheus, Alertmanager and Grafana page.
- How it is installed: Docker Compose or Kubernetes with Helm.
- Capabilities: Session Replay, Agent Behavior Watch, Watch Before You Enforce, Air-Gapped Install and all capabilities.
- Goals: See all AI in your company, audit and accountability, sovereign AI and all goals.
- Use cases: securing AI coding agents, governing customer-facing AI assistants and AI in regulated and air-gapped environments.
- Where it runs: air-gapped on-premises, private cloud or your public cloud VPC. Compare them on deployment options.
- Industries: Financial Services, Retail, Healthcare, Public Sector, Insurance and all industries.
- More integrations: every system Argent connects to, grouped by type, on the integrations page.
Want this running on your own use cases?
All downloads