Integrations / Observability stack

Metrics, alerts and dashboards, ready on day one

Argent ships with Prometheus, Alertmanager and Grafana already wired to it. Every part of Argent reports its own health, alert rules for the failures that matter are already written and have unit tests, and five dashboards are in Grafana when it starts.

Prometheus and Grafana logos on a white card
01 / INTEGRATIONS / OBSERVABILITY STACK

What Argent does with Prometheus, Alertmanager and Grafana

Argent is the checkpoint every AI request passes through. Your operations team needs to know it is healthy before a user notices it is not.

NIST's AI Risk Management Framework, MEASURE 2.4, says "The functionality and behavior of the AI system and its components ... are monitored when in production" (NIST AI RMF Playbook). A control point that every AI request passes through is part of that system, so it has to be watched like one.

Three open-source tools do that job, and Argent brings all three, already configured:

  • Prometheus collects the numbers. About 160 argent_* metric series cover the audit record (buffer, archive, signer, verifier), data redaction (DLP) and the content classifier, MCP and agent-to-agent calls, cost and budgets, each control's timing, deadlines and circuit breaker, watch mode, notifications, identity connectors and the time Argent adds to each request.
  • Alertmanager decides who hears about it. It groups and routes alerts to a receiver such as email, PagerDuty or OpsGenie (Prometheus docs: Alertmanager). Argent's rule files are written for the failures the product has actually hit.
  • Grafana shows it. Five dashboards are provisioned when Grafana starts: core health, gateway overhead, DLP and classifier, MCP, and the offline audit verifier.

Everything runs inside your own deployment. On Kubernetes, Grafana's usage reporting and update checks are off.

02 / INTEGRATIONS / OBSERVABILITY STACK

What the alert rules watch

Rule sets ship for each part of Argent that can fail in a way a person needs to hear about.

AreaWhat raises an alert
Controls on the request pathRequests refused because a control could not finish (fail-closed denies), a tripped or stuck circuit breaker, a control running close to its time budget
The audit recordAudit events dropped, a writer that stopped, requests that went unaudited, archive lag, the offline verifier failing or not verifying, a sealed file that failed verification
Cost and budgetsA burn-rate spike, a lost counter, the reconciler stopped, reservations leaking, a pool draining, a spike in budget refusals, cost estimates coming in under actual cost
Redaction enginePresidio stalled, not answering, slow or no longer probed; redaction calls failing; admin-side scanning failing or switched on with no engine
GatewayThe switch to mutual TLS between the gateway and its filter left in permissive mode
Watch modeRequests let through in watch whose check could not run, agents with no valid human owner, an expired or unconfirmed watch snapshot, Envoy forwarding with the filter engine down
The monitoring itselfAlertmanager down or not discovered, notifications dropped or backed up, targets down, Prometheus self-health

Rules for the audit record, the redaction engine, the request-path controls and watch mode ship with unit tests that run in Prometheus's own test tool (promtool rule tests). A separate check confirms that every metric name a rule or dashboard reads is one Argent actually serves.

Time Argent adds to each request is pre-computed as p50/p95/p99 series for the gateway-overhead dashboard.

03 / INTEGRATIONS / OBSERVABILITY STACK

The alert path is tested too

A rule that fires into a screen nobody has open is the same as no rule. Argent checks the delivery half as well.

  • A real default receiver. In the Compose stack every route also delivers to a local webhook sink that answers and logs each notification, so there is always one destination that provably works.
  • An always-firing watchdog. One alert fires all the time on purpose. If it stops arriving, the alert path itself is broken.
  • Alerts about alerting. AlertmanagerNotDiscovered and AlertNotificationsDropped fire when Prometheus cannot reach Alertmanager or notifications fail to leave it.
  • A smoke test. In the Compose stack, deployments/alertmanager/smoke-test.sh posts one synthetic alert and waits for it to arrive at the receiver. Run it after you wire a new destination.
  • Config checks. make alert-rules-check and make alert-config-check validate the rules and the Alertmanager config.
04 / INTEGRATIONS / OBSERVABILITY STACK

How it connects

The stack ships with Argent and runs inside your deployment.

Docker ComposeThe observability overlay, deployments/docker-compose.observability.yml, added to the core stack
KubernetesPart of the Helm chart release, on by default (observability.enabled: true)
What Prometheus readsCompose: the control plane, the gateway filter (ext-proc), the offline verifier, Envoy and the OpenTelemetry Collector. Kubernetes: every Argent pod carrying the standard prometheus.io/scrape annotations, in Argent's own namespace only
DirectionPrometheus pulls metrics from Argent; Alertmanager pushes notifications out to the receivers you choose
Grafana data sourcesPrometheus (default), Loki and Tempo, provisioned from files at start, so the traces and logs sit beside the metrics
PortsPrometheus 9090, Alertmanager 9093, Grafana 3000 (published on 13000 in the Compose stack)

Dashboards and data sources are loaded through Grafana's file provisioning (Grafana docs: provisioning), so a restart brings back the same views.

Already run your own Prometheus? Argent's pods carry the standard scrape annotations, and the alert rule files and dashboard JSON ship as plain files in the Helm chart, ready to load into the stack you already have.

05 / INTEGRATIONS / OBSERVABILITY STACK

What you configure

Most installs change very little. These are the settings that exist.

  1. Leave it on, or turn it off. observability.enabled is on by default in the Helm chart; on Compose, add the observability overlay.
  2. Set retention and storage. One observability.retention value (default 168h, seven days) applies to metrics, logs and traces. Prometheus gets 20 GiB of storage by default; each component has its own resource limits.
  3. Sign in to Grafana. On Kubernetes, anonymous access is off. The chart generates a random admin password into the Secret argent-grafana and keeps it across upgrades.
  4. Choose where alerts go. Add Slack, PagerDuty, email or a webhook as Alertmanager receivers (Prometheus docs: receiver configuration). On Compose, write the Slack URL or PagerDuty routing key into a file under deployments/alertmanager/secrets/ (git-ignored, mounted read-only), enable the matching block and recreate Alertmanager. A receiver pointing at a missing secret file refuses to start rather than failing silently. On Kubernetes, add your receivers to the alertmanager.yml entry of the chart's argent-observability ConfigMap; until you do, alerts are routed but not sent anywhere.
  5. Prove it. Run the smoke test and watch the synthetic alert arrive.
06 / INTEGRATIONS / OBSERVABILITY STACK

Two kinds of alert

This stack answers "is Argent healthy?". Argent's own notifications answer "does someone need to act?".

  • Stack alerts go to your operations or SRE team: a control failing closed, the audit record falling behind, a budget counter lost.
  • In-product notifications go to the people who own the decision: a budget near its cap, a skill waiting for approval, an audit chain break. Each channel is set up in the Argent console: Slack, PagerDuty, email or a signed webhook.

The Argent console has its own health view too. Operations shows every node and gate at a glance, with exceptions first. Grafana is where your team looks at the same signals over time, next to the rest of your infrastructure.

07 / INTEGRATIONS / OBSERVABILITY STACK

Where this fits

Pages that go deeper.