
Metrics, alerts and dashboards, ready on day one
Argent ships with Prometheus, Alertmanager and Grafana already wired to it. Every part of Argent reports its own health, alert rules for the failures that matter are already written and have unit tests, and five dashboards are in Grafana when it starts.
What Argent does with Prometheus, Alertmanager and Grafana
Argent is the checkpoint every AI request passes through. Your operations team needs to know it is healthy before a user notices it is not.
NIST's AI Risk Management Framework, MEASURE 2.4, says "The functionality and behavior of the AI system and its components ... are monitored when in production" (NIST AI RMF Playbook). A control point that every AI request passes through is part of that system, so it has to be watched like one.
Three open-source tools do that job, and Argent brings all three, already configured:
- Prometheus collects the numbers. About 160
argent_*metric series cover the audit record (buffer, archive, signer, verifier), data redaction (DLP) and the content classifier, MCP and agent-to-agent calls, cost and budgets, each control's timing, deadlines and circuit breaker, watch mode, notifications, identity connectors and the time Argent adds to each request. - Alertmanager decides who hears about it. It groups and routes alerts to a receiver such as email, PagerDuty or OpsGenie (Prometheus docs: Alertmanager). Argent's rule files are written for the failures the product has actually hit.
- Grafana shows it. Five dashboards are provisioned when Grafana starts: core health, gateway overhead, DLP and classifier, MCP, and the offline audit verifier.
Everything runs inside your own deployment. On Kubernetes, Grafana's usage reporting and update checks are off.
What the alert rules watch
Rule sets ship for each part of Argent that can fail in a way a person needs to hear about.
| Area | What raises an alert |
|---|---|
| Controls on the request path | Requests refused because a control could not finish (fail-closed denies), a tripped or stuck circuit breaker, a control running close to its time budget |
| The audit record | Audit events dropped, a writer that stopped, requests that went unaudited, archive lag, the offline verifier failing or not verifying, a sealed file that failed verification |
| Cost and budgets | A burn-rate spike, a lost counter, the reconciler stopped, reservations leaking, a pool draining, a spike in budget refusals, cost estimates coming in under actual cost |
| Redaction engine | Presidio stalled, not answering, slow or no longer probed; redaction calls failing; admin-side scanning failing or switched on with no engine |
| Gateway | The switch to mutual TLS between the gateway and its filter left in permissive mode |
| Watch mode | Requests let through in watch whose check could not run, agents with no valid human owner, an expired or unconfirmed watch snapshot, Envoy forwarding with the filter engine down |
| The monitoring itself | Alertmanager down or not discovered, notifications dropped or backed up, targets down, Prometheus self-health |
Rules for the audit record, the redaction engine, the request-path controls and watch mode ship with unit tests that run in Prometheus's own test tool (promtool rule tests). A separate check confirms that every metric name a rule or dashboard reads is one Argent actually serves.
Time Argent adds to each request is pre-computed as p50/p95/p99 series for the gateway-overhead dashboard.
The alert path is tested too
A rule that fires into a screen nobody has open is the same as no rule. Argent checks the delivery half as well.
- A real default receiver. In the Compose stack every route also delivers to a local webhook sink that answers and logs each notification, so there is always one destination that provably works.
- An always-firing watchdog. One alert fires all the time on purpose. If it stops arriving, the alert path itself is broken.
- Alerts about alerting.
AlertmanagerNotDiscoveredandAlertNotificationsDroppedfire when Prometheus cannot reach Alertmanager or notifications fail to leave it. - A smoke test. In the Compose stack,
deployments/alertmanager/smoke-test.shposts one synthetic alert and waits for it to arrive at the receiver. Run it after you wire a new destination. - Config checks.
make alert-rules-checkandmake alert-config-checkvalidate the rules and the Alertmanager config.
How it connects
The stack ships with Argent and runs inside your deployment.
| Docker Compose | The observability overlay, deployments/docker-compose.observability.yml, added to the core stack |
| Kubernetes | Part of the Helm chart release, on by default (observability.enabled: true) |
| What Prometheus reads | Compose: the control plane, the gateway filter (ext-proc), the offline verifier, Envoy and the OpenTelemetry Collector. Kubernetes: every Argent pod carrying the standard prometheus.io/scrape annotations, in Argent's own namespace only |
| Direction | Prometheus pulls metrics from Argent; Alertmanager pushes notifications out to the receivers you choose |
| Grafana data sources | Prometheus (default), Loki and Tempo, provisioned from files at start, so the traces and logs sit beside the metrics |
| Ports | Prometheus 9090, Alertmanager 9093, Grafana 3000 (published on 13000 in the Compose stack) |
Dashboards and data sources are loaded through Grafana's file provisioning (Grafana docs: provisioning), so a restart brings back the same views.
Already run your own Prometheus? Argent's pods carry the standard scrape annotations, and the alert rule files and dashboard JSON ship as plain files in the Helm chart, ready to load into the stack you already have.
What you configure
Most installs change very little. These are the settings that exist.
- Leave it on, or turn it off.
observability.enabledis on by default in the Helm chart; on Compose, add the observability overlay. - Set retention and storage. One
observability.retentionvalue (default168h, seven days) applies to metrics, logs and traces. Prometheus gets 20 GiB of storage by default; each component has its own resource limits. - Sign in to Grafana. On Kubernetes, anonymous access is off. The chart generates a random admin password into the Secret
argent-grafanaand keeps it across upgrades. - Choose where alerts go. Add Slack, PagerDuty, email or a webhook as Alertmanager receivers (Prometheus docs: receiver configuration). On Compose, write the Slack URL or PagerDuty routing key into a file under
deployments/alertmanager/secrets/(git-ignored, mounted read-only), enable the matching block and recreate Alertmanager. A receiver pointing at a missing secret file refuses to start rather than failing silently. On Kubernetes, add your receivers to the alertmanager.yml entry of the chart's argent-observability ConfigMap; until you do, alerts are routed but not sent anywhere. - Prove it. Run the smoke test and watch the synthetic alert arrive.
Two kinds of alert
This stack answers "is Argent healthy?". Argent's own notifications answer "does someone need to act?".
- Stack alerts go to your operations or SRE team: a control failing closed, the audit record falling behind, a budget counter lost.
- In-product notifications go to the people who own the decision: a budget near its cap, a skill waiting for approval, an audit chain break. Each channel is set up in the Argent console: Slack, PagerDuty, email or a signed webhook.
The Argent console has its own health view too. Operations shows every node and gate at a glance, with exceptions first. Grafana is where your team looks at the same signals over time, next to the rest of your infrastructure.
Where this fits
Pages that go deeper.
- Same stack, other half: traces and logs across the gateway, nodes and control plane on the OpenTelemetry Collector, Tempo and Loki page.
- How it is installed: Docker Compose or Kubernetes with Helm.
- Capabilities: Session Replay, Agent Behavior Watch, Watch Before You Enforce, Air-Gapped Install and all capabilities.
- Goals: See all AI in your company, audit and accountability, sovereign AI and all goals.
- Use cases: securing AI coding agents, governing customer-facing AI assistants and AI in regulated and air-gapped environments.
- Where it runs: air-gapped on-premises, private cloud or your public cloud VPC. Compare them on deployment options.
- Industries: Financial Services, Retail, Healthcare, Public Sector, Insurance and all industries.
- More integrations: every system Argent connects to, grouped by type, on the integrations page.
Want this running on your own use cases?
All downloads