Notifications and alerting

Argent pages through PagerDuty

When an AI budget hits its ceiling, Argent Strategic Intelligence raises a PagerDuty incident with the right severity, so the on-call person hears about it first.

PagerDuty logo
01 / NOTIFICATIONS AND ALERTING

What Argent sends to PagerDuty

Argent raises an incident when something needs the on-call person, and you choose which events count.

EventWhen it fires
Budget threshold budget.thresholdA spend budget crosses 50, 75, 90 or 100 percent of its ceiling. One alert per budget, per period, per level.
Agent budget alert budget.agent_alertOne agent's own budget crosses 70, 90 or 100 percent, when an operator turned that alert on for the agent. It fires once per crossing.
Skill approval requested skill.approval_requestedAn agent loaded a skill version nobody has reviewed, and, with auto-submit turned on, Argent sent it for approval. A skill is a packaged set of instructions an agent pulls in. Its text is never included.
Agent note mention agent.note_mentionSomeone @mentioned colleagues in a note on an agent. The alert names the agent, the author, the people mentioned and the owner, never the note text.

Leave the event list blank and the channel carries every event except the opt-in owner notice. Name any events and it carries only those. An event name Argent does not know is rejected when you save.

A PagerDuty channel can never take the budget owner notice, so a budget edit never pages anyone.

02 / NOTIFICATIONS AND ALERTING

How it connects

Argent sends a trigger event to PagerDuty Events API v2, the API PagerDuty provides for alerts from monitoring tools.

  1. In PagerDuty, add an Events API v2 integration to the service that should own the alerts and copy its integration key, which PagerDuty calls the routing key (PagerDuty: services and integrations).
  2. Store the routing key in the tenant vault.
  3. In the Argent console, open Integrations > Notifications, add PagerDuty and point it at that vault reference.
  4. Tick the live-send option and press Test. Argent sends one sample alert marked TEST - through the same path a real alert takes, so you see it arrive. Without the live-send option, Test checks the configuration only.

Wiring many tenants? argentctl integrations apply reads one JSON document, stores the secrets in the tenant vault, then creates, connects and tests each channel. It can be run again safely, has a --dry-run and never prints a secret. argentctl integrations hosts prints the outside hosts the document needs, so the firewall can be opened precisely.

03 / NOTIFICATIONS AND ALERTING

What you configure

SettingWhat goes in it
Routing keyA reference to the Events API v2 integration key in the tenant vault. Required: a channel with no credential is refused.
EventsBlank for every event a PagerDuty channel may carry, or the exact events that should page.

The endpoint is fixed. Argent always sends to PagerDuty's own Events API v2 address, events.pagerduty.com/v2/enqueue. Nothing in configuration or the environment can change it, so a routing key can never be aimed at another host.

04 / NOTIFICATIONS AND ALERTING

What lands in PagerDuty

Each event is one trigger with a plain summary, a severity from PagerDuty's own set, and a deduplication key.

  • The summary reads like Budget at 100%: Payments team, with names resolved by Argent before the alert leaves.
  • The deduplication key is the event id. PagerDuty applies events with the same key to the alert that is already open (PagerDuty: send an alert event), so a retry is folded into the alert that is already open instead of opening a second one.
What happenedSeverity
Budget at 100 percent or morecritical
Budget at 90 percenterror
Budget at 75 percentwarning
Budget below 75 percent, skill approval, note mentioninfo
05 / NOTIFICATIONS AND ALERTING

Delivery and guard rails

One alert path, built so a bad endpoint cannot hurt anyone else.

  • Retried: network errors, timeouts, server errors (5xx), too-many-requests (429) and request-timeout (408) replies. Each means the endpoint has not judged the alert yet. Up to three attempts, 10 seconds each, spaced out at random so a struggling endpoint is not hit by every retry at once.
  • Not retried: any other 4xx, such as 401, 403, 404 or 422. The endpoint has looked and refused, so a rotated secret or a deleted endpoint shows up at once instead of hiding behind retries.
  • Same event, same id: every retry of one alert carries the same event id, so a duplicate is detectable.
  • Never in the way: alerts go onto a bounded queue. One endpoint that hangs can never stall budget checks for any other team, and a dropped alert is counted in the argent_notify_deliveries_total metric that your Prometheus and Grafana stack can alert on.
  • Secrets are references. The routing key lives in the tenant vault. Argent's configuration holds only a reference to it. It is resolved when the channel connects and is never stored in configuration, written to a log or shown back in the console.
  • Outbound guard. Every connection is checked at dial time. Addresses on the Argent host itself and cloud provider metadata addresses are refused. A receiver on a private network is reachable only when your operator adds it to the deployment's outbound allowlist (ARGENT_CONNECTOR_EGRESS_ALLOWLIST). A tenant admin cannot open that by typing an internal address into a form.
06 / NOTIFICATIONS AND ALERTING

Where it fits

The alerts come from the parts of Argent that hold money, skills, owners and evidence.

Other alert channels: Slack, Email (SMTP), Signed webhook. Approvals that need a ticket go to ServiceNow or Jira Service Management.

Need a channel that is not listed? Request a connector from the Integrations catalog in the console. See every integration or book a demo.

Other alert channels

Argent can send the same alerts to each of these. A channel picks the events it carries.

  • Slack
  • Email (SMTP)
  • Signed webhook