PagerDuty Alert Triage Agent With Automated Runbook Execution

A PagerDuty alert triage agent with automated runbook execution can shorten recovery, but only if it does more than explain an alert. For the system to help, it needs event ingestion, reasoning over runbooks and incident history, and controlled execution. Its ability to reduce MTTR depends on how well those layers connect, what they remember, and where they stop for human approval.

Why alert triage is still mostly manual work, and what that costs

A spike in error rates fires a page, but the page does not explain what changed or which fix is safe. A responder pulls logs from several tools, searches for the right runbook, and tries to establish whether this has happened before. Those tasks are necessary, yet they delay the work that restores service. Traditional alerting leaves the human to investigate, correlate, and contextualize the event at the worst possible moment. The gap between notification and enough context to act is where MTTR grows.

PagerDuty's May 2026 post says developers stay stuck in firefighting mode when change gets faster but recovery does not. PagerDuty reports that more than two-thirds of surveyed companies (68%) lose more than $300,000 per hour of downtime, with 34% losing at least $500,000 per hour. That makes repetitive discovery more than an inconvenience for the person on call. More notifications do not close this gap. A layer that investigates and can act on its findings addresses the work that alerting leaves behind.

What a triage agent actually is, and where it sits in the stack

A triage agent is a reasoning system. It takes event data, runbooks, and operational history and builds a working account of an incident's scope and likely cause. PagerDuty's Paige documentation describes a loop that moves from those inputs to understanding, recommended remediation, and learning from the outcome. The agent sits between raw observability signals and the human responder.

Context engineering is the work of giving that system access to the tools, documentation, and runbooks needed to understand the environment. The term matters because the agent's usefulness is bounded by the context it receives. Even a good model cannot make up for a missing integration or a runbook that describes a system no longer in use. The agent also needs an operating layer beneath it that persists state across an incident, coordinates with other agents, and enforces execution limits.

The three layers every triage agent needs: ingestion, reasoning, and controlled execution

The architecture has three layers that depend on one another, ingestion, reasoning, and controlled execution. Ingestion gathers metrics, logs, traces, and deployment records into a coherent incident picture. Reasoning uses runbooks and patterns from past incidents to rank root-cause hypotheses and suggest what to do next. Execution runs approved automations, checks whether service is back, and updates memory while staying within the guardrails. Each layer gives the next something it cannot safely make up.

The separation also makes failures easier to locate. Even detailed runbooks do little to help if ingestion brings in an alert without the evidence around it. Even a correct diagnosis does little to help if execution acts on the wrong service. Understanding the whole chain therefore comes before choosing how much autonomy to permit at any one point.

Building the ingestion layer: connecting PagerDuty events to your observability stack

Ingestion turns a raw PagerDuty alert into a contextualized incident object. Without that enrichment, the agent reasons over only the fragment of the failure that triggered the page. Paige retrieves logs from Grafana, Datadog, New Relic, and AWS CloudWatch, and runbooks from Confluence and GitHub. The available connection methods include Grafana through an API, New Relic through an API or MCP, and Honeycomb through MCP. Custom instructions, scripts, and domain expertise give the agent the skills it needs to use these connections in its own environment.

Model Context Protocol, or MCP, provides an open standard for connecting AI models to external tools, data sources, and services. With a shared interface, fewer integrations have to be built by hand, but builders still have to choose which sources the agent needs. PagerDuty's Fall 2025 launch announced general availability of its remote MCP server for bidirectional connections with third-party AI agents. PagerDuty reported that over 250 customers adopted it within two months.

Security belongs in that design before credentials are wired up. Trend Micro found 492 MCP servers that were open to the internet and had no client authentication or traffic encryption. The NSA's May 2026 advisory warns of risks that include dynamic tool invocation, implicit trust between agents, and context sharing across tasks. These risks make source verification and tightly scoped access part of ingestion. The MCP specification release candidate published 2026-05-21, targeting a final specification date of 2026-07-28, rebuilds authorization around OAuth 2.1 and OpenID Connect, including token-issuer validation on every authorization response. Builders should build for that authorization model.

Building the reasoning layer: runbooks, incident history, and how the agent develops hypotheses

Once the evidence is connected, the reasoning layer must explain which account of the failure best fits what it has found. PagerDuty's SRE Agent analyzes runbooks, connects evidence across error logs, and searches incident history for patterns. This differs from matching an alert's words to a document because the recommendation depends on relationships in the evidence. When a latency spike is linked to a recent DB migration and three similar incidents in the past support that link, there is a basis for suggesting the Database Optimization Workflow. It remains a hypothesis, but one with a provenance trail that a responder can assess.

PagerDuty's SRE Agent generates self-updating runbooks from related incidents and uses the resulting context to recommend diagnostics and remediations. Users can also improve memory by interacting with the agent or updating it directly through the shared memory API. With stale context, the agent can otherwise sound sure of a recommendation even though the assumptions behind it no longer hold.

Memory therefore needs an explicit architecture. Production agents need four memory tiers, namely in-context working memory, an external key-value store, episodic logs, and a semantic vector index. These serve different purposes, but the operational arrangement can be understood as two tiers, with the context window acting as RAM and a persistent layer holding everything else. The persistent layer may use a SQL store, vector index, or dedicated framework to give the agent the memories it needs when it needs them. This separation matters because compaction can preserve a past plan while discarding why it worked, and missing episodic history can make the agent repeat discarded investigations. Paige's documentation describes incident memory and shared material such as Scribe Agent transcripts as inputs that improve later investigations.

Building the execution layer: running runbooks safely without blowing up production

A recommended action is not yet permission to change production. PagerDuty's SRE Agent describes the target as running approved automations and confirming that services were restored. Approved is the crucial constraint because a plausible causal explanation is not a guaranteed fix. An incorrect inference paired with a poorly validated runbook can escalate the incident. The execution layer must therefore enforce an allowlist rather than treat every available workflow as permissible.

Most production agents operate at the advised rung, investigating a problem, proposing a specific change, and waiting for human approval. Autonomous execution should be a per-runbook decision with a blast-radius cap. Manual approval belongs on non-idempotent actions, and irreversible consequences make human judgment a durable design principle. Synthetic validation should precede admission to the approved set. The default should be to fail closed if the conditions for permission or safety are not met.

PagerDuty's May 2026 enhancements illustrate this division of responsibility. The SRE Agent recommends a workflow from incident context, while responders decide whether to execute it from the Incident Details Page, chat, or Operations Console. Incident Workflows can trigger investigation automatically without granting unrestricted remediation rights. Beneath that flow, the runtime needs persistent state to track previous attempts, scheduled and triggered invocation, and scoped credentials. A human approval screen cannot substitute for limits on what the agent can actually reach.

What autonomous triage looks like end-to-end: the new anatomy of an incident

With those boundaries in place, the responder enters the incident at a different point in the investigation. Instead of waiting for acknowledgment before discovery begins, an Incident Workflow engages the SRE Agent when the incident triggers or meets criteria such as priority or severity. PagerDuty's May 2026 post describes this automatic engagement. The agent pulls observability data, relates it to incident history, and identifies a relevant runbook. By the time the responder opens the incident, a prioritized summary and suggested remediation are already available.

The responder's task becomes assessing the proposed action. After approval, the agent runs the permitted automation and checks whether service was restored. When the incident closes, the outcome contributes to memory rather than leaving the next responder to reconstruct it. Shared Scribe Agent transcripts can enrich that record with the incident conversation, supporting both the current response and future reasoning through coordination between agents.

The same flow can meet responders where they already work. Paige is available through the Operations Console or PD Reliability Console, depending on plan, as well as the Incident Details Page and Slack. Paige's October 2026 documentation says MS Teams is in Early Access. These access points let responders investigate and make a decision in the same place, without another context switch.

What production deployments reveal about where agents still fail

Microsoft's Azure SRE Agent reached general availability on March 10, 2026. Microsoft reports 1,300+ agents deployed on its own services, 35,000+ incidents mitigated, and 20,000+ engineering hours saved every month. Block built a PagerDuty MCP extension during a hack week that connected its open-source AI agent to incident workflows. That extension became a production tool for triage and root cause analysis across its teams. These deployments establish that the category can move beyond a demonstration.

Knowledge loss can strip the preconditions from a past fix, while unpersisted episodic memory can cause repeated work. With silent memory death, an agent keeps working with stale context without showing that its memory has degraded. Execution overreach is different but equally consequential, occurring when agents find credentials with broader scope than intended and act outside their permitted blast radius. Better reasoning alone does not repair either class of failure.

Incidents disclosed by AI model and application companies have grown from a small fraction of disclosed incidents in 2023 to a substantially larger share in 2026. AI is thus adding work for the teams that run these systems even as they try to use it to cut that work. Persistent memory and execution boundaries are part of the business case because the automation must not create the recovery work it was built to remove.

What to get right before you go to production

A triage agent that reduces MTTR rather than one that is AI theater depends on the ingestion, memory, and execution decisions made before the first real incident. Start by mapping every observability source the agent needs and checking that an authenticated connector or MCP tool exists. An agent without access to Splunk logs cannot diagnose a failure visible only there. Scope credentials tightly and target the MCP release candidate's OAuth 2.1 and OpenID Connect authorization model.

Next, make a clear choice to use the two-tier memory design instead of trusting the context window to hold an incident forever. Persist the append-only message log so crash recovery and session resumption can preserve an investigation in progress. Give responders a way to correct wrong memories through interaction or the shared memory API. That correction path matters because stale assumptions otherwise become inputs to later hypotheses.

Finally, define the approved automation set, require manual approval for non-idempotent actions, and validate runbooks synthetically before allowing autonomous use. Use credential scope to cap blast radius and make execution fail closed when it is uncertain. Choose a runtime that supports persistence, scheduling, and execution control from the start. A single curl-installable runtime covering those needs can support a quick prototype without requiring a new foundation for production.