Incident Response Agent
Detects, diagnoses and coordinates bounded remediation for production incidents.
Uses service telemetry to connect a spike to a probable cause and follow the approved runbook for restart, scale-out, rollback or failover. It records the incident timeline, prepares the post-mortem and, for a severe event, convenes the owning service roles with a current situation report.
Authority
Act within policy
Team role
Coordinates the work
Handoffs
Named collaborators
The role
What it owns and where its authority ends
Desk
Platform & Site Reliability (SRE)
Desk workflow
Signal, then alert, then agentic triage, then diagnosis, remediation and post-mortem. Known incidents run runbook autopilot (restart, scale, roll back); novel outages route to a responder agent that reasons them end to end.
Collaboration
Coordinates specialist contributions
Decision boundary
Acts only within an explicit policy, permission and escalation boundary.
Systems and capabilities involved
Observability stack (metrics/logs/traces)
Prometheus/Grafana/OTel
Kubernetes control plane
restart, scale, drain, roll back
Diagnostic sandbox
ad-hoc queries, log parsing
Runbook + post-mortem store
Sev-1 command bridge (service-owner agents)
Handoffs
What this role gives and receives
Capabilities offered
The handoffs name the next owner or specialist and the work that moves between them.
Handoff to
Handoff to
Receives from
External handoff
Operations (service-owner agents) for the Sev-1 command bridge
Context
What the role needs to do the work
- Current work
- The live incident: alerts firing, hypotheses, actions taken, timeline.
- Prior interactions
- Past incidents with the same signature and what actually fixed them.
- Policies and reference
- Service topology, dependency graph, SLOs and the runbook library.
- Working method
- Remediation playbooks refined from what worked in prior incidents.
Illustrative workflow
How the work moves
Starting point
Worst-case response times on the payments platform triple; error rate climbing.
- 01
Correlate the spike to a deploy 4 minutes prior and a connection-pool exhaustion log pattern.
- 02
Match the signature to two prior incidents; both fixed by rolling back.
- 03
Roll back the canary, confirm latency recovers, hold the full fleet.
- 04
Write the incident timeline and draft the post-mortem.
Result
Latency restored via rollback; a completed post-mortem and a flagged regression handed to the owning service agent. No pager fired.
Checks and boundaries
What must be tested or reviewed
- 01Hard guardrail: blast-radius cap, cannot restart/roll back beyond N services without an oversight-agent gate.
- 02Data-loss-risk actions require a second responder agent to re-derive the action as a check before commit.
- 03Game-day red-teaming: scripted chaos injections it must triage correctly.
- 04Every action fully traced; immutable incident audit log.
Human authority
Acts only within an explicit policy, permission and escalation boundary.
Keep exploring