A case study in automating operational work end to end — and the parts that turned out to be hard.
This case study summarizes work published on the LaunchDarkly engineering blog: Building a Self-Driving Ops Triage Loop.
Most automation projects stop at the interesting part. A model reads an alert, produces a plausible explanation, and the demonstration ends there. What happens next — deciding whether the explanation is trustworthy, whether the fix is safe, who is accountable, and what the system does when it is not sure — is where the actual engineering lives.
This project went the rest of the way.
At LaunchDarkly, the Foundation team received a steady volume of operational alerts from Datadog and Spinnaker. Each one interrupted an engineer to do the same mechanical sequence: read the alert, pull the logs, decide whether it was real, decide what to do about it, and either fix it or close it. Some of those alerts paged on-call unnecessarily. Most of the work required no judgment at all — but the few that did required a great deal of it.
The goal was not to replace the engineer. It was to automate the mechanical portion completely, and to route everything genuinely ambiguous to a human, on purpose, with the evidence already assembled.
What Was Built
Three specialized agents, each with a narrow job, handing work to each other through Jira.
1Alert fires in Datadog or Spinnaker2Triage agent investigates logs and execution output3Triage agent opens a Jira ticket with a remediation plan4Validator agent independently reviews the evidence and the plan5Implementation agent executes the approved plan6Pull request opened, on-call engineer tagged for review
The triage agent reads the alert, investigates across logs and execution output, decides whether the alert is actionable, and writes a remediation plan. It posts its reasoning back into the original alert thread and opens a Jira ticket only when its confidence clears a threshold.
The validator agent reviews that conclusion independently. It is explicitly not a rubber stamp — it rejects or rewrites plans it does not agree with, and nothing reaches code without passing through it.
The implementation agent executes approved plans, opens a scoped pull request, and requests review from whoever is actually on-call, resolved live through incident.io.
The result routes an incident from alert to pull request without routine human intervention, while the cases that genuinely need a person still reach one.
The Design Decisions That Made It Work
The agents were the easy part. The system worked because of a handful of unglamorous decisions.
State lives in one system, not in the seams between tools
The first version used Slack emoji reactions to hand work between agents. It failed. Reaction triggers did not fire consistently, there was no audit trail, and when a handoff silently dropped there was nothing to debug and nothing to retry from.
Replacing that with Jira tickets — where each status transition triggers the next agent — turned an invisible failure into an inspectable one. Every handoff became a durable record with a retry path.
The general lesson applies well beyond this project: splitting orchestration across multiple tools multiplies fragile trigger points. Each tool has its own triggering model and handoff semantics, and none of them is a superset of the others. Consolidating state into one system removed an entire category of failure.
Deduplication is a day-one requirement, not a later refinement
A human triaging alerts naturally recognizes that they are looking at the same failure for the fourth time. An automated system does not, and it runs at machine frequency.
The fix was a deterministic deduplication key built from stable failure attributes — source, service, environment, monitor or pipeline name, and a normalized error signature with the volatile parts stripped out — checked at three separate points: during triage, again in the validator to catch races, and once more before opening a pull request.
Terminal states must stay semantically distinct
"Won't Fix," "Done," "Duplicate," and "Waiting" mean genuinely different things, and collapsing any two of them breaks downstream logic. A ticket a human deliberately rejected must not suppress a future real occurrence the way a merged fix legitimately does. Every transition carries a comment explaining the reasoning, so the record documents decisions rather than just status.
Match the model to the cost of being wrong
Triage and validation use heavier reasoning models, because those steps make judgment calls where errors are expensive. Implementation uses a cheaper, faster model, because it executes instructions that have already been scoped and approved.
This is a reusable pattern: spend reasoning capacity where ambiguity gets resolved, and spend as little as possible carrying out a concrete plan.
Guardrails are structural, not advisory
The validator gate stops an overconfident triage conclusion from becoming code. A repository scope boundary stops the implementation agent from touching repos outside its configuration — it fails by design rather than guessing. Ambiguous, unsafe, or repeatedly recurring fixes route to a "Waiting" state and escalate to a person.
None of these are instructions in a prompt asking the model to be careful. They are constraints in the system that hold regardless of what the model concludes.
What This Demonstrates
This is the shape of work that turns an AI idea into something a team depends on daily:
- Automating a complete workflow, from the triggering event through to a reviewable artifact, rather than a single assistive step.
- Multi-agent orchestration with real separation of duties — investigation, independent review, and execution held apart deliberately, so no single model's confidence is load-bearing.
- Integrating the systems a business already runs on — Datadog, Spinnaker, Jira, GitHub, incident.io, Slack — including the custom MCP connections and authorization plumbing that cloud automation requires and that local development quietly hides.
- Designing the failure paths first: deduplication, distinct terminal states, scope boundaries, confidence thresholds, and a defined escalation route for everything the system should not decide on its own.
- Earning autonomy in stages, with a human review gate on every change that reaches production code.
- Controlling operating cost by stratifying models against the actual difficulty of each task.
Where the Same Approach Applies
An ops triage loop is a specific instance of a general pattern: a high-volume, event-triggered workflow where most cases are mechanical, a minority require judgment, and the cost of a silent mistake is real.
That pattern is everywhere in a business — inbound support triage, invoice and document intake, compliance review queues, data quality exceptions, lead qualification, order and claims processing. The mechanics change; the requirements do not. You still need durable state, deduplication, distinct outcomes, explicit scope limits, a defined escalation path, and review that a human can perform quickly and meaningfully.
The most useful takeaway from this project is also the least exciting one:
The hard part was the handoffs, not the agents.
Reasoning quality mattered far less than the orchestration between steps. Teams that discover this early build systems that survive contact with production. Teams that discover it late build convincing demonstrations.
Read the Full Write-Up
The complete technical account — including the failed approaches, the deduplication key construction, and the open problems still being worked on — is published on the LaunchDarkly engineering blog:
Building a Self-Driving Ops Triage Loop →
If you have a workflow with the same shape — high volume, event-triggered, mostly mechanical, with a minority of cases that genuinely need a person — a Workflow Audit is the right place to start.