Investigate Prometheus alerts with HolmesGPT and AlertManager
HolmesGPT, the open-source SRE agent in the CNCF Sandbox, connects to your AlertManager, pulls the active alerts and investigates each one against your cluster, then reports a likely root cause and next steps. You run it from the terminal with one command and can narrow it to one alert, a severity or a namespace.
The problem
When an alert fires, the on-call engineer has to open the alert, find the workload, read pod events and logs and check related metrics before they even know where the problem is. That first pass is repetitive, and it is the part of incident response that is easiest to hand to an agent that can read the cluster for you.
Steps
- Install the HolmesGPT CLI, for example with brew tap robusta-dev/homebrew-holmesgpt and brew install holmesgpt, or pipx install holmesgpt, and check it with holmes ask --help.
- Set an API key for your AI provider, for example ANTHROPIC_API_KEY or OPENAI_API_KEY, and choose a model with --model if you do not want the default.
- Forward your AlertManager service to your machine with kubectl port-forward svc/<your-alertmanager-service> 9093:9093.
- To try it safely, deploy the crashing demo pod from the robusta-dev/kubernetes-demos repository and post a test KubePodCrashLooping alert to AlertManager with curl, as the walkthrough shows.
- Run holmes investigate alertmanager --alertmanager-url http://localhost:9093 to investigate all active alerts, or add --alertmanager-alertname "KubePodCrashLooping" to focus on one alert.
- Narrow the run with --alertmanager-label, for example severity=critical for critical alerts only or namespace=production for one namespace.
- Connect more data sources, such as your logs, APM tools, databases or remote MCP servers, so investigations can look beyond the cluster.
From the official docs
kubectl port-forward svc/<Your-Alertmanager-Service> 9093:9093
holmes investigate alertmanager --alertmanager-url http://localhost:9093 --alertmanager-alertname "KubePodCrashLooping"
Results
- HolmesGPT fetches all active alerts from AlertManager by default and investigates each of them.
- For each investigation it returns the potential root cause and next steps.
- Label filters limit a run to the alerts you care about, such as critical alerts or one namespace.
As reported by the source (HolmesGPT docs (walkthrough)); AgentGid did not measure these figures.
This suits SRE and platform teams running Kubernetes with Prometheus and AlertManager who want a free, self-hosted first responder and are comfortable giving an agent read access to the cluster; scope its kubectl credentials to read-only. It suits teams without Kubernetes or Prometheus less, although HolmesGPT can also be asked about other systems. Per AgentGid's HolmesGPT page (pricing checked Oct 6, 2026), HolmesGPT is free and open source (Apache-2.0) and you bring your own model API key, so you pay only for the model calls; its docs note that model choice has a large impact on results. This recipe follows the project's own walkthrough, not a measured result, and the agent's conclusions still need a human check before anyone acts on them. Alternatives listed on AgentGid include K8sGPT, also free and open source, and hosted AI SRE products such as Cleric (Free + $700 a month) and Datadog Bits AI SRE (usage-based).
The agent used here
Similar use cases
How Eon unified multi-cloud incident tooling behind one MCP Gateway on AgentCore
Eon, a cloud data protection vendor, replaced per-tool credentials and local MCP servers with a single authenticated MCP gateway on Amazon Bedrock AgentCore. Developers and an aut…
As reported by AWS (guest post co-written with Eon): developer onboarding dropped from 2 days to 2 minutes.
Build a CI-health monitoring agent with the Claude Agent SDK and GitHub MCP
In this Anthropic cookbook notebook, an agent gets the official GitHub MCP server and is told to look over recent CI runs. It reports failing jobs, flaky patterns and recommended …
A status summary of recent CI runs and what triggered them.
Automate remediation after an AWS DevOps Agent investigation with Lambda Durable Functions
An AWS blog post shows how to pair AWS DevOps Agent, which only observes and reports, with a Lambda Durable Functions workflow that uses Amazon Bedrock to propose and apply fixes.…
In the demo, a function with a 3-second timeout was diagnosed, and Bedrock proposed raising it to 30 seconds.
