Prometheus Integration

Prometheus / Alertmanager Integration

Overview

The Prometheus integration connects your Alertmanager instance with Overwatch through the standard Alertmanager webhook receiver. Alertmanager sends alerts in its native webhook v4 format, which Overwatch parses to extract labels, annotations, and infrastructure context. This provides the AI layer with the full set of Prometheus labels for root cause analysis.

Overwatch supports all standard Prometheus label conventions, including Kubernetes labels (namespace, pod, container), cloud provider labels (region, aws_account), and application labels (job, instance, service).

Prerequisites

• A Prometheus instance with Alertmanager configured
• An Overwatch organization with an active subscription
• The integration ID from your Overwatch Settings page
• SSH or configuration access to your alertmanager.yml file

Setup Steps

1. In Overwatch, navigate to Settings > Integrations > New Integration and select Prometheus.
2. Copy the generated Integration ID and the full webhook URL:

   https://your-overwatch-instance/api/v1/webhooks/prometheus/{integration_id}

3. Edit your Alertmanager configuration file (alertmanager.yml) to add Overwatch as a webhook receiver:

   receivers:
     - name: 'overwatch'
       webhook_configs:
         - url: 'https://your-overwatch-instance/api/v1/webhooks/prometheus/{integration_id}'
           send_resolved: true

4. Assign the receiver to a route in your Alertmanager routing tree:

   route:
     receiver: 'default'
     routes:
       - receiver: 'overwatch'
         continue: true
         matchers:
           - severity=~"critical|warning"

5. Reload Alertmanager to apply the configuration:

   curl -X POST http://localhost:9093/-/reload

6. In Overwatch, click Test Connection to verify the integration.

Tip: Use continue: true in the route configuration to send alerts to both Overwatch and your existing receivers simultaneously.

Alert Format

Overwatch extracts the following fields from Alertmanager webhook payloads:

Labels (primary alert identification):

Label	Description
alertname	Alert rule name, used as the alert identifier
job	Prometheus job name, often represents the service
instance	Target instance in hostname:port format
severity	Alert severity level
namespace	Kubernetes namespace
pod	Kubernetes pod name
container	Container name
service	Service name
deployment	Kubernetes deployment name
node	Kubernetes node name
env / environment	Deployment environment
region / aws_region	Cloud region
aws_account	AWS account identifier

Note: The instance label is parsed to extract the hostname. For example, payments-api:8080 yields a hostname of payments-api.

Annotations (human-readable context):

Annotation	Description
summary	Short alert description, used as incident title
description	Detailed alert description
runbook_url	Link to operational runbook
dashboard_url	Link to monitoring dashboard

Severity mapping: Overwatch maps Prometheus severity labels to incident severity levels:

Prometheus Label	Overwatch Severity
critical	CRITICAL
page	CRITICAL
warning	HIGH
ticket	MEDIUM
info	LOW

Chrome Extension Detection

The Overwatch Chrome extension activates on:

• localhost:9090 (Prometheus UI)
• localhost:9093 (Alertmanager UI)

When viewing an alert or expression in the Prometheus or Alertmanager UI that matches an active Overwatch incident, the extension panel displays resolution suggestions and incident context.

Troubleshooting

Webhook not receiving alerts

• Verify the Alertmanager configuration was reloaded after adding the webhook receiver.
• Check that the route matcher is not too restrictive. Try temporarily removing matchers to test delivery.
• Confirm the Overwatch endpoint is reachable from the Alertmanager host (check DNS and firewall rules).

Alerts not creating incidents

• Overwatch requires the alerts array in the webhook payload. Verify your Alertmanager version supports webhook v4 format (Alertmanager 0.20+).
• Check that alerts have a status of firing. Resolved alerts update existing incidents but do not create new ones.

Missing labels in incident context

• Overwatch extracts all labels from the first alert in the webhook payload. If labels are inconsistent across alerts in the same group, some context may be missing.
• Use group_by in your Alertmanager routes to control which labels are preserved in grouped alerts.

Duplicate incidents

• Overwatch deduplicates using the alert fingerprint. If fingerprints are not present, it generates a unique ID from alertname, instance, and job. Ensure these labels are consistent across alert firings.

Resolved alerts not updating incidents

• Set send_resolved: true in the webhook configuration. Without this setting, Alertmanager does not send resolution notifications.