Grafana Integration

Grafana Integration

Overview

The Grafana integration connects your Grafana alerting environment with Overwatch through contact point webhooks. Grafana sends alerts in a Prometheus-compatible Alertmanager format enriched with Grafana-specific data such as dashboard URLs, panel URLs, silence URLs, and alert rule details.

Overwatch goes beyond basic alert parsing by fetching additional context from the Grafana API, including alert rule definitions and dashboard metadata. This enrichment gives the AI layer a complete picture of the alerting condition, the query that triggered it, and the dashboards related to the affected service.

Prerequisites

• A Grafana instance (Grafana Cloud or self-hosted) with alerting and contact point configuration permissions
• An Overwatch organization with an active subscription
• The integration ID from your Overwatch Settings page
• (Optional) A Grafana API key for rule and dashboard enrichment
• (Optional) A webhook secret for HMAC signature validation

Setup Steps

1. In Overwatch, navigate to Settings > Integrations > New Integration and select Grafana.
2. Copy the generated Integration ID and the full webhook URL:

   https://your-overwatch-instance/api/v1/webhooks/grafana/{integration_id}

3. (Optional) Enter your Grafana API key in the Overwatch integration settings to enable rule and dashboard enrichment.
4. In Grafana, navigate to Alerting > Contact points.
5. Click New contact point and configure:
   • Name: Overwatch
   • Integration type: Webhook
   • URL: Paste the webhook URL from step 2
   • HTTP Method: POST
   • (Optional) Authorization Header: If you configured a webhook secret, add the appropriate header
6. Save the contact point.
7. Assign the contact point to a Notification policy to route alerts to Overwatch.
8. In Overwatch, click Test Connection to verify.

Tip: Assign the Overwatch contact point as a secondary contact in your notification policies so it receives alerts alongside your existing notification channels.

Alert Format

Overwatch extracts the following fields from Grafana webhook payloads:

Field	Source	Description
alertname	labels.alertname	Alert rule name
fingerprint	alert.fingerprint	Unique alert identifier for deduplication
severity	labels.severity	Alert severity label
status	alert.status	Alert state (firing, resolved)
labels	alert.labels	All alert labels (service, environment, instance)
annotations	alert.annotations	Summary, description, runbook URL
dashboard_url	alert.dashboardURL	Link to related Grafana dashboard
panel_url	alert.panelURL	Link to specific dashboard panel
silence_url	alert.silenceURL	Link to silence the alert
generator_url	alert.generatorURL	Link to the alert rule

Severity mapping: Overwatch maps Grafana severity labels to incident severity levels:

Grafana Label	Overwatch Severity
critical	CRITICAL
high	HIGH
warning	MEDIUM
medium	MEDIUM
low	LOW
info	LOW

API enrichment: When a Grafana API key is configured, Overwatch fetches:

• Alert rule details: Rule UID, title, condition expression, evaluation interval, folder
• Dashboard details: Dashboard UID, title, tags, and panel configuration

Chrome Extension Detection

The Overwatch Chrome extension activates on:

• grafana.net
• *.grafana.com (includes Grafana Cloud instances)

When viewing a dashboard or alert rule that matches an active Overwatch incident, the extension panel displays resolution suggestions and related incident context.

Troubleshooting

Webhook not receiving alerts

• Verify the contact point is assigned to an active notification policy.
• Check the Grafana Alerting > Contact points page for test delivery results.
• Ensure the webhook URL is reachable from your Grafana instance (check firewall rules for self-hosted deployments).

No API enrichment data

• Confirm the Grafana API key is entered in the Overwatch integration settings.
• Verify the API key has Viewer or higher permissions for alert rules and dashboards.
• For Grafana Cloud, use a Service Account token with the appropriate role.

Signature validation failures

• The X-Grafana-Alerting-Signature header is validated against the webhook secret configured in Overwatch. Ensure the secrets match.
• Overwatch also reads the X-Grafana-Timestamp header for replay attack prevention.

Duplicate incidents

• Overwatch deduplicates based on the Grafana alert fingerprint. If you see duplicate incidents, verify that the same alert is not being sent through multiple contact points.

Severity not detected

• Add a severity label to your Grafana alert rules. Without this label, Overwatch defaults to MEDIUM severity.