Datadog Integration

Datadog Integration

Overview

The Datadog integration connects your Datadog monitoring environment with Overwatch through webhook notifications. When Datadog monitors detect issues, they send alert payloads to Overwatch, which automatically parses Datadog’s tag-based metadata, creates incidents, and begins AI-powered resolution analysis.

Overwatch extracts infrastructure context from Datadog’s key:value tag format, giving the AI layer full visibility into which services, clusters, and environments are affected.

Prerequisites

• An active Datadog account with monitor creation permissions
• An Overwatch organization with an active subscription
• The integration ID from your Overwatch Settings page (generated when you create the integration)
• (Optional) A webhook secret for HMAC signature validation

Setup Steps

1. In Overwatch, navigate to Settings > Integrations > New Integration and select Datadog.
2. Copy the generated Integration ID and the full webhook URL:

   https://your-overwatch-instance/api/v1/webhooks/datadog/{integration_id}

3. In Datadog, go to Monitors > Settings > Integrations > Webhooks.
4. Click New Webhook and configure:
   • Name: overwatch-alerts
   • URL: Paste the webhook URL from step 2
   • Payload: Use the default JSON payload, or customize it to include additional fields
   • Custom Headers: If you configured a webhook secret in Overwatch, add:

     X-Datadog-Signature: $WEBHOOK_SIGNATURE

5. Save the webhook configuration.
6. Assign the webhook to one or more Datadog monitors by editing a monitor and adding @webhook-overwatch-alerts to the notification section.
7. In Overwatch, click Test Connection to verify the integration is receiving data.

Tip: You can assign the webhook to all monitors using Datadog’s monitor notification template variables, or selectively add it to critical monitors only.

Alert Format

Overwatch extracts the following fields from Datadog alert payloads:

Field	Source	Description
monitor_id	monitor.id	Datadog monitor identifier
service_name	Tags: service, service_name, kube_service, ecs_service	Affected service
cluster_name	Tags: cluster, cluster_name, kube_cluster_name, ecs_cluster	Infrastructure cluster
namespace	Tags: namespace, kube_namespace	Kubernetes namespace
pod_name	Tags: pod_name, kube_pod	Kubernetes pod
host_name	hostname, host, or tags	Affected host
environment	Tags: env, environment	Deployment environment
aws_region	Tags: region, aws_region	AWS region
aws_account_id	Tags: aws_account, account_id	AWS account

Datadog uses key:value tag pairs as its primary metadata format. Overwatch parses all tags automatically and stores any custom tags as additional context for AI analysis.

Note: Severity is mapped from the Datadog priority field. critical maps to CRITICAL, high to HIGH, normal to MEDIUM, and low to LOW.

Chrome Extension Detection

The Overwatch Chrome extension automatically detects when you are browsing Datadog and surfaces relevant context. The extension activates on:

• app.datadoghq.com
• eu.app.datadoghq.com

When an active incident matches the monitor or service you are viewing in Datadog, the extension displays resolution suggestions and related incident history in the sidebar panel.

Troubleshooting

Webhook not receiving alerts

• Verify the webhook URL matches the format https://your-overwatch-instance/api/v1/webhooks/datadog/{integration_id} exactly.
• Confirm the integration ID in the URL matches what is shown on the Overwatch Settings page.
• Check that the Datadog monitor has @webhook-overwatch-alerts in its notification body.

Signature validation failures

• Ensure the X-Datadog-Signature header value matches the shared secret configured in both Datadog and Overwatch.
• Signature validation is optional. If you are not using it, remove the header from the Datadog webhook configuration.

Missing context in incidents

• Overwatch relies on Datadog tags for context extraction. Add tags like service, env, cluster, and namespace to your monitors and infrastructure.
• Verify tags are using the key:value format (for example, service:payments-api).

Duplicate incidents

• Overwatch deduplicates based on the monitor ID and aggregation key. If you see duplicates, check that monitors are not sending multiple webhook notifications for the same alert group.