Integration Testing

Integration Testing

Complete guide to testing Overwatch integrations locally using Docker Compose, with support for Prometheus, Elasticsearch, Grafana, and more.

Testing Categories

Integrations fall into three testing categories:

Category 1: Local Docker

Fully local testing with Docker Compose

• Prometheus + Alertmanager
• Elasticsearch + Kibana
• Grafana
• SigNoz (separate setup)

Category 2: Cloud Free Tiers

Cloud services with free tiers

• Datadog (14-day trial)
• New Relic (100 GB/month free)
• PagerDuty (14-day trial)

Category 3: Webhook Tunneling

Cloud with local webhook testing

• Slack (ngrok/CloudFlare Tunnel)
• Microsoft Teams
• Custom webhooks

Quick Start

Start All Testing Services

# Start all testing services using Docker Compose profiles
docker compose --profile testing up -d

# Or start specific services
docker compose up -d prometheus alertmanager elasticsearch kibana grafana

# Check service status
docker compose ps

# View service logs
docker compose logs -f prometheus

Access Testing Services

Service	URL	Credentials
Prometheus	http://localhost:9090	None
Alertmanager	http://localhost:9093	None
Elasticsearch	http://localhost:9200	elastic / elastic_password_123
Kibana	http://localhost:5601	elastic / elastic_password_123
Grafana	http://localhost:3001	admin / admin123

Category 1: Local Docker Testing

Prometheus + Alertmanager

Status: ✅ Fully implemented in docker-compose.yml

1. Start Services

   docker compose up -d prometheus alertmanager

2. Access Prometheus UI

   open http://localhost:9090

3. Create Integration in Overwatch

   Navigate to: http://localhost:3000/dashboard/integrations

   • Click “Add Integration”
   • Select “Prometheus”
   • Name: Local Prometheus
   • Base URL: http://prometheus:9090
   • Alertmanager URL: http://alertmanager:9093
   • Auth Type: None
   • Click “Test Connection” then “Save”

4. Test PromQL Query

   curl http://localhost:9090/api/v1/query?query=up

5. Trigger Test Alert

   The configuration includes a test alert that fires immediately:

   testing/prometheus/alerts.yml

   - alert: TestAlertAlwaysFiring
     expr: vector(1)
     for: 0s
     labels:
       severity: warning
       environment: testing
     annotations:
       summary: "Test alert that always fires"

   Check http://localhost:9093 for alerts, and verify webhook delivery in Overwatch incidents.

Elasticsearch + Kibana

What to Test: Log analysis, API key authentication, integration connectivity

Watcher Licensing

Elasticsearch Watcher (alerting) requires a paid license (Gold/Platinum/Enterprise). Free tier testing focuses on log search and integration setup.

1. Start Services

   docker compose up -d elasticsearch kibana

2. Wait for Elasticsearch Health

   docker logs -f overwatch-elasticsearch
   # Wait for "started" message

3. Configure Kibana User (Required for Kibana 8.x)

   docker compose exec elasticsearch curl -X POST -u elastic:elastic_password_123 \
     "http://localhost:9200/_security/user/kibana_system/_password" \
     -H 'Content-Type: application/json' \
     -d '{"password": "kibana_password_123"}'
   
   # Restart Kibana to apply credentials
   docker compose restart kibana

4. Wait for Kibana (may take 1-2 minutes)

   docker logs -f overwatch-kibana
   # Look for: "Kibana is now available"

5. Create API Key for Overwatch

   curl -X POST "http://localhost:9200/_security/api_key" \
     -u elastic:elastic_password_123 \
     -H "Content-Type: application/json" \
     -d '{
     "name": "overwatch-integration",
     "role_descriptors": {
       "overwatch_role": {
         "cluster": ["monitor", "manage_watcher"],
         "indices": [
           {
             "names": ["logs-*", ".watcher-history-*"],
             "privileges": ["read", "view_index_metadata"]
           }
         ]
       }
     }
   }'

   Response (save the “encoded” value):

   {
     "id": "VuaCfGcBCdbkQm-e5aOx",
     "name": "overwatch-integration",
     "api_key": "ui2lp2axTNmsyakw9tvNnw",
     "encoded": "VnVhQ2ZHY0JDZGJrUW0tZTVhT3g6dWkybHAyYXhUTm1zeWFrdzl0dk5udw=="
   }

6. Create Integration in Overwatch

   Navigate to: http://localhost:3000/dashboard/integrations

   • Click “Add Integration”
   • Select “Elasticsearch”
   • Name: Local Elasticsearch
   • Elasticsearch URL: http://elasticsearch:9200
   • Authentication Type: API Key
   • API Key: [Paste the “encoded” value from Step 5]
   • Click “Test Connection” then “Save”

7. Index Test Logs

   curl -X POST "http://localhost:9200/logs-test/_doc" \
     -u elastic:elastic_password_123 \
     -H "Content-Type: application/json" \
     -d '{
       "timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%S.%3NZ)'",
       "level": "ERROR",
       "message": "Test error for Overwatch integration",
       "service": "test-service"
     }'

8. Access Kibana UI

   open http://localhost:5601
   # Login: elastic / elastic_password_123

Grafana

What to Test: Dashboard embedding, alert rules, data source queries

1. Start Grafana

   docker compose up -d grafana

2. Access Grafana UI

   open http://localhost:3001
   # Login: admin / admin123

3. Create API Key

   In Grafana UI:

   • Navigate to Configuration → API Keys
   • Click “Add API key”
   • Name: “Overwatch Integration”
   • Role: Admin
   • Click “Add”
   • Copy the generated key

4. Create Integration in Overwatch

   Navigate to: http://localhost:3000/dashboard/integrations

   • Click “Add Integration”
   • Select “Grafana”
   • Name: Local Grafana
   • Base URL: http://grafana:3000
   • API Key: [Paste key from Step 3]
   • Click “Test Connection” then “Save”

5. Create Test Dashboard and Alert

   Use Grafana UI to create test dashboards and configure alert rules.

SigNoz

What to Test: OpenTelemetry traces, metrics, logs, alert rules

Separate Setup Required

SigNoz provides official Docker Compose. Clone and run separately.

# Clone SigNoz repository
git clone https://github.com/SigNoz/signoz.git testing/signoz
cd testing/signoz/deploy/docker/clickhouse-setup

# Start SigNoz
docker compose up -d

# SigNoz UI: http://localhost:3301
# Backend API: http://localhost:8080

Category 2: Cloud Free Tiers

Datadog

Free Trial: 14-day full-feature trial

Setup

1. Sign up at https://www.datadoghq.com/
2. Get API key: Organization Settings → API Keys
3. Get Application key: Organization Settings → Application Keys
4. Select site (US1, US3, EU, etc.)

Create Integration

Navigate to: http://localhost:3000/dashboard/integrations

• Click “Add Integration”
• Select “Datadog”
• Name: Datadog Trial
• Base URL: https://api.datadoghq.com
• API Key: [Your DD_API_KEY]
• App Key: [Your DD_APP_KEY]
• Site: datadoghq.com
• Click “Test Connection” then “Save”

Install Agent

DD_API_KEY=YOUR_API_KEY \
DD_SITE="datadoghq.com" \
bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

Create monitors in Datadog UI to trigger alerts.

New Relic

Free Tier: 100 GB/month data ingest, 1 free user

Setup

1. Sign up at https://newrelic.com/signup
2. Get User API key: Account Settings → API Keys (NOT Ingest key)
3. Get Account ID from account settings

Create Integration

Navigate to: http://localhost:3000/dashboard/integrations

• Click “Add Integration”
• Select “New Relic”
• Name: New Relic Free
• Base URL: https://api.newrelic.com
• API Key: NRAK-XXXXX
• Account ID: [Your account ID]
• Region: US
• Click “Test Connection” then “Save”

Install APM Agent

Follow New Relic documentation to install APM agents: https://docs.newrelic.com/docs/apm/agents/

PagerDuty

Free Trial: 14-day trial with full features

1. Sign up at https://www.pagerduty.com/sign-up/
2. Create API key: Configuration → API Access → Create New API Key
3. Get email for API requests (required by PagerDuty)
4. Create integration in Overwatch:
   • Name: PagerDuty Trial
   • API Key: [Your PD_API_KEY]
   • From Email: your-email@example.com
5. Configure webhooks in PagerDuty:
   • Services → Select service → Integrations → Add
   • Webhook URL: https://your-overwatch.com/api/v1/webhooks/pagerduty/{integration_id}

Category 3: Webhook Tunneling

Local Webhook Testing with ngrok

For testing webhooks from cloud services locally:

1. Install ngrok

   # macOS
   brew install ngrok
   
   # Or download from https://ngrok.com/download

2. Start ngrok Tunnel

   ngrok http 8000
   
   # Your webhook URL becomes:
   # https://abcd1234.ngrok.io/api/v1/webhooks/{platform}/{integration_id}

3. Configure Webhook in External Service

   Use the ngrok URL in your external service configuration:

   • Datadog: Integrations → Webhooks → Add Webhook
   • PagerDuty: Services → Integrations → Add Webhook
   • Slack: Slash Commands → Request URL

4. Monitor Webhook Traffic

   Access ngrok web interface: http://localhost:4040

   • View all webhook requests
   • Inspect payloads
   • Replay requests for testing

Slack

Free Tier: Unlimited with 10,000 message history

Bot Token (Full Features)

1. Create Slack App at https://api.slack.com/apps
2. OAuth & Permissions → Add scopes:
   • chat:write, chat:write.public
   • channels:read, users:read
   • files:write, reactions:write
3. Install to Workspace → Copy Bot User OAuth Token (xoxb-…)
4. Setup ngrok tunnel:

   ngrok http 8000
   # Copy forwarding URL: https://abcd1234.ngrok.io

5. Create integration in Overwatch:
   • Name: Slack Local Test
   • Bot Token: xoxb-YOUR-BOT-TOKEN
   • Default Channel: #incidents
   • Auth Type: Bot Token
6. Configure slash commands:
   • Command: /overwatch
   • Request URL: https://abcd1234.ngrok.io/api/v1/webhooks/slack/{integration_id}/commands

Incoming Webhook (Simple)

1. Go to https://api.slack.com/apps
2. Create app → Incoming Webhooks → Activate
3. Add New Webhook → Select channel
4. Copy webhook URL
5. Create integration in Overwatch:
   • Name: Slack Webhook
   • Webhook URL: https://hooks.slack.com/services/T00/B00/XXXX
   • Default Channel: #incidents
   • Auth Type: Incoming Webhook
6. Test connection to send test notification

Testing Checklist

For Each Integration

1. Setup Phase

   • Start required Docker services (if local)
   • OR sign up for cloud service (if cloud)
   • Get API credentials/keys
   • Configure webhook URLs (if applicable)

2. Connection Testing

   • Create integration via Web UI
   • Use “Test Connection” button
   • Verify green checkmark or success message
   • Check integration appears in active list
   • Verify health status shows “Connected”

3. Data Ingestion Testing

   • Send test data to external service
   • Verify data appears in external UI
   • Trigger alert/incident in external service
   • Confirm webhook received in Overwatch
   • Verify incident created in Overwatch

4. API Operations Testing

   • Query metrics/logs via integration
   • Verify data transformation correctness
   • Test error handling (invalid queries)
   • Verify rate limit tracking

5. Health Monitoring Testing

   • Force API error (invalid credentials)
   • Verify failure recorded
   • Check success_rate calculation
   • Confirm auto-disable after 10 failures
   • Re-enable and verify recovery

6. Cleanup

   • Delete test integrations
   • Verify soft delete (data retained)
   • Check audit log entries
   • Stop Docker services (if local)

Troubleshooting

Common Issues

Webhook Not Received:

# Check Overwatch logs
docker compose logs -f backend | grep webhook

# Verify webhook URL is correct
echo "http://backend:8000/api/v1/webhooks/{type}/{integration_id}"

# Test webhook manually
curl -X POST http://localhost:8000/api/v1/webhooks/prometheus/{id} \
  -H "Content-Type: application/json" \
  -d '{"status": "firing", "alerts": [{"labels": {"alertname": "test"}}]}'

Connection Test Fails:

# Check network connectivity from backend container
docker exec overwatch-backend curl -v http://prometheus:9090/api/v1/status/config

# Verify credentials
docker exec overwatch-backend env | grep API

# Check integration status
curl http://localhost:8000/api/v1/integrations/{id}/health \
  -H "Authorization: Bearer $TOKEN"

Quota Still Blocks After Delete:

# Verify soft delete worked
docker compose exec postgres psql -U overwatch -c \
  "SELECT id, name, deleted_at FROM integrations WHERE organization_id = 'YOUR_ORG_ID';"

# Should show deleted_at timestamp for deleted integration

# If quota still blocks, restart backend
docker compose restart backend

Stop Testing Services

# Stop and remove testing services
docker compose --profile testing down

# Stop but keep data
docker compose stop prometheus alertmanager elasticsearch kibana grafana

# Remove volumes (WARNING: deletes all data)
docker compose --profile testing down -v

Quick Reference

Service Management

# Start all testing services
docker compose --profile testing up -d

# Start specific service
docker compose up -d prometheus alertmanager

# Check service status
docker compose ps

# View logs
docker compose logs -f prometheus

# Restart service
docker compose restart prometheus

# Stop testing services
docker compose --profile testing down

Access URLs

# Prometheus
open http://localhost:9090

# Alertmanager
open http://localhost:9093

# Elasticsearch (elastic/elastic_password_123)
open http://localhost:9200

# Kibana (elastic/elastic_password_123)
open http://localhost:5601

# Grafana (admin/admin123)
open http://localhost:3001

# Overwatch Web UI
open http://localhost:3000/dashboard/integrations

Next Steps

• Building Integrations - Integration development guide
• Incoming Webhooks - Webhook implementation patterns
• Webhooks Overview - Webhook architecture

Need Help?

For testing assistance, contact support@overwatch-observability.com.

---

Related Documentation:

• Admin Guide - Administrative functions
• API Authentication - API auth methods
• Integrations - Platform integrations