How To: Datadog Monitor Integration with Heii On-Call

While Heii On-Call can provide a great first line basic alerting and uptime monitoring, many of our users wish to integrate Heii On-Call with other APM solutions that can provide deeper insight and more nuanced analytics for complex applications. Heii On-Call can integrate with most of them using our simple, flexible HTTP-based API. This guide covers how to integrate Heii On-Call with Datadog using their provided webhooks integration so you can alert the correct responsible individual directly from the Datadog monitors you already have set up.

Set up Heii On-Call

First, make sure you have a Heii On-Call API key you wish to use for Datadog. To find your API keys go to your organization's dashboard and click the API Keys link. We recommend using a new key for every service that will be connecting to our API, such as one key for Datadog and another key for your own application code.

You will need the API Key later to enter it in Datadog.

Now create a new Trigger in Heii On-Call of type Datadog. This is a special type of trigger that will parse part of the payload sent from the webhook.

Once the trigger is created, note the trigger URL Heii On-Call gives you, as you will need to enter it in Datadog.

Set up Webhook on Datadog

Now head over to Datadog and open the webhooks integration tile.

Create a new variable named HEII_ON_CALL_API_KEY and enter the API Key you made earlier as the value, then hit save. We recommend clicking the “hide from view” checkbox.

Now hit the + New button to create a new webhook. Give it a name, in this case we are using heii_on_call_general_trigger. Paste the Heii On-Call trigger URL into the URL field.

For the payload, Heii On-Call only requires that the "alert_transition": "$ALERT_TRANSITION" key/value pair be present. This is the value we use to know if the alert is triggering or has been resolved. All the other fields are simply stored in and displayed on the trigger when you get an alert. We recommend adding a minimal set of additional fields that includes the $LINK variable so the responsible individual can quickly jump to the right place in Datadog. Here is an example payload you can use:

{
    "alert_transition": "$ALERT_TRANSITION",
    "body": "$EVENT_MSG",
    "title": "$EVENT_TITLE",
    "link": "$LINK"
}

Click the “custom headers” checkbox and add the following snippet so that Datadog adds the Authorization header to the request.

{
     "Authorization": "Bearer $HEII_ON_CALL_API_KEY"
}

The resulting webhook should look like this

Use the Webhook

In order to trigger the Heii On-Call trigger from Datadog, you simply have to include @webhook-WEBHOOK_NAME anywhere inside the message of any monitor in Datadog. Create a new monitor, or edit an existing monitor. Scroll down to the “Notify your team” section, and simply add @webhook-heii_on_call_general_trigger (or whatever your trigger is called).

Since we only have one rotation in our organization, it suffices to have only one webhook set up that we use for every Datadog monitor. When a monitor becomes triggered, the currently on call individual will receive the alert. (If you have more than one on-call rotation in Heii On-Call, simply create multiple Datadog webhooks with appropriate names and use a different Heii On-Call trigger for each.)

If you wish you can scroll to the bottom of the page and click Test Notifications. This allows you to send the Alert and Alert Recovery webhooks to Heii On-Call. Note: this will alert the currently on-call individual, so make sure you either silence the trigger beforehand on Heii On-Call or set yourself as the currently on-call individual while testing.

Done

That’s it! Datadog will send a webhook to our API when your monitor fires and Heii On-Call will do the rest based on the rotations and escalation policies you have set up.

Happy Monitoring!