Aviatrix CoPilot

CoPilot leverages the intelligence, advanced network, and security services delivered by Aviatrix’s multi-cloud network platform to provide enterprise cloud network operations teams both familiar day-two operational features such as packet capture, trace route and ping and new operational capabilities specifically built for multi-cloud network environments.

The following previous blog post provides more details:

Taking the left seat of your cloud deployment with Aviatrix CoPilot

The following previous posts go into details on how to deploy Aviatrix:

Deploying Aviatrix Controller and CoPilot on Azure

Deploying Aviatrix Controller and CoPilot on GCP behind Cloud Armor

“Terraform-ing” your way towards Secure Multi Cloud Networking with Aviatrix

Avitrix CoPilot Notifications is where alerts can be configured so that you can be notified about changes in your Aviatrix transit network. The alerts can be based on common telemetry data monitored in the network.

The full list of metrics is listed at:

https://docs.aviatrix.com/HowTos/copilot_reference_guide.html#metrics-used-for-triggering-notification

When configuring alerts, you can choose a notification channel of email or Webhook destinations.

Webhooks

CoPilot supports Webhook alerts. Webhooks are user-defined HTTP callbacks. They are usually triggered by some event, such as pushing code to a repository or a comment being posted to a blog. When that event occurs, the source site makes an HTTP request to the URL configured for the webhook. CoPilot alerts expose the following variables (objects):

Name	Type
alert	object
alert.closed	bool
alert.metric	string
alert.name	string
alert.threshold	int
alert.unit	string
event	object
event.exceededOrDropped	string
event.matchingHosts.[(n>-1)]	string
event.matchingHosts.[0]	string
event.message	string
event.newlyAffectedHosts	array
event.newlyAffectedHosts.[(n>=0)]	string
event.newlyAffectedHosts.[0]	string
event.receiveSeparateAlert	bool
event.recoveredHosts	array
event.recoveredHosts.[(n>-0)]	string
event.recoveredHosts.[0]	string
event.timestamp	string
extra	object
extra.hosts	object
extra.hosts.HOSTNAME	object
extra.hosts.HOSTNAME.userTags	object
extra.hosts.HOSTNAME.userTags.TAG	string
webhook.name	string
webhook.secret	string
webhook.url	string

You can customize webhooks using the Handlebars templating language. Handlebars uses a template and an input object to generate HTML or other text formats. Handlebars templates look like regular text with embedded Handlebars expressions. The template is located under the Tag/Labels on the recipients configuration:

A handlebars expression is a {{, some contents, followed by a }}. When the template is executed, these expressions are replaced with values from an input object. More info on Handlebar can be found at https://handlebarsjs.com/

Configuration

I’m going to initially use https://webhook.site to check if my configuration is working properly:

Webhooks are configured under CoPilot -> Monitor -> Notifications -> Recipients:

We can use the built in “test” at the end of the Recipient page configuration:

Back to webhook.site we can see the test details:

Customizing the Handlebar Template

I’m going to use the example below:

{
  "status": "{{#if alert.closed}}ok{{else}}critical{{/if}}",
  "check": {{alert.name}},
  "copilotstatus": {{alert.status}},
  "host": {{#if event.receiveSeparateAlert}}
  {{#if event.newlyAffectedHosts}}
    {{event.newlyAffectedHosts.[0]}}
  {{else}}
    {{event.recoveredHosts.[0]}}
  {{/if}}
{{else}}
  {{#if event.newlyAffectedHosts}}
    {{event.newlyAffectedHosts}}
  {{else}}
    {{event.recoveredHosts}}
  {{/if}}
{{/if}},
  "alert_timestamp": "Received <<alert.metric>> at <<event.timestamp>>"
}

The preview window provides a quick mechanism to visualize the handlebar template:

webhook.site:

Once all the required recipients are created, we can create alerts and associate the webhooks to the alert configuration:

Testing

I shutdown a “few” AVX gws to trigger the gw_status alert:

Because I selected “Receive Separate Notification For Each Host” we see four requests associate to the “gw_status” down:

Creating Severity Levels

Alerts have as status open or closed but what most enterprises want is to know as well how severe is the incident. We can accomplish it creating webhooks with different handlebar templates:

• alerts are associated to the recipient indicating the severirty of the monitoring metric. Ex.: a gateway down is critical

• while CPU utilization equal or above 60% generates a Warning:

Thanks Reid for helping me understand CoPilot alerts and webhooks.

References

https://handlebarsjs.com/guide/#what-is-handlebars