Pierre Tessier [Sales Engineer|Honeycomb]:
We’re going to talk about Honeycomb Triggers and getting notified when queries meet specific thresholds. We’re going to go ahead and create a trigger inside of Honeycomb. Typically, we do this when we’re looking at a query. We’ll find something interesting in one of our visualizations, and we want to institutionalize that into a trigger to be notified should it happen again in the future.
In this case, our P95 duration visualization, we want to be notified anytime that goes over 800 milliseconds. To make this a trigger, you click on a three-dot button in the top navbar and then click on Make Trigger. Because our query contains multiple visualizations, we’re going to be asked which one we want to make the trigger out of. In this case, we have a P95 duration and count time series. We’re going to say “P95” and say “make trigger.”
From here we’re presented with a new trigger screen. We want to give this trigger a meaningful name. In this case, let’s call it Latency too high, and you want to give a description that makes sense and that people could do actions with. Typically, what runbook to go after, maybe just some links to some documentation internally. You want to provide some details so somebody knows what to do right away. Below just that we have the actual query definition for this trigger. We could further refine that if we will. We also get a visual of what the values would be and how often it would happen at execution points.
Next is a threshold. When should we be notified of this trigger? Well, we want to look at whenever the duration is greater than or equal to 800 milliseconds. This next part about frequency and duration, this has to do with event latency. Not all events come in immediately. There is some kind of delay to them. Typically, just a couple seconds, though this could be as long as 30 seconds or even over a minute. This is due to sampling strategies, data processing pipelines, things of that nature. So, to make sure we capture data that comes in late, we’re going to want to run our query to go back in time a little tiny bit. In this case, we’re going to say three minutes to make sure we capture all those steps.
And finally, we want to add a recipient of who’s going to be notified when this trigger goes off. We can notify by email, Slack channel. On this setup, we’ve got multiple PagerDuty queues and even some webhooks to go off to, run books, or other automation tasks that you may have behind the scenes. We’re going to go ahead and go with the Slack channel and we’re going to call it Ops Alerts and we add that.
The last step, Create Trigger. And I do this, present a list all my triggers that we have defined and a new one we’ve just created, Latency too high. We go ahead and click on this and see that all the details of both this trigger, including its original query. Maybe we want to modify its threshold, or even if you wanted to add recipients. Maybe you wanted to add a new kind of recipient. You could do that by going to the Integration Center.
Inside the Integration Center, which you could also get by going to your Team Settings and then clicking on the Integrations tab, you could find all of your integrations for trigger notifications as well as how you work with Slack in sharing collaboration within Honeycomb itself. This top part deals with all the Slack. This should be authorized by a user who has the appropriate permissions for your Slack team. Below we have all of the integrations configured in our environment, and we go ahead and add a new one if desired.
PagerDuty, if you want to set up a new queue in its key, or if you wanted to do a webhook to go kick off maybe some internal process or something of that nature. In this case here, you need to give us a webhook URL as well as a shared secret. We’re going to do a post on this webhook whenever the trigger is happening, and you’re going to be giving JSON body with a list of all the groups or all the pieces of data that are actually against that threshold.
If you see any typos in this text or have any questions, reach out to firstname.lastname@example.org.