Blog

Follow the Trail with Honeycomb and AWS CloudTrail

By | 2018.03.20
 

Querying and searching through CloudTrail trails and subsequent logs has been a pain point for many. The versions of each log’s format is often different and thus requires a new dashboard schema – or you might even have to write a dashboard schema in the first place!

Worry not! Honeycomb is proud to present – HoneyAWS CloudTrail integrations!

Following the Trail

AWS CloudTrail offers an excellent tool to track user activity and API usage. You can attach a trail to monitor S3 activity, who’s talking to your RDS instances, or even alert when a user goes over their allotted usage! It’s an incredibly useful tool that’s integral to security monitoring, alerting and compliance within AWS. However, CloudTrail parsing and aggregation of these data points can be a hassle to work with. Using the CloudTrail UI to query (and you can only do that per Trail) or pulling down the logs yourself to read over them is typically cumbersome and hard to grok.

Observation is key

Here at Honeycomb, we deeply believe that Observability is key to supporting and maintaining an infrastructure. When your Trails are hard to observe, that can be a problem and you lose some of the power that CloudTrail offers.

For example, one of your S3 buckets is growing rapidly and you’re not sure why. It’s expensive to store unnecessary things, and hard to find out who exactly or what exactly is writing to this bucket. It’d be nice if we could break down and find out what’s happening there! You have a Trail attached to S3… but reading that can be difficult.

Luckily, we can do that with the new CloudTrail integration!

If we BREAK DOWN by:

  • EventName – the AWS API request
  • AccountId – the Account ID making these requests – conveniently, this is something you can lookup in IAM roles to find out who’s ID it is
  • SourceIpAddress – the IP address where the API originated from – sometimes this is useful to find out if it’s an internal IP or something external

Then we can further narrow this down with Filter to specifically S3 requests using:

  • EventSource = [s3.amazonaws.com](http://s3.amazonaws.com/)
  • EventName = PutObject – after all, we care about the reason WHY our S3 bucket is growing!

graph showing spikes in writes

Huh, interesting – spikes in writes are happening every 10 minutes. Let’s look at the break down.

table showing breakdown by IP

From here, we can see that the PutObject requests are originating from internal IP addresses – that’s awesome! Now to look up what that AccountId is associated with and we can see what erroneous application might be sending too many objects to S3.

Honeycomb specializes in handling and displaying high cardinality data that can be almost impossible to parse and find quickly and efficiently. With Honeycomb, it’s very easy to find out what is going on within your CloudTrail trails!

Check it out Today

You can start using the new integration by following the documentation here for the new CloudTrail integration. This integration is incredibly helpful all the way down the scale – from running a small collective of Trails to hundreds of them!

We hope you enjoy, and as always, we’d love it if you signed up for a free trial to learn how to scan millions of data points in seconds to solve your problems.