How Secure Tenancy Keeps Your Secrets Secret

The best way to be sure that you keep a secret is not to know it in the first place. 

Managing secrets is a notoriously difficult engineering problem. Across our industry, secrets are stored in a bewildering variety of secure (and sometimes notoriously insecure) systems of varying complexity. Engineers are often trying to balance the least worst set of tradeoffs. At Honeycomb, we asked: What if we didn’t need to know your secrets to begin with? 

This post details how our patented Secure Tenancy proxy works. 

Sending data to Honeycomb: How it works

For most Honeycomb customers, when they send us telemetry, we process it with our high-performance columnar database, and then the customers can query it using a web application that is served from the honeycomb.io cloud. Imagine an organization with a couple of services in a Virtual Private Cloud (VPC) sending us data and querying it with a browser:

The Honeycomb software is hosted in our cloud and it’s under our immediate and direct control. It means that we can modify and release it many times a day and test in prod, and each time we do, all of our customers get the new stuff immediately. All of the customer telemetry data is hosted by Honeycomb and stored in our database.

For most customers, telemetry data is low risk. It doesn’t contain much that is personally identifiable, and most organizations are comfortable sending telemetry to Honeycomb in bulk. We follow industry best practices to protect and secure it, and all access to customer data by our support or engineering teams is logged and audited.

Enter Secure Tenancy

But that’s not always good enough. We also offer a patented feature to our enterprise customers called Secure Tenancy. It’s an application (a proxy) that sits inside a customer’s private network. Customers configure their applications to send telemetry data to the Secure Proxy. That proxy analyzes the telemetry, “masks” (encrypts) all of the strings in it, and forwards it to Honeycomb for normal processing (except with masked strings instead of meaningful ones). The system now looks like this:

For the users, there’s almost no difference in their experience. They send telemetry, and they can use Honeycomb to analyze it.

When these clients use Honeycomb from inside their VPC, the same browser software does the same actions it normally would, except that it uses the proxy to mask the requests. And when results are returned from the Honeycomb data center, they come back masked.

It’s gibberish, but it’s your gibberish.

If the results didn’t get unmasked, the Raw Data tab would look something like this:

Note that the field names (on the left) and the field values (those that are strings) are encrypted (except for Timestamp, which is special). Also, purely numeric values are not masked. This allows Honeycomb to do most of the same server-side query operations on masked data that it would on unmasked data.

Fortunately, the Honeycomb client software (which has been given access to the proxy) forwards the masked data to the proxy to unmask it, so that it can be displayed in its original form. Once unmasked, you’ll see a normal query result:

There are, of course, a few key limitations. For example, suppose the telemetry includes the request URL—something like http://example.com/user. It’s pretty common to write a Honeycomb query that includes a WHERE clause similar to request_url contains user. That’s not possible with Secure Tenancy, because the entire URL string is encrypted on the server, and we have no way to look inside it. On the other hand, a WHERE clause like “errorCode > 500” works just fine, because the proxy masks the string “errorCode” before sending the query to Honeycomb, so that the query engine can look up that column even though it’s masked.

It’s see-krit

The nice thing about this is that sensitive customer data never leaves their network, and the customers control the keys used for masking and unmasking. Since we don’t have the keys, that means there’s no possibility that Honeycomb can decrypt the strings stored in our database—even if we wanted to.

To us, it’s just a bunch of meaningless noise upon which we do our magic. What’s opaque to us is observability to you.

Secure Tenancy is available to Honeycomb Enterprise customers. It’s not a feature for everyone—but for those who need it, it’s vitally important. Interested in trying it out? Sign up for a free Honeycomb Enterprise trial.