Operations   Observability   Dogfooding  

Using Honeycomb to remember to delete a feature flag

By Alyson  |   Last modified on February 5, 2020

Feature flags are great and serve us in so many ways. However, we do not love long-lived feature flags. They lead to more complicated code, and when we inevitably default them to be true for all our users, they lead to unused sections of code. In other words, tech debt. How do we stay on top of this? Find out how Honeycomb's trigger alerts proactively tell you to go ahead and clean up that feature flag tech debt!

The problem:

We wrote some new code in our Secure Tenancy service that is incompatible with the existing code in our web app. We still support the previous way that Secure Tenancy responds to the end point in question, and so we decided to feature flag the code that decides which end point to hit. The trouble is, we do not love long-lived feature flags. They lead to more complicated code, and when we inevitably default them to be true for all our users, they lead to unused sections of code. In other words, tech debt. Even worse, its tech debt that we inevitably forget to revisit. This thinking lead us to wonder, how can we help eliminate that tech debt, or at the very least, how do we remember to come back and revisit that tech debt? Do we create a ticket to check when all of our users have upgraded their instance of the service? Do we make a monthly calendar event to check and see if we can delete that lingering feature flag? How can we know when to do that work without the toil involved in checking every once in a while? Then we realized, we should just use Honeycomb to tell us.

Creating a derived column with regex

All of our integrations and customer-hosted services are instrumented to send us the user_agent string in the request header that tells us which version of a particular service or instrumentation our customers are using. As an event passes through an integration or service, it appends to the user_agent the string associated with what it passes through. For example, libhoney-py/1.9.0 beeline-python/2.9.2 via basenji/1.0.1. The part that we want is just the basenji/1.0.1 part, and really, just the version number of 1.0.1. Off I went to work on my regex.
It took me a few iterations, but I landed on something like this for my derived column. This pulled out just the version from that user_agent string. (You can read about using regex in derived columns here.)

REG_VALUE($request.header.user_agent, `.+basenji/([0-9/.a-z]+).*`)

However, that wasn’t going to be enough. I wanted to be able to say: Show me all of the teams whose version is LESS THAN 1.1. I know how to do that in our query builder, but you can’t use less than on a string field. So I needed to turn that version into a float. This lead me to the following derived column:

IF(
    CONTAINS($request.header.user_agent, "basenji"),
    FLOAT(
        REG_VALUE($request.header.user_agent, `.+basenji/([0-9/.]{3}).*`),
    )
)

This gave me a column I could filter where the value was less than 1.1

image.png

Creating a trigger to tell us when to delete the feature flag

From this query, I clicked on the ... icon in top right, and selected Make Trigger (Learn more about creating a trigger here)
image.png
I set up the trigger to alert the project channel when there was no longer any traffic less than 1.1. In the triggers UI, that would look like when the COUNT was <= 0

image.png

Conclusion

We can now sit back, relax and wait for the trigger to fire to tell us to go ahead and clean up that feature flag tech debt!

Many people think about using triggers or alerts only in dire cases where things have gone wrong. But, I’m here to remind you that you can use them to remind yourself about lots of different things about your system!

 

Related Posts

Dogfooding   Databases  

Virtualizing Our Storage Engine

Our storage engine, affectionately known as Retriever, has served us faithfully since the earliest days of Honeycomb. It’s a tool that writes data to disk...

Observability   News & Announcements  

Announcing Honeycomb Support Business Hours in Europe

Earlier this year, Honeycomb announced the launch of data residency in Europe. To meet the growing needs of our customers in the region, we are...

Observability   Metrics  

The Cost Crisis in Metrics Tooling

In my February 2024 piece The Cost Crisis in Observability Tooling, I explained why the cost of tools built atop the three pillars of metrics,...