Operations   Observability   Dogfooding  

Using Honeycomb to remember to delete a feature flag

By Alyson  |   Last modified on February 5, 2020

Feature flags are great and serve us in so many ways. However, we do not love long-lived feature flags. They lead to more complicated code, and when we inevitably default them to be true for all our users, they lead to unused sections of code. In other words, tech debt. How do we stay on top of this? Find out how Honeycomb's trigger alerts proactively tell you to go ahead and clean up that feature flag tech debt!

The problem:

We wrote some new code in our Secure Tenancy service that is incompatible with the existing code in our web app. We still support the previous way that Secure Tenancy responds to the end point in question, and so we decided to feature flag the code that decides which end point to hit. The trouble is, we do not love long-lived feature flags. They lead to more complicated code, and when we inevitably default them to be true for all our users, they lead to unused sections of code. In other words, tech debt. Even worse, its tech debt that we inevitably forget to revisit. This thinking lead us to wonder, how can we help eliminate that tech debt, or at the very least, how do we remember to come back and revisit that tech debt? Do we create a ticket to check when all of our users have upgraded their instance of the service? Do we make a monthly calendar event to check and see if we can delete that lingering feature flag? How can we know when to do that work without the toil involved in checking every once in a while? Then we realized, we should just use Honeycomb to tell us.

Creating a derived column with regex

All of our integrations and customer-hosted services are instrumented to send us the user_agent string in the request header that tells us which version of a particular service or instrumentation our customers are using. As an event passes through an integration or service, it appends to the user_agent the string associated with what it passes through. For example, libhoney-py/1.9.0 beeline-python/2.9.2 via basenji/1.0.1. The part that we want is just the basenji/1.0.1 part, and really, just the version number of 1.0.1. Off I went to work on my regex.
It took me a few iterations, but I landed on something like this for my derived column. This pulled out just the version from that user_agent string. (You can read about using regex in derived columns here.)

REG_VALUE($request.header.user_agent, `.+basenji/([0-9/.a-z]+).*`)

However, that wasn’t going to be enough. I wanted to be able to say: Show me all of the teams whose version is LESS THAN 1.1. I know how to do that in our query builder, but you can’t use less than on a string field. So I needed to turn that version into a float. This lead me to the following derived column:

IF(
    CONTAINS($request.header.user_agent, "basenji"),
    FLOAT(
        REG_VALUE($request.header.user_agent, `.+basenji/([0-9/.]{3}).*`),
    )
)

This gave me a column I could filter where the value was less than 1.1

image.png

Creating a trigger to tell us when to delete the feature flag

From this query, I clicked on the ... icon in top right, and selected Make Trigger (Learn more about creating a trigger here)
image.png
I set up the trigger to alert the project channel when there was no longer any traffic less than 1.1. In the triggers UI, that would look like when the COUNT was <= 0

image.png

Conclusion

We can now sit back, relax and wait for the trigger to fire to tell us to go ahead and clean up that feature flag tech debt!

Many people think about using triggers or alerts only in dire cases where things have gone wrong. But, I’m here to remind you that you can use them to remind yourself about lots of different things about your system!

 

Related Posts

Observability  

Frontend Debugging Is Bad and it Should Feel Bad

There’s a sentence that strikes fear into the heart of every frontend developer I've ever met: Users are reporting issues, and we don't know how...

Observability   News & Announcements  

Focused Labs & Honeycomb: Better Together

We're excited to unveil a new collaboration with Focused Labs, a leap forward in our shared commitment to advancing modern observability practices and enhancing the...

Observability   Monitoring  

APM From a Developer’s Perspective

In twenty years of software development, I did not have the privilege of being on call, of tending to my software in production. I’ve never...