Why We Built Our Own Distributed Column Store

Building Honeycomb, we needed a low-latency, distributed, schemaless database. NBD for Facebook, but we faced multi-tenancy, cost to serve, and the limited resources of a startup. This talk describes Retriever, a custom-built database inspired by Scuba. Retriever ingests events from Kafka and chooses disk over memory, using an efficient column-oriented storage model. Hear interesting aspects of the implementation, and lessons learned from operating a hand-rolled database at production scale with paying customers.