Blackbird: Storing Billions of Rows a Couple of Milliseconds Away

Would you use HBase to make billions of rows available for real-time lookup under 10 ms with 99% guarantee? We, at Rocket Fuel, do just that.

Date: Monday, May 05 2014

Description

Would you use HBase to make billions of rows available for real-time lookup under 10 ms with 99% guarantee? We, at Rocket Fuel, do just that. Blackbird, our system built on top of HBase, makes billions of rich user profiles available for AI based optimization under the tight latency requirements of real time auction. It relies on our novel collections API, a constrained yet useful append only model that is sympathetic to HBase internals and allows us to scale our writes easily while keeping strict read performance guarantees. In this talk, we describe the key abstractions Blackbird exposes, utilities we built over time to support our use cases and our hardware and software configuration (including HBase configs) that helps us achieve our strict latency guarantees. We also share the key challenges and lessons learned scaling the system ten fold in a short span of time and some common beginner mistakes that we made and fixed later that you should avoid.