HBase at Hadoop World NYC

HBase, Hadoop World NYC Ryan Rawson, Stumbleupon.com, su.pr Jonathan Gray, Streamy.com A presentation in 2 parts Part 1 About Me • Ryan Rawson • Senior Software Developer @ Stumbleupon • HBase committer, core contributor Stumbleupon • Uses HBase in production • Behind features of our su.pr service • More later Adventures with MySQL • Scaling MySQL hard, Oracle expensive (and hard) • Machine cost goes up faster speed • Turn off all relational features to scale • Turn off secondary (!) indexes too! (!!) MySQL problems cont. • Tables can be a problem at sizes as low as 500GB • Hard to read data quickly at these sizes • Future doesn’t look so bright as we contemplate 10x sizes • MySQL master becomes a problem... Limitations of masters • What if your write speed is greater than a single machine? • All slaves must have same write capacity as master (can’t cheap out on slaves) • Single point of failure, no easy failover • Can (sort of) solve this with sharding... Sharding Sharding problems • Requires either a hashing function or mapping table to determine shard • Data access code becomes complex • What if shard sizes become too large... Resharding! What about schema changes? • What about schema changes or migrations? • MySQL not your friend here • Only gets harder with more data HBase to the rescue • Clustered, commodity(ish) hardware • Mostly schema-less • Dynamic distribution • Spreads writes out over the cluster What is HBase? • HBase is an open-source distributed • Part of the Hadoop ecosystem • Layers on HDFS for storage • Native connections to map reduce database, inspired by Google’s bigtable HBase storage model • Column-oriented database • Column name is arbitrary data, can have large, variable, number of columns per row • Rows stored in sorted order • Can random read and write Tables • Table is split into roughly equal sized “regions” • Each region is a contiguous range of keys, from [start, to end) • Regions split as they grow, thus dynamically adjusting to your data set Server architecture • Similar to HDFS: • Master = Namenode (ish) • Regionserver = Datanode (ish) • Often run these alongside each other! Server Architecture 2 • But not quite the same, HBase stores state in HDFS • HDFS provides robust data storage across machines, insulating against failure • Master and Regionserver fairly stateless and machine independent Region assignment • Each region from every table is assigned to a Regionserver • The master is responsible for assignment and noticing if (when!) regionservers go down Master Duties • When machines fail, move regions from affected machines to others • When regions split, move regions to balance cluster • Could move regions to respond to load • Can run multiple backup masters What Master does NOT do • Does not handle any write requests (not a DB master!) • Does not handle location finding requests • Not involved in the read/write path! • Generally does very little most the time Distributed coordination • To manage master election and server availability we use ZooKeeper • Set up as a cluster, provides distributed coordination primitives • An excellent tool for building cluster management systems Scaling HBase • Add more machines to scale • Base model (bigtable) scales past 1000TB • No inherent reason why HBase couldn’t What to store in HBase? • Maybe not your raw log data... • ... but the results of processing it with Hadoop! • By storing the refined version in HBase, can keep up with huge data demands and serve to your website HBase & Hadoop • Provides a real time, structured storage layer that integrates on your existing Hadoop clusters • Provides “out of the box” hookups to map-reduce. • Uses the same loved (or hated) management model as Hadoop HBase @ Stumbleupon & HBase • Started investigating the field in Jan ’09 • Looked at 3 top (at the time) choices: • Cassandra • Hypertable • HBase cassandra didnt work, didnt like data model - hypertable fast but community and project viability (no major users beyond zvents) - hbase local and good community Stumbleupon & HBase • Picked HBase: • Community • Features • Map-reduce, cascading, etc • Now highly involved and invested su.pr marketing • “Su.pr is the only URL shortener that also helps your content get discovered! Every Su.pr URL exposes your content to StumbleUpon's nearly 8 million users!” su.pr tech features • Real time stats • Done directly in HBase • In depth stats • Use cascading, map reduce and put results in hbase su.pr web access • Using thrift gateway, php code accesses HBase • No additional caching other than what HBase provides Large data storage • Over 9 billion rows and 1300 GB in HBase • Can map reduce a 700GB table in ~ 20 min • That is about 6 million rows/sec • Scales to 2x that speed on 2x the hardware Micro read benches • Single reads are 1-10ms depending on disk seeks and caching • Scans can return hundreds of rows in dozens of ms Serial read speeds • A small table • A bigger table • (removed printlns from the code) Deployment considerations • Zookeeper requires IO to complete ops • Consider hosting on dedicated machines • Namenode and HBase master can coexist What to put on your nodes • Regionserver requires 2-4 cores and 3gb+ • Can’t run HDFS, HBase, maps, reduces on a 2 core system • On my 8 core systems I run datanode, regionserver, 2 maps, 2 reduces Garbage collection • GC tuning becomes important. • Quick tip: use CMS, use -Xmx4000m • Interested in G1 (if it ever stops crashing) Batch and interactive • These may not be compatible • Latency goes up with heavy batch load • May need to use 2 clusters to ensure responsive website Part 2 HBase @ Streamy • History of Data • RDBMS Issues • HBase to the Rescue • Streamy Today and Tomorrow • Future of HBase About Me • Co-Founder and CTO of Streamy.com • HBase Committer • Migrated Streamy from RDBMS to HBase and Hadoop in June 2008 History of Data The Prototype • Streamy 1.0 built on PostgreSQL ‣ All of the bells and whistles • Powered by single low-spec node ‣ 8 core / 8 GB / 2TB / $4k Functionally powerful, Woefully slow • Streamy 1.5 The Alpha built on optimized PostgreSQL History of Data partitioning ‣ Remove bells and whistles, add • Powered by high-powered master node ‣ 16 core / 64 GB / 15x146GB 15k RPM / $40k Less powerful, still slow... Insanely expensive History of Data • Streamy 2.0 built entirely on HBase ‣ Custom caches, query engines, and The Beta API • Powered by 10 low-spec nodes ‣ 4 core / 4GB / 1TB / $10k for entire cluster Less functional but fast, scalable, and cheap RDBMS Issues • Poor disk usage patterns • Black box query engine • Write speed degrades with table size • Transactions/MVCC unnecessary overhead • Expensive The Read Problem • View 30 newest unread stories from blogs ‣ Not RDBMS friendly, no early-out ‣ PL/Python heap-merge hack helped ‣ We knew what to do but DB didn’t listen The Write Problem • Rapidly growing items table ‣ Crawl index from 1k to 100k feeds ‣ Indexes, static content, dynamic statistics ‣ Solutions are imperfect RDBMS Conclusions • Enormous functionality and flexibility • Stripped down RDBMS still not attractive ‣ But you throw it out the door at scale • Turned entire team into DBAs • Gets in the way of domain-specific optimizations What We Wanted • Transparent partitioning • Transparent distribution • Fast random writes • Good data locality • Fast random reads What We Got • Transparent partitioning • Transparent distribution • Fast random writes • Good data locality • Fast random reads Regions RegionServers MemStore Column Families HBase 0.20 What Else We Got • Transparent replication • High availability • MapReduce • Versioning • Fast Sequential Reads HDFS No SPOF Input/OutputForm ats Column Versions Scanners HBase @ Streamy Today HBase @ Streamy Today • All data stored in HBase • Additional caching of hot data • Query and indexing engines • MapReduce crawling and analytics • Zookeeper/Katta/Lucene HBase @ Streamy Tomorrow • Thumbnail media server • Slave replication for Backup/DR • More Cascading • Better Katta integration • Realtime MapReduce HBase on a Budget • HBase works on cheap nodes ‣ But you need a cluster (5+ nodes) ‣ $10k cluster has 10X capacity of $40k node • Multiple instances on a single cluster • 24/7 clusters + bandwidth != EC2 Lessons Learned • Layer of abstraction helps tremendously ‣ Internal Streamy Data API ‣ Storage of serialized types • Schema design is about reads not writes • What’s good for HBase is good for Streamy • Inter-cluster / Inter-DC replication ‣ Slave and Multi-Master • Master rewrite, more Zookeeper • Batch operations, HDFS uploader • No more data loss ‣ Need HDFS appends What’s Next for HBase HBase Information • Home Page http://hbase.org • Wiki http://wiki.apache.org/hadoop/Hbase • Twitter http://twitter.com/hbase • Freenode IRC #hbase • Mailing List hbaseuser@hadoop.apache.org