HBase, Hadoop World NYC
Ryan Rawson, Stumbleupon.com, su.pr Jonathan Gray, Streamy.com
A presentation in 2 parts
Part 1
About Me
• Ryan Rawson • Senior Software Developer @
Stumbleupon
• HBase committer, core contributor
Stumbleupon
• Uses HBase in production • Behind features of our su.pr service • More later
Adventures with MySQL
• Scaling MySQL hard, Oracle expensive
(and hard)
• Machine cost goes up faster speed • Turn off all relational features to scale • Turn off secondary (!) indexes too! (!!)
MySQL problems cont.
• Tables can be a problem at sizes as
low as 500GB
• Hard to read data quickly at these sizes • Future doesn’t look so bright as we
contemplate 10x sizes
• MySQL master becomes a problem...
Limitations of masters
• What if your write speed is greater than
a single machine?
• All slaves must have same write
capacity as master (can’t cheap out on slaves)
• Single point of failure, no easy failover • Can (sort of) solve this with sharding...
Sharding
Sharding problems
• Requires either a hashing function or
mapping table to determine shard
• Data access code becomes complex • What if shard sizes become too large...
Resharding!
What about schema changes?
• What about schema changes or
migrations?
• MySQL not your friend here • Only gets harder with more data
HBase to the rescue
• Clustered, commodity(ish) hardware • Mostly schema-less • Dynamic distribution • Spreads writes out over the cluster
What is HBase?
• HBase is an open-source distributed • Part of the Hadoop ecosystem • Layers on HDFS for storage • Native connections to map reduce
database, inspired by Google’s bigtable
HBase storage model
• Column-oriented database • Column name is arbitrary data, can
have large, variable, number of columns per row
• Rows stored in sorted order • Can random read and write
Tables
• Table is split into roughly equal sized
“regions”
• Each region is a contiguous range of
keys, from [start, to end)
• Regions split as they grow, thus
dynamically adjusting to your data set
Server architecture
• Similar to HDFS: • Master = Namenode (ish) • Regionserver = Datanode (ish) • Often run these alongside each other!
Server Architecture 2
• But not quite the same, HBase stores
state in HDFS
• HDFS provides robust data storage
across machines, insulating against failure
• Master and Regionserver fairly
stateless and machine independent
Region assignment
• Each region from every table is
assigned to a Regionserver
• The master is responsible for
assignment and noticing if (when!) regionservers go down
Master Duties
• When machines fail, move regions from
affected machines to others
• When regions split, move regions to
balance cluster
• Could move regions to respond to load • Can run multiple backup masters
What Master does NOT do
• Does not handle any write requests (not
a DB master!)
• Does not handle location finding
requests
• Not involved in the read/write path! • Generally does very little most the time
Distributed coordination
• To manage master election and server
availability we use ZooKeeper
• Set up as a cluster, provides distributed
coordination primitives
• An excellent tool for building cluster
management systems
Scaling HBase
• Add more machines to scale • Base model (bigtable) scales past
1000TB
• No inherent reason why HBase couldn’t
What to store in HBase?
• Maybe not your raw log data...
• ... but the results of processing it with
Hadoop!
• By storing the refined version in HBase,
can keep up with huge data demands and serve to your website
HBase & Hadoop
• Provides a real time, structured storage
layer that integrates on your existing Hadoop clusters
• Provides “out of the box” hookups to
map-reduce.
• Uses the same loved (or hated)
management model as Hadoop
HBase @
Stumbleupon & HBase
• Started investigating the field in Jan ’09 • Looked at 3 top (at the time) choices: • Cassandra • Hypertable • HBase
cassandra didnt work, didnt like data model - hypertable fast but community and project viability (no major users beyond zvents) - hbase local and good community
Stumbleupon & HBase
• Picked HBase: • Community • Features • Map-reduce, cascading, etc • Now highly involved and invested
su.pr marketing
• “Su.pr is the only URL shortener that
also helps your content get discovered! Every Su.pr URL exposes your content to StumbleUpon's nearly 8 million users!”
su.pr tech features
• Real time stats • Done directly in HBase • In depth stats • Use cascading, map reduce and put
results in hbase
su.pr web access
• Using thrift gateway, php code
accesses HBase
• No additional caching other than what
HBase provides
Large data storage
• Over 9 billion rows and 1300 GB in
HBase
• Can map reduce a 700GB table in ~ 20
min
• That is about 6 million rows/sec • Scales to 2x that speed on 2x the
hardware
Micro read benches
• Single reads are 1-10ms depending on
disk seeks and caching
• Scans can return hundreds of rows in
dozens of ms
Serial read speeds
• A small table • A bigger table • (removed printlns from the code)
Deployment considerations
• Zookeeper requires IO to complete ops • Consider hosting on dedicated
machines
• Namenode and HBase master can coexist
What to put on your nodes
• Regionserver requires 2-4 cores and
3gb+
• Can’t run HDFS, HBase, maps, reduces
on a 2 core system
• On my 8 core systems I run datanode,
regionserver, 2 maps, 2 reduces
Garbage collection
• GC tuning becomes important. • Quick tip: use CMS, use -Xmx4000m • Interested in G1 (if it ever stops
crashing)
Batch and interactive
• These may not be compatible • Latency goes up with heavy batch load • May need to use 2 clusters to ensure
responsive website
Part 2
HBase @ Streamy
• History of Data • RDBMS Issues • HBase to the Rescue • Streamy Today and Tomorrow • Future of HBase
About Me
• Co-Founder and CTO of Streamy.com • HBase Committer • Migrated Streamy from RDBMS to
HBase and Hadoop in June 2008
History of Data
The Prototype
• Streamy 1.0 built on PostgreSQL ‣ All of the bells and whistles
• Powered by single low-spec node ‣ 8 core / 8 GB / 2TB / $4k
Functionally powerful, Woefully slow
• Streamy 1.5 The Alpha built on optimized
PostgreSQL
History of Data
partitioning
‣ Remove bells and whistles, add
• Powered by high-powered master node ‣ 16 core / 64 GB / 15x146GB 15k
RPM / $40k Less powerful, still slow... Insanely expensive
History of Data
• Streamy 2.0 built entirely on HBase ‣ Custom caches, query engines, and
The Beta
API
• Powered by 10 low-spec nodes ‣ 4 core / 4GB / 1TB / $10k for entire
cluster Less functional but fast, scalable, and cheap
RDBMS Issues
• Poor disk usage patterns • Black box query engine • Write speed degrades with table size • Transactions/MVCC unnecessary
overhead
• Expensive
The Read Problem
• View 30 newest unread stories from
blogs
‣ Not RDBMS friendly, no early-out ‣ PL/Python heap-merge hack helped ‣ We knew what to do but DB didn’t
listen
The Write Problem
• Rapidly growing items table ‣ Crawl index from 1k to 100k feeds ‣ Indexes, static content, dynamic
statistics
‣ Solutions are imperfect
RDBMS Conclusions
• Enormous functionality and flexibility • Stripped down RDBMS still not
attractive
‣
But you throw it out the door at scale
• Turned entire team into DBAs • Gets in the way of domain-specific
optimizations
What We Wanted
• Transparent partitioning • Transparent distribution • Fast random writes • Good data locality • Fast random reads
What We Got
• Transparent partitioning • Transparent distribution • Fast random writes • Good data locality • Fast random reads
Regions
RegionServers
MemStore
Column Families
HBase 0.20
What Else We Got
• Transparent replication • High availability • MapReduce • Versioning • Fast Sequential Reads
HDFS No SPOF Input/OutputForm ats Column Versions Scanners
HBase @ Streamy
Today
HBase @ Streamy
Today
• All data stored in HBase • Additional caching of hot data • Query and indexing engines • MapReduce crawling and analytics • Zookeeper/Katta/Lucene
HBase @ Streamy
Tomorrow
• Thumbnail media server • Slave replication for Backup/DR • More Cascading • Better Katta integration • Realtime MapReduce
HBase on a Budget
• HBase works on cheap nodes ‣ But you need a cluster (5+ nodes) ‣ $10k cluster has 10X capacity of
$40k node
• Multiple instances on a single cluster • 24/7 clusters + bandwidth != EC2
Lessons Learned
• Layer of abstraction helps tremendously ‣ Internal Streamy Data API ‣ Storage of serialized types • Schema design is about reads not
writes
• What’s good for HBase is good for
Streamy
• Inter-cluster / Inter-DC replication ‣ Slave and Multi-Master • Master rewrite, more Zookeeper • Batch operations, HDFS uploader • No more data loss ‣ Need HDFS appends
What’s Next for HBase
HBase Information
• Home Page http://hbase.org • Wiki
http://wiki.apache.org/hadoop/Hbase
• Twitter http://twitter.com/hbase • Freenode IRC #hbase • Mailing List hbaseuser@hadoop.apache.org