Hadoop Architecture

Document Sample
Hadoop Architecture Powered By Docstoc
					Hadoop Architecture
Philippe Julio
EMEA Next Generation Datacenter Solutions Specialist
Data Management Insight

    « The Data are not created relevant,
            they become so ! »

2                                Global 500 Solutions
Data-Driven on-Line Websites

• To run the apps : messages, posts, blog entries, video
  clips, maps, web graph...
• To give the data context : friends networks, social
  networks, collaborative filtering...
• To keep the applications running : web logs, system
  logs, system metrics, database query logs...

3                                            Global 500 Solutions
New Data and Management Economics

4                           Global 500 Solutions
What Is Hadoop ?

      “Flexible and available
    architecture for large scale
      computation and data
    processing on a network of
      commodity hardware”
       Open Source + Hardware Commodity
              = IT Costs Reduction

5                                  Global 500 Solutions
What Is Hadoop used for ?

•   Searching
•   Log processing
•   Recommendation systems
•   Business Intelligence / Data Warehousing
•   Video and Image analysis

6                                     Global 500 Solutions
Who Used Hadoop ?

• Top level Apache Foundation project
• Large, active user base, mailing lists, user groups
• Very active development, strong development team

7                                             Global 500 Solutions
Who Support Hadoop ?
    • 101tec Inc. Integration, customization, consulting. (Hadoop, Pig,
      Zookeeper, Lucene, Nutch)
    • Cloudera, Inc. Get Cloudera's Distribution for Hadoop - it's free, and help
      you to optimize your configuration. We also provide commercial support
      and professional training for Hadoop. Basic training is online for free
    • Cloudify - assist organizations in integrating Cloud Computing into their
      IT and Business strategies and in building and managing scalable, next-
      generation infrastructure environments (Hadoop, Solr, AWS, distributed
    • Doculibre Inc. Open source and information management consulting.
      (Lucene, Nutch, Hadoop, Solr, Lius etc.)
    • ScaleUnlimited, Inc. Training and mentoring on large architectures.
      Hadoop Bootcamp now available
    • Tinvention -Ingegneria Informatica - Italian Consulting Company, offer
      support on open source architecture based on Java, including
      architectures based on Hadoop.

8                                                                Global 500 Solutions
Infrastructure as a Service
General Purpose Storage Servers
• Combine server with disk & networking
• Specialized software enables general purpose systems designs to provide
  high performance data services

Open Platform direction               Data moves to the infrastructure

    Data Compute and Storage          Legacy         Emerging                 Future
                                      Application     Application            Application
                                     Data Services   Data Services
                                     Metadata Mgnt

                                                                            Data Services
                                                     Metadata Mgnt         Metadata Mgnt
                                        Storage         Storage                Storage

9                                                                    Global 500 Solutions
Hadoop Ecosystem

                                  PIG (Data Flow)            HIVE (Batch SQL)             SQOOP (Data import)
     ZOOKEEPER (Coordination)

                                                                                                                     AVRO (Serialization)
                                                     (Displaying, Monitoring, Analyzing Logs)

                                MAP REDUCE (Job scheduling - Raw processing)
                                   HBASE (Real Time Query)

                                          (Hadoop Distributed File System – Unstructured Storage)

10                                                                                                  Global 500 Solutions
Hadoop Common

• Hadoop Common is a set of
  utilities that support the
  Hadoop subprojects.
• Hadoop Common includes
  Filesystem, RPC, and
  serialization libraries.

11                                      Global 500 Solutions
HDFS & MapReduce
                                                             Master Node
 • Hadoop Distributed File System
     -   A scalable, Fault tolerant, High performance
         distributed file system capable of running on Sun
     -   Hadoop cluster with 3 nodes minimum
     -   Data divided into 64MB or 128MB blocks, each
         block replicated 3 times (default)
     -   No 15k RPM disks or RAID required
     -   NameNode holds filesystem metadata
     -   Files are broken up and spread over the
 • Hadoop Map Reduce
     -   Software framework for distributed computation
     -   Input | Map() | Copy/Sort | Reduce() | Output
     -   JobTracker schedules and manages jobs
     -   TaskTracker executes individual map() and
         reduce() tasks on each cluster node
                                                             Slave Nodes

12                                                              Global 500 Solutions
Hadoop Distributed File System

NameNode                                               DataNode
• Manages file system NameSpace                        • Block Server
     -   Maps a file name to set of blocks                -   Stores data in the local file
     -   Maps a block to the DataNodes where                  system
         it resides                                       -   Stores the metadata of a block
                                                          -   Serves data and metadata to
• Cluster configuration management                            clients
• Replication engine for blocks                        • Block Report
• Metadata management                                     -   Periodically sends a report of
                                                              all existing blocks to the
     -   Metadata are in main memory                          NameNode
     -   List of files, list of blocks in each file
     -   List of DataNode in each block
                                                       • Pipeline of Data
                                                          -   Forwards data to other
     -   File attributes, replication factor...               specified DataNodes
• Transaction Log
     -   Records for file creation, file deletion...

13                                                                     Global 500 Solutions
Hadoop Distributed File System

 •   Data Correctness                                       •   Blocks Placement
     -   File creation : Client computes checksum per           -   First replica on a node in a local rack
         512 bytes – DataNode stores the checksum               -   Second replica on different rack
     -   File Access : Client retrieves the data and            -   3rd replica on other rack
         checksum from DataNode –
         If Validation fails, Client tries other replicas       -   Clients read from nearest replica

 •   Data Pipeline                                          •   Heartbeats
     -   Client retrieves a list of DataNodes on which          -   DataNodes send heartbeat to the
         to place replicas of a block                               NameNode
                                                                    (once every 3 seconds)
     -   Client writes block to the first DataNode
                                                                -   NameNode used heartbeats to detect
     -   The first DataNode forwards the data to the
                                                                    DataNode failure
         next DataNode in the Pipeline
     -   When all replicas are written, the client moves    •   Replication Engine
         on to write the next block in file
                                                                -   Chooses new DataNodes for new
 •   Rebalancer                                                     replicas
                                                                -   Balances disk usage
     -   Usually run when new DataNodes are added
                                                                -   Balances communication traffic to
     -   Cluster is online when Rebalancer is active                DataNodes
     -   Rebalancer is throttled to avoid network
     -   Command line tool

14                                                                                 Global 500 Solutions
Hadoop Distributed File System

15                                   Global 500 Solutions

16                                        Global 500 Solutions

Map Phase                         Input | Map() | Copy/Sort | Reduce() | Output

• Raw data analyzed and
  converted to name/value
Shuffle Phase
• All name/value pairs are
  sorted and grouped by their
Reduce Phase
• All values associated with a
  key are processed for results

17                                                          Global 500 Solutions
• Clone of Big Table (Google)                            •   It's not a relational database (No joins)
• Implemented in Java (Clients : Java, C++,              •   Sparse data – nulls are stored for free
  Ruby...)                                               •   Semi-structured or unstructured data
• Data is stored “Column‐oriented”                       •   Data changes through time
• Distributed over many servers                          •   Versioned data
• Tolerant of machine failure                            •   Scalable – Goal of billions of rows x
• Layered over HDFS                                          millions of columns
• Strong consistency
                Row              Timestamp             Animal              Repair
                                               Type              Size           Cost

                                     12       Zebra             Medium         1000€
     Region                          11        Lion               Big
                    Enclosure2       13       Monkey             Small         1500€

              Key                                      Family
                                  Column                                    Cell

              (Table, Row_Key, Family, Column, Timestamp) = Cell (Value)

18                                                                             Global 500 Solutions
Hadoop Architecture
 • Table
      -   Regions for scalability, defined
          by row [start_key, end_key)
      -   Store for efficiency, 1 per
           - 1..n StoreFiles
             (HFile format on HDFS)

 • Everything is byte
 • Rows are ordered sequentially
   by key
 • Special tables -ROOT- ,
      -   Tell clients where to find user

19                                                                      Global 500 Solutions

• Data Warehouse infrastructure that
  provides data summarization and ad hoc
  querying on top of Hadoop
     -   MapReduce for execution
     -   HDFS for storage

• MetaStore
     -   Table/Partitions properties
     -   Thrift API : Current clients in Php (Web
         Interface), Python interface to Hive, Java (Query
         Engine and CLI)
     -   Metadata stored in any SQL backend

• Hive Query Language
     -   Basic SQL : Select, From, Join, Group By
     -   Equi-Join, Multi-Table Insert, Multi-Group-By
     -   Batch query

20                                                           Global 500 Solutions
 • A high-level data-flow
   language and execution
   framework for parallel
   computation                   PIG Language Example

 • Simple to write MapReduce
 • Abstracts you from specific
 • Focus on data processing
 • Data flow
 • For data manipulation

21                                          Global 500 Solutions

 • Sqoop is a tool designed to
   help users of large data import
   existing relational databases
   into their Hadoop clusters
 • Automatic data import
 • SQL to Hadoop
 • Easy import data from many
   databases to Hadoop
 • Generates code for use in
   MapReduce applications
 • Integrates with Hive

22                                      Global 500 Solutions
 • A high-performance coordination service for distributed
 • ZooKeeper is a centralized service for maintaining
   configuration information, naming, providing distributed
   synchronization, and providing group services

• All servers store a copy of the data
• A leader is elected at startup
• Followers service clients, all updates go through leader
• Update responses are sent when a majority of servers have
  persisted the change

23                                                            Global 500 Solutions

 • A data serialization system that provides dynamic
   integration with scripting languages
 • Avro Data
     - Expressive
     - Smaller and Faster
     - Dynamic
        - Schema store with data
        - APIs permit reading and creating

     - Include a file format and a textual encoding
 • Avro RPC
     - Leverage versioning support
     - For Hadoop service provide cross-language access

24                                                    Global 500 Solutions

 • A data collection system for managing large distributed
 • Build on HDFS and MapReduce
 • Tools kit for displaying, monitoring and analyzing the log

25                                              Global 500 Solutions
Oracle and HDFS

 • External tables present data
   stored in a file system in a
   table format
 • Use SQL queries
 • The FUSE : File system in
 • FUSE drivers allow users to
   mount a HDFS store and treat
   it like a normal file system
 • Oracle Table Functions
   provide an alternate way to
   fetch data from Hadoop

26                                Global 500 Solutions
High Availability
                                 Hadoop Cluster
           Master Node                          Slaves Nodes
 Single Point
 Of Failure         0                    1     2         3               4

                         High Availability Hadoop Cluster
           Master Nodes                         Slaves Nodes

            0               0'           1      2        3               4

            FsImage + EditLog

27                                                           Global 500 Solutions
Components inspired for Hadoop

              1U 2S RACK                                       2U 2S RACK
                 DELL                                             DELL
            POWEREDGE R610                                  POWEREDGE C2100

                     STORAGE CENTER                         DELL EQUALLOGIC

     Purpose-built for scale-out rack deployments, large homogenous cloud/cluster
     application environments where density is required and the software stack
     provides platform availability and resiliency

28                                                                  Global 500 Solutions
High Availability with Secondary NameNode

 • Is usually run on different servers
     - Primary et Secondary NameNode

 • Copies FSImage and transaction
   Log (EditLog) from NameNode to a      M1 Server                          M2 Server

   temporary directory
 • Merges FSImage and Transaction               Primary                Secondary

   Log periodically into a new                 NameNode                NameNode

   FSImage in temporary directory
 • Uploads new FSImage to the

   NameNode                                     FSImage                 FSImage
                                                EditLog                 EditLog

 • Purges transaction Log on the                                  Temporary directory


29                                                        Global 500 Solutions
High Availability with Cluster

Active Master Node                        Cluster                            Passive Master Node
+ NameNode                                                                   + NameNode
+ JobTracker                                                                 + JobTracker
                     M1 Server                                   M2 Server

                                     Disks Mirroring RAID1


                                     TCP/IP Replication

                           FsImage                           FsImage
                           EditLog                           EditLog

30                                                                      Global 500 Solutions
High Availability with AvatarNode
• HDFS clients are configured to
  access the AvatarNode via a Virtual
  IP Address (VIP)
• When PrimaryAvatarNode is down,                  Active Primary AvatarNode          Active Standby AvatarNode
  the Standby AvatarNode takes the                 Active Standby AvatarNode          Active Primary AvatarNode
                                                   + NameNode                         + NameNode
  relay                                                                    Cluster
• The Standby AvatarNode ingests all
  committed transactions because it
                                                 M1 Server                                        M2 Server
  reopens the edits log and
  consumes all transactions until the           Read / Write      Read-only     Read / Write      Read Only
                                                                  (SafeMode)                      (SafeMode)
  end of the file
• The Standby AvatarNode finishes
  ingestion of all transactions from
  the shared NFS filer and then leaves                       FsImage   NFS-v4 Replication   FsImage
  SafeMode                                                   EditLog                        EditLog
• The VIP switches from Primary
  AvatarNode to Standby AvatarNode
 Code has been contributed to the Apache HDFS project via HDFS-976. A prerequisite for this patch is HDFS-966.

31                                                                                      Global 500 Solutions
World Wide IP Cluster HA

                                                Data Node

     Data Node                                             Data Node
                                         Data Node                            Data Node        Data Node
                 Data Node

                                                                  Data Node

                             Data Node               NameNode
                                                                                     Data Node

 Compute and storage                     High Availability with                           High Scalability
 for all data nodes                      load balancing                                   by adding

32                                                                                   Global 500 Solutions
•     Business Data Volume = Customer needs
                                                        Cloud Storage Model
•     No RAID factor
•     No HBA port
•     2 CPU Quad-core for all servers
•     2 System hard disks
•     Number of replication blocks = 3
•     Block size = 128 MB

Sizing for HA Cluster
•    Temporary Space = 25% of the total hard disk
•    Raw Data Volume = 1.25 * (Business Data Volume * Nb. of replication blocks)
•    Number of NameNode Servers = 2
•    NameNode RAM = 64 GB
•    DataNode RAM = 32 GB mini

33                                                            Global 500 Solutions
Proof Of Concept

• 1 x Primary NameNode server
                                              Cloud Storage Model
• 1 x Secondary NameNode server
• 3 x Data Nodes servers

                Dell PowerEdge C2100

                                            Network Switches

            2 CPU – Up to 24 TB SATA Disk

34                                                             Global 500 Solutions
Internal Storage
                 Rack #1   Dell PowerEdge C2100   Rack #2   Dell PowerEdge C2100   Rack #3
     24TB SATA

                              24TB SATA

     DataNodes                                                DataNodes
     24TB SATA                                                24TB SATA
       Disks                                                    Disks
                              24TB SATA

     Network                   Network                         Network
     switches                  switches                        switches

                                  LAN 1GbE                         LAN 1GbE

35                                                                            Global 500 Solutions
SAN Storage iSCSI
                             Dell PowerEdge R610
                   Rack #1                         Rack #2   Dell PowerEdge R610
                                                                                       Rack #3
      2 Local HD

                                  2 Local HD
     DataNodes                      146GB                       DataNodes
     2 Local HD                                                 2 Local HD
       146GB                                                      146GB
                                  2 Local HD

         Dell                        Dell                          Dell
      EqualLogic                  EqualLogic                    EqualLogic
       Storage                     Storage                       Storage

       Network                     Network                       Network
       switches                    switches                      switches

                                      LAN 10GbE                      LAN 10GbE

36                                                                                 Global 500 Solutions
SAN Storage FCoE
                          Rack #1   Dell PowerEdge R610     Rack #2   Dell PowerEdge R610     Rack #3

     2 Local HD SAS
      Data Nodes                        NameNode
      2 Local HD                        2 Local HD
                                                                         Data Nodes
        146GB                             146GB
                                                                         2 Local HD
                                        Data Nodes
                                        2 Local HD

                                         Network                          Network
       Network                           Switches                         Switches

                                           LAN 10GbE                       LAN 10GbE

              Dell Compellent                    Dell Compellent                 Dell Compellent
              Storage Center                     Storage Center                  Storage Center

37                                                                                          Global 500 Solutions

Description: Great presentations from experts about various web technologies and programming languages.