ETIS11 - Agile Business Intelligence - Presentation

Document Sample
ETIS11 - Agile Business Intelligence - Presentation Powered By Docstoc
					DATA MANAGEMENT & WAREHOUSING        CYPRUS ‘11

DAVID M WALKER




                                   AGILE
                                BUSINESS
                           INTELLIGENCE
AGILE MANIFESTO
                                                                                                     CYPRUS ‘11




                   We are uncovering better ways of developing
                   software by doing it and helping others do it.
                    Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
  Customer collaboration over contract negotiation
     Responding to change over following a plan

                           That is, while there is value in the items on
                         the right, we value the items on the left more.

                       http://www.agilemanifesto.org/                                     Utah, Feb 2001
 14-­‐Oct-­‐2011	
              ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
                   2	
  
                                                                                                                              CYPRUS ‘11
TWELVE PRINCIPLES OF AGILE SOFTWARE
                      •    Our highest priority is to satisfy the                   •      Working software is the primary
                           customer through early and                                      measure of progress.
                           continuous delivery of valuable
                           software.
                      •    Welcome changing requirements,                           •      Agile processes promote sustainable
                           even late in development. Agile                                 development. The sponsors,
                           processes harness change for the                                developers, and users should be able
                           customer's competitive advantage.                               to maintain a constant pace
                                                                                           indefinitely.
                      •    Deliver working software frequently,                     •      Continuous attention to technical
                           from a couple of weeks to a couple of                           excellence and good design enhances
                           months, with a preference to the                                agility.
                           shorter timescale.
                      •    Business people and developers must                      •      Simplicity--the art of maximizing the
                           work together daily throughout the                              amount of work not done--is essential.
                           project.
                      •    Build projects around motivated                          •      The best architectures, requirements,
                           individuals. Give them the                                      and designs emerge from self-
                           environment and support they need,                              organizing teams.
                           and trust them to get the job done.
                      •    The most efficient and effective                         •      At regular intervals, the team reflects
                           method of conveying information to                              on how to become more effective,
                           and within a development team is                                then tunes and adjusts its behavior
                           face-to-face conversation.                                      accordingly.
14-­‐Oct-­‐2011	
                        ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
                                       3	
  
LARGE ORGANIZATIONS FORGET
                                                                                         CYPRUS ‘11




                       •  Most companies start with an
                          innovative and entrepreneurial
                          people and processes that get things
                          done – they are inherently agile
                       •  As organizations grow they put
                          structures in place that standardize
                          the organization but these limit
                          creativity, increase timescales and
                          reduce risk at the cost of reducing
                          benefits and reward
 14-­‐Oct-­‐2011	
             ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
        4	
  
THE PROCESS IMPROVEMENT PARADOX
                                                                                            CYPRUS ‘11




                       •  The paradox (…) is that process
                          improvement is good, but process
                          improvement programs aren't, or at least
                          they often aren't.
                       •  Organizations become more and more
                          averse to risk as they "mature”. An
                          organization under the gun to demonstrate
                          increased CMM level is not going to go
                          looking for real challenge.
                       •  This (…) entices the play-it-safe behavior of
                          low-risk, and therefore low-benefit, projects.

                       Peopleware; Tom De Marco & Timothy Lister; 1987

 14-­‐Oct-­‐2011	
                ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
        5	
  
SUCCESSFUL LARGE ORGANIZATIONS                                                              CYPRUS ‘11


RECOGNIZE AND REACT
                       •  SkunkWorks
                         –  Lockheed Martin Advance Development Projects
                         –  A group within an organization given a high degree
                            of autonomy and unhampered by bureaucracy,
                            tasked with working on advanced projects.
                         –  Responsible for U-2 & Blackbird Spy Planes, Stealth
                            Fighter & Bomber
                         –  Founded in June 1943 specifically to overcome the
                            hurdles of government and large corporations
                            working together
                         –  SkunkWorks has 14 rules and practices
                             •  Very Similar to the 12 Agile Principles

                       Ben Rich & Leo Janos; SkunkWorks; 1996


 14-­‐Oct-­‐2011	
                ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
        6	
  
OTHER EXAMPLES
                                                                                          CYPRUS ‘11




                       •  In the late 80s/early 90s Telcos led the
                          way in Agile BI development
                         –  Vodafone: Mast Placement Project
                         –  Swisscom: Geo-located SMS Marketing
                       •  But they have been overtaken by the
                          search and social networks websites
                          that rely on rapidly creating and
                          consuming BI data to survive
                         –  Examples: Facebook, Google & LinkedIn
                         –  These companies are also creating and
                            using the next generation of BI tools and BI
                            engines
 14-­‐Oct-­‐2011	
              ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
        7	
  
AGILE BI IS NOW A 2-SPEED HIGHWAY
                                                                                           CYPRUS ‘11




                       •  Truly Agile BI Organizations
                          –  Small, strong, highly skilled teams with a
                             strong, trusting, focused user relationship
                          –  Project team having delegated authority and
                             responsibility for delivery with long term
                             funding
                          –  The ability to operate outside standard
                             corporate procedures (e.g. procurement)
                          –  Regular delivery of incremental improvement
                       •  Those that describe themselves as “wanting
                          to be more agile”
                          –  Willing to adopt some of the methods but
                             unable to break free from the corporate chains

 14-­‐Oct-­‐2011	
               ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
        8	
  
                                                                                   CYPRUS ‘11




                                         TECHNIQUES

                      BECOMING MORE AGILE




14-­‐Oct-­‐2011	
        ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
        9	
  
HOW TO BE MORE AGILE
                                                                                            CYPRUS ‘11




                       •    Team Structures
                       •    Wiki-fy Everything
                       •    Use A Dedicated Platform
                       •    Build Literal Staging Areas
                       •    Throw out your ETL Tool?
                       •    More Dynamic Reporting Tools
                       •    Develop reports with your users
                       •    Embrace (not so) New Technologies

 14-­‐Oct-­‐2011	
                ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
       10	
  
TEAM STRUCTURES
                                                                                         CYPRUS ‘11




                       •  Agile teams are generally smaller but
                          with broader and deeper skill sets
                         –  These resources are more expensive
                            individually but cheaper collectively
                         –  Smaller teams significantly reduce the
                            management and communication
                            overhead
                         –  Close communications and broad skills
                            are more likely to generate innovative
                            solutions

 14-­‐Oct-­‐2011	
             ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
       11	
  
WIKI-FY EVERYTHING
                                                                                          CYPRUS ‘11




                       •  Projects need a consistent, persistent,
                          versioned knowledge store
                       •  Use a Wiki and train your business
                          users how to use it
                       •  Optimizes the ‘documentation’ set
                          collected in a single structured way
                       •  Abandon office products and
                          SharePoint for critical documents
                       •  Don’t store your requirements in
                          e-mails, IM chats and documents
 14-­‐Oct-­‐2011	
              ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
       12	
  
USE A DEDICATED PLATFORM
                                                                                          CYPRUS ‘11




                       •  Marketplace has a choice of BI specific,
                          low maintenance, low TCO appliances
                         –  IBM Netezza
                         –  Teradata 14 (AsterData)
                         –  Sybase IQ
                       •  And some high power workhorses
                         –  Oracle Exadata
                         –  Teradata
                       •  There are plenty of emerging
                          technologies
                         –  Curt Monash: http://www.dbms2.com/

 14-­‐Oct-­‐2011	
              ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
       13	
  
BUILD LITERAL STAGING AREAS
                                                                                           CYPRUS ‘11




                       •  These are 1:1 copies of your source systems
                          (hence the ‘Literal’)
                       •  Do this whilst others are collecting
                          requirements, building data models, etc.
                       •  Do some test reporting off the LSA with the
                          users – it starts the user engagement, helps
                          evolve the business requirements, and
                          develops communication
                       •  They will be an essential source for the data
                          warehouse as it evolves and remove load
                          from the operational systems

 14-­‐Oct-­‐2011	
               ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
       14	
  
THROW OUT YOUR ETL TOOL ?
                                                                                           CYPRUS ‘11




                       •  ETL tools require product specific expertise
                          –  Value add features are under-utilized
                          –  Developers and DBAs use them as glorified
                             scheduling tools
                       •  Good Source Code Control and Management
                          Scripting can compensate for much of the
                          lost benefits
                       •  Significantly reduces the project cost and
                          increases the available skilled resources
                       •  InsureTheBox/Netezza; YapiKredi/SybaseIQ;
                          NonDisclosure/Greenplum have major data
                          warehouses with no ETL product

 14-­‐Oct-­‐2011	
               ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
       15	
  
MORE DYNAMIC REPORTING TOOLS
                                                                                          CYPRUS ‘11




                       •  Reports on your hand held device
                         –  Qlikview, RoamBI, PushBI, Tableau
                       •  Use Dynamic Web Deployed Tools
                         –  Panopticon, Tableau
                       •  RSS Feeds
                         –  Usable by any RSS Reader on any platform

                       •  Requires dedicated, reactive reporting
                          experts that work enthusiastically with
                          the business users and are willing to go
                          the extra mile
 14-­‐Oct-­‐2011	
              ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
       16	
  
DEVELOP REPORTS WITH YOUR USERS
                                                                                                                 CYPRUS ‘11




                                                                                      •    De La Rue / Tableau
                                                                                           Reporting
                                                                                      •    First 50 reports deployed
                                                                                           over 28 days
                                                                                      •    Initially deployed to users
                                                                                           from the LSA and then
                                                                                           migrated to the data
                                                                                           marts
                                                                                      •    Near real-time (fifteen
                                                                                           minutes maximum delay)
                                                                                           data



                       •  Averages
                         –  Day 5: 17 changes / 5 users
                         –  Day 10: 1 change / 42 users
                         –  Day 15: 0 changes / 50 users
 14-­‐Oct-­‐2011	
          ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
                                   17	
  
EMBRACE (NOT SO) NEW TECHNOLOGY -                                                            CYPRUS ‘11


MAPREDUCE/HADOOP/NOSQL
                       •  Software frameworks that supports data-intensive
                          distributed applications and enables them to work with
                          thousands of nodes and petabytes of data.
                       •  Hadoop is a filesystem (HDFS) and distributed
                          programming framework (MapReduce) whilst a NoSQL
                          database consists of key-value pairs, no joins, data is
                          sharded and replicated, no single point of failure.
                       •  Notable BI Use (but not the only one): Rapid loading
                          and processing of volume data to create and validate
                          data sets for onward inclusion in the data warehouse
                          whilst doing large scale, near real-time ‘dirty’ analysis
                       •  Notable Users: Facebook, Twitter, LinkedIn, Yahoo!,
                          eBay, Amazon, Google
                       •  Technologies in use since at least 2004


 14-­‐Oct-­‐2011	
                 ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
       18	
  
EMBRACE (NOT SO) NEW TECHNOLOGY –                                                          CYPRUS ‘11


COMPLEX EVENT PROCESSING (CEP)
                       •  Complex event processing (CEP) consists of
                          processing many events happening across all
                          the layers of an organization, identifying the
                          most meaningful events within the event cloud,
                          analyzing their impact, and taking subsequent
                          action in real time.
                       •  Already used for fraud and network
                          management but has applications in customer
                          service and market to fine tune customer
                          interactions in real time
                       •  Notable BI Use: real-time information alongside
                          existing BI – rather than trying to create an
                          entire real-time solution
                       •  Technologies in use since at least 1999
 14-­‐Oct-­‐2011	
               ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
       19	
  
BUT REMEMBER …
                                                                                              CYPRUS ‘11




                       •  Agile is all about approach and people:

                          Individuals and interactions over processes and tools
                          Working software over comprehensive documentation
                          Customer collaboration over contract negotiation
                          Responding to change over following a plan


                       •  Your management need to be willing to create:

                          A group within an organization given a high degree of
                          autonomy and unhampered by bureaucracy, tasked with
                          working on advanced projects.



 14-­‐Oct-­‐2011	
                  ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
       20	
  
REFERENCE MATERIAL
                                                                                            CYPRUS ‘11




                       •  Websites
                         –  Agile Manifesto
                            •  12 Principles of Agile
                         –  SkunkWorks
                            •  14 Rules & Practices
                         –  Hadoop and NoSQL Myth-busting
                       •  Books
                         –  Peopleware: Productive Projects & Teams
                         –  SkunkWorks: A Personal Memoir
                         –  Mythical Man Month: Essays on Software
                            Engineering

 14-­‐Oct-­‐2011	
                ©	
  2011	
  Data	
  Management	
  &	
  Warehousing	
       21	
  
THANK YOU           CYPRUS ‘11




                  AGILE
               BUSINESS
            INTELLIGENCE

				
DOCUMENT INFO
Shared By:
Tags:
Stats:
views:32
posted:4/24/2012
language:
pages:22