Docstoc

White Paper - Data Warehouse Governance

Document Sample
White Paper - Data Warehouse Governance Powered By Docstoc
					Data Management & Warehousing




                                                              WHITE PAPER


        Data Warehouse Governance
                                                      DAVID M WALKER
                                                                           Version: 1.1
                                                                      Date: 06/04/2007




                      Data Management & Warehousing

   138 Finchampstead Road, Wokingham, Berkshire, RG41 2NU, United Kingdom

                          http://www.datamgmt.com
                                            White Paper - Data Warehouse Governance




Table of Contents

Table of Contents ...................................................................................................................... 2
Synopsis .................................................................................................................................... 3
Intended Audience .................................................................................................................... 3
About Data Management & Warehousing ................................................................................. 3
Introduction................................................................................................................................ 4
What is governance?................................................................................................................. 5
What affects the data warehouse? ............................................................................................ 6
What is affected within the data warehouse? ............................................................................ 7
How often will change occur?.................................................................................................... 9
What organisational structures are needed?........................................................................... 10
   Executive Sponsor(s) .......................................................................................................... 10
   Steering Committee ............................................................................................................ 11
   Programme Management ................................................................................................... 11
   Implementation Teams........................................................................................................ 12
   Exploitation Teams.............................................................................................................. 12
   User Forums ....................................................................................................................... 12
   Certification Committee....................................................................................................... 13
Project Development Methodologies....................................................................................... 14
   Waterfall .............................................................................................................................. 14
   Iterative ............................................................................................................................... 14
   Agile .................................................................................................................................... 15
   Cowboy Coding................................................................................................................... 16
Best practices .......................................................................................................................... 17
   Momentum .......................................................................................................................... 17
   Monitor and Measure .......................................................................................................... 17
   Prioritise .............................................................................................................................. 17
   Light touch........................................................................................................................... 17
   Communication ................................................................................................................... 17
   Standards............................................................................................................................ 18
   Track processes.................................................................................................................. 18
   Understand cost and value ................................................................................................. 18
   Continuous Learning and Improvement .............................................................................. 18
   Pilot & Prototype ................................................................................................................. 18
   Authority to act .................................................................................................................... 18
   Plan for the long term.......................................................................................................... 19
Governance and Success ....................................................................................................... 19
Summary ................................................................................................................................. 20
Appendices.............................................................................................................................. 21
   Appendix 1 - Programme or Project?.................................................................................. 21
   Appendix 2 – The "Declaration of Interdependence" .......................................................... 22
   Appendix 3 - Team Values and Principles .......................................................................... 23
References .............................................................................................................................. 24
   Web resources .................................................................................................................... 24
Acknowledgements ................................................................................................................. 24
Copyright ................................................................................................................................. 24




         © 2007 Data Management & Warehousing                                                                                          Page 2
                                White Paper - Data Warehouse Governance




Synopsis
An organisation that is embarking on a data warehousing project is undertaking a long-term
development and maintenance programme of a computer system. This system will be critical
to the organisation and cost a significant amount of money, therefore control of the system is
vital. Governance defines the model the organisation will use to ensure optimal use and re-
use of the data warehouse and enforcement of corporate policies (e.g. business design,
technical design and application security) and ultimately derive value for money.

This paper has identified five sources of change to the system and the aspects of the system
that these sources of change will influence in order to assist the organisation to develop
standards and structures to support the development and maintenance of the solution. These
standards and structures must then evolve, as the programme develops to meet its changing
needs.
                                                                                                 1
       “Documentation is not understanding, process is not discipline, formality is not skill”

The best governance must only be an aid to the development and not an end in itself. Data
Warehouses are successful because of good understanding, discipline and the skill of those
involved. On the other hand systems built to a template without understanding, discipline and
skill will inevitably deliver a system that fails to meet the users’ needs and sooner rather than
later will be left on the shelf, or maintained at a very high cost but with little real use.



Intended Audience
Reader                                              Recommended Reading
Executive                                           Synopsis and Summary
Business Users                                      Entire Document
IT Management                                       Entire Document
IT Strategy                                         Entire Document
IT Project Management                               Entire Document
IT Developers                                       Entire Document




About Data Management & Warehousing
Data Management & Warehousing is a specialist consultancy in data warehousing based in
Wokingham, Berkshire in the United Kingdom. Founded in 1995 by David M Walker, our
consultants have worked for major corporations around the world including the US, Europe,
Africa and the Middle East. Our clients are invariably large organisations with a pressing need
for business intelligence. We have worked in many industry sectors but have specialists in
Telco’s, manufacturing, retail, financial and transport as well as technical expertise in many of
the leading technologies.

For further information visit our website at: http://www.datamgmt.com




1
    Agile Software Development, Cockburn, A., Addison-Wesley, 2002


        © 2007 Data Management & Warehousing                                                     Page 3
                                White Paper - Data Warehouse Governance




Introduction
Data warehouse governance defines the model the organisation will use to ensure optimal
use and re-use of the data warehouse and enforcement of corporate policies (e.g. business
design, technical design and application security).

Governance structures must be in place as early as possible, ideally before the project starts
and before development begins. They are implemented in parallel with the initial development
to ensure that they work together from the outset.

A data warehouse is, by definition, a system that interacts with large parts of the organisation.
At a technical level this involves the number of systems feeding data to and being fed data
from the data warehouse. At a human level it relates to the people who interact with the
system. These include those querying the system, those operating the technical environment
and those directing the business who may not use the system directly but will mandate that
others do so.

The published literature on the subject falls into three categories:

    •     Home grown – what a specific project did within a specific environment
    •     Vendor specific – what to do when using the vendors tool set
    •     System Integrator proprietary – how an SI say they will do it if they are given the
          contract

The problem is that the data warehouse interacts with and across so much of the
organisation. A programme must deploy standards and processes that integrate into the
organisations’ existing policies and procedures.

This paper looks at the areas that must be covered by governance and discusses some of the
standards, options and issues. These ideas should be incorporated into the governance of the
data warehouse.




        © 2007 Data Management & Warehousing                                               Page 4
                                White Paper - Data Warehouse Governance




What is governance?
In the introduction governance is defined as ‘the model the organisation will use to ensure
optimal use and re-use of the data warehouse and enforcement of corporate policies’, but
what does it mean?

Data Warehouses are not a single short-term project. They are long-term programmes of
work that includes a number of projects and a significant amount of maintenance. The
decision to deploy a data warehouse commits the business to a programme that needs to be
controlled to ensure that it provides business benefit and value for money.

The role of governance is to provide the policies, processes and procedures necessary to
ensure that the programme of work is effective. These must be clearly communicated to
everyone involved. Governance covers the integrated management of the programme from its
initial development, through production running to end-of-life close down. The governance
structures must also be reviewed over the lifetime of the programme. This is to ensure that
the governance is neither too little to allow effective control or so much such that programme
work is stifled.

Different organisations also have different requirements for the level of detail and breadth of
scope that should be covered by Data Warehouse governance. For example, does
governance imply coding standards? For some people this is far too much detail, for others it
is essential that the governance of the programme ensures that specific standards are
developed and that staff adhered to them.

The key to designing a suitable level of governance within an organisation is in understanding
the following:

    •     What events affect the data warehouse environment?

    •     What is affected in the data warehouse by the impacting event?

    •     What type of organisation is needed to support the data warehouse?

    •     What are the policies and procedures needed to control the data warehouse?

From this it is possible to develop the standards, processes and communications that will both
drive the programme forward and culturally fit the specific organisation.

The processes, used in conjunction with the standards, are the way in which decisions are
made. Good processes enabling quick, effective and well-communicated decisions and
consequently effective management of the solution. The converse is also true as poor
standards and processes lead to delays and unresolved issues that discredit the data
warehouse and create significant cost or overheads that eventually destroy the programme.




        © 2007 Data Management & Warehousing                                             Page 5
                                White Paper - Data Warehouse Governance




What affects the data warehouse?
Data Management & Warehousing has identified five types of change that impact a data
warehouse:

    •     Demand

          Demand is the change needed to the system to give the users what they want. This
          may include loading new data, possibly from new sources or restructuring existing
          information. It can also include adjusting system availability or the time at which data
          is loaded or any other user requirement.

    •     External Change

          A data warehouse has a large number of systems feeding it. The systems operate in
          a complex and integrated environment. For example, a data warehouse might have
          three source systems and each source system is only allowed to perform upgrades
          and patches once a quarter. As a result the data warehouse is faced with twelve
          changes a year or one a month. In an ideal environment these changes will be
          developed and regression tested in line with the change in the source system. In
          practice most data warehouses have many more sources and more frequent
          changes. When these changes are not communicated they become issues.
          Managing external change prevents issues and therefore reduces ‘emergency fixes’
          and ‘tactical solutions’

    •     Maintenance

          All computer system have to manage routine maintenance issues such as software
          patches, upgrades to software or hardware, special backups for year end, etc. It is
          easy to ignore upgrades - the system is currently working so why bother? There
          comes a point when the software or hardware vendor moves the product out of
          support. At that point any system failure can have critical consequences. Routine
          maintenance should be regarded as a critical part of “business as usual” and
          allowance made for its impact.

    •     Risks

          Risks to the data warehouse can be identified by those in control. These risks need
          to be addressed before they affect the system. Over recent years typical risks have
          included changes in regulatory policies, Year 2000 issues, currency changes (e.g.
                                          2                             3
          revaluations, moving to the Euro or removing trailing zeros ). There are also risks
          from corporate mergers and acquisitions and from the bankruptcy of other
          companies that provide data to supplement the data warehouse (e.g. with market
          share information). These risks will create significant work with tight deadlines.




2
  Currently 13 countries are in the Euro zone
(http://en.wikipedia.org/wiki/Eurozone#Official_members)
3
  49 countries including Turkey, Greece, Israel, Brazil and Argentina
(http://www.allaboutturkey.com/ytl.htm)


        © 2007 Data Management & Warehousing                                                Page 6
                                White Paper - Data Warehouse Governance



    •     Issues

          In an ideal world all of the above would ensure that the data warehouse would never
          have any issues. The reality is a continuous flow of low-level problems and
          occasionally critical issues that need to be handled. In some ways this can be a
          measure of the success of the solution that people care enough to report issues and
                        4
          need support. There are cost and resource implications in handling these issues.



What is affected within the data warehouse?5
Once a data warehouse development has begun these areas can be affected:

    •     Requirements Catalogue

          All data warehouse projects should have a comprehensive catalogue of
          requirements. These will be developed at the start of the system and maintained
          when new or changed requirements emerge. Requirements are not the same as the
          functionality of the system. This is because not all requirements may be met,
          however they provide a catalogue of work that the business needs to be delivered.
          This also provides a check that new demands have not already been met by another
          means.

    •     Technical Infrastructure

          The technical infrastructure is the hardware, software and network used to build the
          system. It can be affected by many different factors including upgrades for better
          performance, platform re-hosting or a re-architecting exercise (e.g. moving from
          desktop to web-based clients)

    •     Configuration Management

          Configuration management covers all aspect of managing the developed code and
          any system configuration files that exist outside the code. Suitable tools must be
          used to ensure that the people responsible for the maintenance of the system can
          look at all the different versions of code over time. This enables the deployment of
          new code. It also can help when there is a need to roll back changes, for example
          reverting to older code after a failed upgrade or to read an old format source file. It is
          also important that the code can be related to data models for the data warehouse
          and to the code versions and data models of the external systems that either supply
          or use data from the data warehouse.




4
  One Data Management & Warehousing client organisation with 2000 users and a 20Tb
system raised 10,000 issues in one year post implementation – or about 40 per day
5
  This section refers to various documents such as requirements etc. Depending on who has
developed the system they will have their own names and templates for these documents.
For consistency this document refers to the templates and tools used in the Data
Management & Warehousing (http://www.datamgmt.com) Documentation Roadmap which
can be found at: http://www.datamgmt.com/index.php?module=article&view=77


        © 2007 Data Management & Warehousing                                                  Page 7
                            White Paper - Data Warehouse Governance



•     Test Management

      Test management is about asking questions to ensure that the impact of any change
      is well understood and that preventable errors are avoided. Questions like:

          o   After a change is made, how is it tested?
          o   Is there sufficient data to be considered a representative sample?
          o   Is the whole system regression tested, or are changes isolated and therefore
              separately testable?
          o   How can changes or testing be modified so that tests can be isolated?
          o   How long does it take to test a change and will this impact emergency fixes to
              the system?
          o   Does the change affect the performance or the data quality of the solution?
          o   Can the figures be calculated by an alternative independent method to
              ensure that the numbers produced are correct?
          o   Does the project have sufficient resources (e.g. size of machine, people,
              time, etc.) to perform the tests?

•     Data Stewardship

      Data stewardship is about understanding who is responsible for the data within the
      system. Again the role is about asking questions:

          o   Who decides which data quality rules should be applied?
          o   Where is data cleansed?
          o   Who is responsible for cleaning the data?
          o   Can systems or processes be changed to improve the quality of the data
              before it is sent to the data warehouse?
          o   Is data cleansed in the source system and if so is this work also done on
              historical data?
          o   What is the impact if historical data that is already in the data warehouse is
              cleansed in the source?
          o   What is the relationship between time and data e.g. is 95% of the data
              available after one day good enough or must 100% of the data be there
              before it can be used?

•     Service Level Agreement

      The service level agreement defines much about the performance and availability of
      the system. It will include the times set aside for tasks such as backups and
      maintenance. It also defines the times that the system will be available to the users.
      It should also include agreements on how to respond in case of emergencies and
      how to manage systems failures. The Service Level Agreement also defines how any
      change that affects the system (either positively or negatively) needs to be
      communicated to the user community.

•     Support Model

      The support model describes the processes and escalation paths for any user
      support request. This model must be updated to reflect any changes to the system or
      any changes in the organisation. The support model should be modified to improve
      the responsiveness of support by understanding the frequently asked questions and
      improving the response time explicitly on these items.




    © 2007 Data Management & Warehousing                                                 Page 8
                                White Paper - Data Warehouse Governance



    •     Training Plans

          Users and operators will need to be trained to use the system when it is first
          deployed. There is need for on-going training as the system changes, when new staff
          arrive or when parts of the solution are either commissioned or de-commissioned.
          Users will also need refresher courses to reduce the number and cost of calls to the
          support desk.

    •     Schedules

          The system will have an operational schedule that will determine not only when
          users can be on the system (also covered in part by the service level agreement).
          The schedule defines which jobs to extract, transform or load data can be run, when
          they are run and the dependencies between them.

    •     Monitoring

          The system must be monitored and alerts directed to some function that can
          prioritise them, act and if necessary escalate appropriately. The process by which
          this is managed and its efficiency is critical to providing a viable service to the users.
          Monitoring also ensures that service level agreements are met. Analysing the system
          events allows technical support to improve the system in the same way that
          analysing support calls assists the support team improve the support. The analysis
          must look for ways in which to bring about significant improvement to the system by
          either changing the system or improving the response to frequently occurring events.

How often will change occur?
The ten areas affected by change and the five types of change described above result in the
table below along with the likely frequency of any given type of change on a specific area of
the data warehouse:
                                                           External Change


                                                                               Maintenance
                                                  Demand




                                                                                                     Issues
                                                                                             Risks




                 Requirements Catalogue            3          1                  1            2       1
                  Technical Infrastructure         2          1                  2            1       1
                Configuration Management           1          3                  3            1       3
                    Test Management                2          3                  3            1       3
                    Data Stewardship               2          3                  1            1       3
                 Service Level Agreement           2          2                  2            2       1
                      Support Model                1          1                  2            1       3
                      Training Plans               1          1                  1            1       3
                        Schedules                  2          3                  2            1       2
                        Monitoring                 1          3                  3            1       1

                       1: Infrequently    2: Occasionally                    3: Frequently

This table can be used throughout the life cycle of the programme to prioritise governance
development to ensure that the most frequent changes are under the tightest controls.




        © 2007 Data Management & Warehousing                                                                  Page 9
                             White Paper - Data Warehouse Governance




What organisational structures are needed?
This paper has identified a large amount of potential change from different sources on
different processes on a long-term programme. It is necessary to have sufficient
organisational structure in place no control the changes and the communication of that
change. The basic model organisation for this can be described as follows:




Figure 1 - Data Warehouse Organisational Structure




     Executive Sponsor(s)
     An executive sponsor is a senior (executive) member of the organisation with an active
     interest and an understanding of the business uses of the data warehouse. A system of
     this magnitude and with such cross-functional reach needs to be sponsored at the very
     highest level. It is likely that the executive sponsor for the solution as a whole will be
     committed to the idea of developing better strategic information. A number of executive
     sponsors for individual functional developments will assist this person. The sponsors of
     individual functional developments are responsible for the delivery of the benefits
     promised for any given development.

     The executive sponsor(s) also need to ensure that the steering committee is directing
     the development in line with the vision, strategic direction and priorities for the
     organisation.




     © 2007 Data Management & Warehousing                                               Page 10
                        White Paper - Data Warehouse Governance




Steering Committee
The steering committee is the hub of the on-going data warehouse solution from a
management perspective. It is here that the development is aligned with the business
objectives. Monitoring ensures that the programme is delivering the right projects at the
right time and at fair value.

By setting the principles and policies the steering committee can control the direction
that the development goes in. This maintains an enterprise wide business perspective
for the data warehouse.

The steering committee is also the centre of communication. It takes input from the
user forums and the certification committee as to what is needed. In return the
committee manages the expectations of both the business and IT departments as to
what is possible. The steering committee is most effective when it is empowered to
make quick, effective, cross-functional decisions and communicate to all the
stakeholders.

Programme Management
Programme management is the co-ordinated management of a portfolio of projects to
achieve a set of business objectives. It delivers the co-ordinated support, planning,
prioritisation and monitoring of projects to meet changing business needs.

To achieve the business objectives the programme manager defines a series of
projects with quantifiable benefits that together will meet the long-term objectives of the
organisation. These projects may run simultaneously or at least overlap with each other
and they may share resources. Such projects might have some resources devoted to
the project for a period of time. In addition projects will require a range of specialists
available from the programme team whose services are used for shorter periods of
time. Projects within the programme can also be linked. Delays within one project will
then cause knock on effects in other projects due to logical links between tasks in both
of them. Resource and timing conflict resolution is also an integral part of the function
of programme management.

The programme is usually reflected in the management structure with a programme
manager to whom the project managers will report. The programme manager will be
concerned with recruiting and maintaining their project management teams, allocation
of key, shared resources and on the integration of the deliverables from each project
into the overall programme.

In this environment every project plays its part towards the organisation’s ultimate aims
and objectives. Often, as projects are completed, this translates back into a revised set
of corporate objectives.




© 2007 Data Management & Warehousing                                                Page 11
                             White Paper - Data Warehouse Governance




     Implementation Teams
     The implementation teams are those that will actually do the work. They will consist of
     a team of people (either for some or the entire project) that will fulfil the following roles:

         •   Project Manager
         •   Technical Architect
         •   Systems Administrator
         •   Network Administrator
         •   Database Administrator
         •   Metadata Administrator
         •   Data Modeller
         •   ETL Developer
         •   Front End Tool / Report Developer
         •   Product Specialist
         •   Test Manager
                                                                                           6
     Many organisations may have outsourced some or all of this functionality . The
     resources may have the skills and the time to fulfil more than one role, or for some
     projects more than one resource may be required to perform a given role. The
     management and resource levels must be determined by the project scope.

     Exploitation Teams
     Whilst the implementation teams focus on building the system the exploitation team are
     trying to ensure that the business is extracting the most value from the solution.
     Exploitation teams work on the current version of the system to help the business use
     the current system and develop new requirements to exploit the system further. Typical
     roles for the teams will include:

         •   Exploitation Team Leader
         •   Business Analyst
         •   Business Requirements Specialist
         •   Technical Author / Documentation Specialist
         •   Trainer
         •   End User Support Specialist
         •   Communications Specialist
         •   Helpdesk Support Specialist

     As with the implementation teams the resource level is determined by the project
     scope.

     User Forums
     The programme should also consider setting up a number of user forums that involve
     end users, subject matter specialists and staff from the exploitation teams. These
     forums are useful to allow various teams to express their issues and aspirations for the
     system and expand on the art of the possible. These forums should also appoint
     representatives on the steering committee to represent their perspective on the system.



6
   Data Warehouses benefit from people with detailed knowledge of the subject and
outsourced development to coding shops with no experience is often more costly than the
initial perceived savings. When choosing resources Data Management & Warehousing
recommend that organisations look at the skills of named individuals on the project


     © 2007 Data Management & Warehousing                                                  Page 12
                              White Paper - Data Warehouse Governance




      Certification Committee
      A number of groups within the organisation will also assess the data warehouse to
      ensure that it is fit for purpose. These groups can either be consulted individually or
      brought together as a committee to advise the programme. They include:

          •   Audit

              Most organisations will have some internal audit function. The audit function
              will need to ensure that the system meets the required standards and has
              sufficient checks and balances especially if the system is used for any form or
              statutory or fiscal reporting. The internal audit function should also examine
              how the system is managed. Where there is no (required) internal audit
              function an individual or team representing those with a stake in the quality of
              the information should perform the role of auditor.

          •   Regulatory Compliance

              Depending on the organisation and industry there may be a need to comply
              with requirements from government, local administration or industry regulators
              over the data that is held and who has access to that data. Current examples
                                          7
              include data protection law , financial management laws such as Sarbanes-
                        8           9                                       10
              Oxley Act and Mifid and telecoms regulators such as Ofcom .

          •   IT Strategy & Architecture

              Many large organisations have a team responsible for the strategy and
              architecture of all IT systems. This team ensures inter-operability, re-use and
              cost reduction of IT systems. The team will need to review and assess any
              technical infrastructure and changes to the solution that are proposed.

          •   Security

              Security of information is a growing concern for many organisations. Either a
              security team, if one exists in the organisation, or a nominated individual
              should review the system periodically to ensure its security.




7
  DPA Law website: http://www.dpalaw.info/
8
  Sarbanes Oxley Act: http://en.wikipedia.org/wiki/Sarbanes-Oxley_Act
9
  Mifid: http://www.fsa.gov.uk/Pages/About/What/International/EU/fsap/mifid/index.shtml
10
   Ofcom website: http://www.ofcom.org.uk/


      © 2007 Data Management & Warehousing                                              Page 13
                              White Paper - Data Warehouse Governance




Project Development Methodologies
There are four main methodologies for the development itself:

      Waterfall
      The waterfall model has the following phases that are followed in order:

          •    Requirements specification
          •    Design
          •    Build
          •    Integration
          •    Testing
          •    Deployment
          •    Maintenance

      To follow the waterfall model, one proceeds from one phase to the next in a purely
      sequential manner, starting each phase once the previous one has been completed.
      Phases of development in the waterfall model are thus discrete and there is no jumping
      back and forth or overlap between them, however there are practical variations on this
      model.

      Iterative11
      The basic idea behind iterative development is to develop a software system
      incrementally. This allows the developer to take advantage of what was being learned
      during the development of earlier, incremental, deliverable versions of the system.

      Learning comes from both the development and use of the system where possible. Key
      steps in the process are to start with a simple implementation of a subset of the
      software requirements. The system is then iteratively enhanced in a sequence of
      evolving versions until fully implemented. At each iteration design modifications are
      made and new functional capabilities are added.

      The process itself consists of the initialization step, the iteration step and the project
      control list. The initialization step creates a base version of the system. The goal for this
      initial implementation is to create a product to which the user can react. It should offer a
      sample of the key aspects of the problem and provide a solution that is simple enough
      to understand and implement easily.

      To guide the iteration process, a project control list is created that contains a record of
      all tasks that need to be performed. It includes such items as new features to be
      implemented and areas of redesign of the existing solution. The project control list is
      constantly being revised because of the analysis phase.

      The iteration step involves the redesign and implementation of a task from the project
      control list and the analysis of the current version of the system. The goal for the design
      and implementation of any iteration is to be simple, straightforward and modular,
      supporting redesign at that stage or as a task added to the project control list.




11
   Description is an edited form of the text from Wikipedia:
http://en.wikipedia.org/wiki/Iterative_development


      © 2007 Data Management & Warehousing                                                 Page 14
                               White Paper - Data Warehouse Governance



      The code can represent the major source of documentation of the system. The analysis
      of an iteration is based upon user feedback and the programme analysis facilities
      available. It involves analysis of the structure, modularity, usability, reliability, efficiency
      and achievement of goals. The project control list is modified in light of the analysis
      results.

      Agile12
      Agile methods are a family of development processes that build on iterative
      development, not a single approach to software development. Agile evolved in the mid
      1990s as part of a reaction against "heavyweight" methods such as the waterfall model
      of development that often became heavily regulated, regimented and micro-managed.
      The processes originating from this use of the waterfall model were seen as
      bureaucratic, slow, demeaning and inconsistent with the ways that software engineers
                                      13
      actually perform effective work.

      In 2001 seventeen prominent figures in the field of agile development (then called
      "light-weight methodologies") came together to discuss ways of creating software in a
      lighter, faster, more people-centric way. They created the Agile Manifesto and
                                     14
      accompanying agile principles:

          •   The highest priority is to satisfy the customer through early and continuous
              delivery of valuable software.

          •   Welcome changing requirements, even late in development. Agile processes
              harness change for the customer's competitive advantage.

          •   Deliver working software frequently, from a couple of weeks to a couple of
              months, with a preference to the shorter timescale.

          •   Business people and developers must work together daily throughout the
              project.

          •   Build projects around motivated individuals. Give them the environment and
              support they need and trust them to get the job done.

          •   The most efficient and effective method of conveying information to and within
              a development team is face-to-face conversation.

          •   Working software is the primary measure of progress.

          •   Agile processes promote sustainable development. The sponsors, developers
              and users should be able to maintain a constant pace indefinitely.

          •   Continuous attention to technical excellence and good design enhances agility.

          •   Simplicity--the art of maximizing the amount of work not done--is essential.

          •   The best architectures, requirements and designs emerge from self-organising
              teams.

          •   At regular intervals, the team reflects on how to become more effective, then
              tunes and adjusts its behaviour accordingly


12
   Taken from The Agile Manifesto (http://agilemanifesto.org/)
13
   Source: Wikipedia - http://en.wikipedia.org/wiki/Agile_software_development
14
   Source: Wikipedia - http://en.wikipedia.org/wiki/Agile_software_development


      © 2007 Data Management & Warehousing                                                    Page 15
                              White Paper - Data Warehouse Governance




      Cowboy Coding15
      Cowboy coding is a form of software development method without an actual defined
      method – team members do whatever they feel is right. Typical cowboy coding will
      involve no initial definition of the purpose or scope of the project, no formal description
                                                                 16
      of the project and will often involve only one developer . The developer will often be
      working from his own idea of what the software should do. It is often characterised by a
      lack of any documentation for either the requirements of the project or the design of the
      software overall.


Within a data warehouse programme it is strongly advised never to allow the cowboy
methodology to appear, however any of the other three can be used. There are two deciding
factors as to the approach that will be used. The first is the approach that is already being
used within the organisation and this may dominate over all other factors. The second is the
point in the lifecycle of the warehouse. In the early part of the programme projects using agile
methodologies offer fast development and quick delivery for new applications built by expert
teams that engage the users. Towards the end of the lifecycle a more waterfall like approach
to maintenance and de-commissioning operations run by systems management teams is
likely to be more successful for individual projects. This highlights the need for the
governance itself to be adaptive to the changing environment.

All formal methodologies (lightweight and heavyweight) can still lead to failure as the
methodology team members attempt to operate within the social/political environments of the
organisation. The probability of such a failure is related to the degree of processes inhibiting
the user(s) from deviating from the organisation standard on one hand and the cost in terms
of lost efficiency and lost creativity of implementing such processes on the other. Success is
therefore reliant on the management choosing appropriate processes that find the correct
balance between these two competing aspects.




15
  Cowboy coding definition: Wikipedia: http://en.wikipedia.org/wiki/Cowboy_coding
16
  Cowboy coding can occur even within projects that are using more appropriate
development methods. Warning signals that cowboy coding may be occurring include:
   • Secrecy about what a developer is working on.
   • The inability to describe the functionality of current development.
   • The tendency to do lots of quick fixes.
   • Working hard and not working clever, working without prioritisation.


      © 2007 Data Management & Warehousing                                                Page 16
                              White Paper - Data Warehouse Governance




Best practices
The following items are some of the key best practices that should be implemented:

      Momentum
      Get the project going fast and then keep it moving by keeping the scope and deadlines
      for delivery very tight. This avoids analysis paralysis and ensures that there is space to
      take corrective actions if necessary.


      Monitor and Measure17
      Large long-term projects are difficult but if the management team cannot see what is
      happening then they do not know when things are going wrong. Ask questions about
      whether teams deliver on time, to budget and scope. Track the number of issues raised
      in development, test and production, monitor the service level agreements, design
      metrics for the quality of the code and the user satisfaction, etc. Much of this is built in
      to approaches such as agile.


      Prioritise
      Everyone will want their component now, but it cannot all be done at once. Have a
      clear, transparent process for deciding the priories and then stick to them. One useful
      technique is to say that (provided the project uses small scopes) once started a phase
      cannot be stopped but that before the next phase is started all priorities can be re-
      assessed. This forces low value priorities to the back but rapidly brings higher priorities
      to the fore. The steering committee should be responsible for these priorities and the
      project methodology should support the prioritisation process.


      Light touch
      This document has covered many aspects of governance however the programme
      should implement the minimum necessary because otherwise the project will become
      about governance and not delivery. Review the processes regularly to add process
      where required and do not be afraid to remove un-necessary process.


      Communication
      The data warehouse is a long-term project for which the perception will shift over time
      from all parts of the business. Clear communications will help people understand what
      is available for them, what the priorities are going forward and how to access the
      information. Publish these widely to encourage involvement and interaction with the
      system.



17
  A useful book on this subject is “Controlling Software Projects: Management, Measurement
and Estimates” by Tom DeMarco



      © 2007 Data Management & Warehousing                                                Page 17
                             White Paper - Data Warehouse Governance




     Standards
     Develop and enforce clear standards for all aspects of the warehouse such as naming
     conventions for code and documents, data ownership, data cleaning, access to data,
                                      18
     service levels, performance etc.


     Track processes
     Ensure that there are clear formal processes for handling change requests, risks and
     issues. These are the aspects of the project that cause divergence from the baseline
     and therefore it is critical to know what is happening, when and why. It allows scarce
     resource to be focused on the correct actions. Use key design decision templates to
     ensure that design topics are not constantly revisited. Ensure that all change related
     processes have powerful change management routings to back them up.


     Understand cost and value
     Define methods for determining the cost and value of any action, some apparently low
     cost (often called quick-win or tactical) solutions deliver little value and their true cost is
     disproportionately higher as they have to be backed out after a short life. Do not forget
     to include end of life cost in the calculations and make budget holders responsible for
     demonstrating the benefit that has been derived because of the effort.


     Continuous Learning and Improvement
     Carry out stage point audits and post-mortems on all aspects of the system, use the
     lessons learnt from these to improve the processes to ensure that the next
     development will not make the same mistakes.


     Pilot & Prototype
     Pilot and prototype high risk or complex aspects of the development using
     methodologies like ‘Agile’ but ensure that the pilots and prototypes do not go into
     production until the normal quality thresholds have been met.


     Authority to act
     Ensure that the steering committee has the authority to act cross functionally using
     sufficient information to make good decisions. A steering committee that wants a
     change to a source system that has a critical effect on the data quality is ineffective if
     they are told that it will be put in a queue and should be delivered in two years time. In
     the mean time is it acceptable that the data warehouse will just have to make do with
     poor data? Conversely a lack of sufficient information might be making the same
     demand of a system that is about to be de-commissioned or for data for which the
     business user cannot articulate a use.



18
   See the Data Management & Warehousing Documentation Roadmap at
http://www.datamgmt.com


     © 2007 Data Management & Warehousing                                                   Page 18
                                White Paper - Data Warehouse Governance




        Plan for the long term
        Ensure that the business and users understand that whilst aspects of the data
        warehouse will be available quickly the system is a long-term commitment. It will need
        funding and maintenance across its entire life span. During that lifetime the system will
        also grow in size and complexity, increasing the financial burden.



Governance and Success
Governance is the control aspect of management and should promote a situation where
senior management are in control of the situation much like the maestro conductor of a fine
orchestra. They need to understand what is going on and where to go for the next action and
how to respond to a changing situation.

Sometimes the reality is that managers are working in a suppressed panic, not believing what
people are telling them or what they themselves are communicating to their management.
When managers know that people have to work outside the process to get things done and
not knowing what is actually going on then governance has failed.

Governance is therefore not the documentation, the processes or the formality, but is about
developing a culture where understanding, discipline and skill are regarded as virtues in
teams that have leaders with strong technical skills, initiative, communications skills and
personal authority.

Organisations that ‘buy the book’, be it from a bookshop or as a formal methodology from a
vendor and follow it rigidly are guaranteed to achieve the lowest common denominator
solution, one that checks all the boxes but underwhelms the management and disappoints the
users. Those organisations that use governance and methodologies as enablers and allow
systems to fulfil their potential by meeting the constantly changing requirements of the users
succeed.

Creating effective governance for an organisation requires imagination:

“What we need is imagination, but imagination in a terrible strait-jacket. We have to find a new
      view of the world that has to agree with everything that is known, but disagree in its
 predictions somewhere .... And in that disagreement it must agree with nature. If you can find
   any other view of the world which agrees over the entire range where things have already
  been observed, but disagrees somewhere else, you have made a great discovery. ...A new
                                                                                         19
             idea is extremely difficult to think of. It takes a fantastic imagination.”




19
     Richard Feynman, The Character of Physical Law, 1965, Chapter 7, "Seeking New Laws.”


        © 2007 Data Management & Warehousing                                              Page 19
                                 White Paper - Data Warehouse Governance




Summary
An organisation that is embarking on a data warehousing project is undertaking a long-term
development and maintenance programme of a computer system. This system will cost a
significant amount of money and should be well controlled. Governance defines the model
that the organisation will use to ensure optimal use and re-use of the data warehouse. It will
also enforce corporate policies (e.g. business design, technical design and application
security) that ultimately derive value for money.

This paper has identified five sources of change to the system:

     •     Demand
     •     External Change
     •     Maintenance
     •     Risks
     •     Issues

It has also looked at the aspects of the system that they will affect:

     •     Requirements Catalogue
     •     Technical Infrastructure
     •     Configuration Management
     •     Test Management
     •     Data Stewardship
     •     Service Level Agreement
     •     Support Model
     •     Training Plans
     •     Schedules
     •     Monitoring

Using these it has been able to highlight standards and best practices that should be
employed by the organisation to deliver effective governance. The paper has also highlighted
the fact that because governance is about fitting into the existing organisation. It also
highlights the fact that the system grows significantly over time. Consequently there is no
single right solution for governance but instead there are ways of working and adapting over
time that will ensure success.




         © 2007 Data Management & Warehousing                                          Page 20
                                White Paper - Data Warehouse Governance




Appendices
        Appendix 1 - Programme or Project?
        This document has several times differentiated between programmes and projects. The
                   20
        table below outlines the differing characteristics of the two:


        Programmes:                                  Projects:
        Address the entire business change           Deliver a specific change component
        Focus on strategic goals                     Focus on tactical delivery
        May have imprecise definition                Have a precise objective
        May have uncertain timing                    Are defined with a specific timeline
                                                     and budget
        Evolve over a period of time to derive       Try to avoid change to the defined
        optimum benefit for the organisation         scope in order to ensure delivery
        Require much senior management               Require management communication
        attention, often including strategic and     primarily at an operational level
        political debate across organisational       concerning operational details
        boundaries
        Produce an overall improvement in the        Produce      specific     pre-defined
        business that may be multi-faceted and       deliverables
        not fully defined at the outset of the
        programme
        Require a manager who is high-               Require a manager who pays attention
        powered, high-level, visionary, strategic,   to detail, has good team leadership,
        political, sales-oriented and works with     plans in detail, follows a disciplined
        people at the top and across the             approach and delivers the goods.
        organisation




20
     The ePMBook by Simon Wallace (http://www.epmbook.com/)


        © 2007 Data Management & Warehousing                                              Page 21
                        White Paper - Data Warehouse Governance




Appendix 2 – The "Declaration of Interdependence"
Throughout this document reference has been made to the agile development
methodology. Because of the development of agile, a set of principles characterised by
statements “We accomplish X by doing Y.” was developed for project managers. These
principles are of value outside the agile methodology itself and should be used by
project managers in any data warehouse programme.

The “Declaration of Interdependence” for modern (agile/adaptive) (product/project)
           21
management

"We ...

     •    increase return on investment by making continuous flow of value our focus.
     •    deliver reliable results by engaging customers in frequent interactions and
          shared ownership.
     •    expect uncertainty and manage for it through iterations, anticipation and
          adaptation.
     •    unleash creativity and innovation by recognizing that individuals are the
          ultimate source of value and creating an environment where they can make a
          difference.
     •    boost performance through group accountability for results and shared
          responsibility for team effectiveness.
     •    improve effectiveness and reliability through situationally specific strategies,
          processes and practices."




21
   2005 David Anderson, Sanjiv Augustine, Christopher Avery, Alistair Cockburn, Mike
Cohn, Doug DeCarlo, Donna Fitzgerald, Jim Highsmith, Ole Jepsen, Lowell Lindstrom,
Todd Little, Kent McDonald, Pollyanna Pixton, Preston Smith and Robert Wysocki:
http://alistair.cockburn.us/index.php/The_declaration_of_interdependence_for_modern
_management


© 2007 Data Management & Warehousing                                               Page 22
                         White Paper - Data Warehouse Governance




Appendix 3 - Team Values and Principles
The author of this white paper used to work at Sequent Computer Systems. The
company had a set of values and principles that the author feels have always proved
useful to teams building data warehouses by creating a culture in which governance
can work.

      Our Values:
           •   Easy to Do Business With
                  o The Customer ALWAYS Comes First
                  o Do All We Can to Ensure Customer Success - Does Not Always
                       Mean Saying Yes
                  o Three Keys -- Listen, Consider and Act
                  o Go the "Extra Mile" for the Customer

           •   Profitability
                   o Required for Our Success
                   o Allows for Growth & Investments for Future
                   o Everyone is Responsible for Profitability
                   o Follow the "Spend Smart and Invest Smart" Concept

           •   Teamwork
                  o Strong & Balanced Teams
                  o Put Group Goals & Interests Ahead of Personal Reward
                  o Open & Honest Communication
                  o You're Accountable for Your Commitments
                  o Nobody Wins Unless the Team Wins

           •   Quality
                  o      Exceed the Customer Expectations
                  o      Strive for Continuous Improvement in All You Do
                  o      Know Who Your Customers Are
                  o      Establish Clear Mutually Acceptable Expectations
                  o      Managed Expectations, Consistently Achieved

           •   People
                  o      Assimilating & Integrating New Members is Critical
                  o      Open, Honest & Timely Communication
                  o      Treat Each Other With Respect
                  o      Take Time to Say "Thank You" for a Good Job
                  o      Strive for Win-Win Relationships


      Our Principles:
          •    Act With Honesty & Uncompromising Integrity
          •    Take Responsibility for Our Customers' Success
          •    Strive to be the Best
          •    Have a "Can Do" Attitude
          •    Respect Each Other
          •    Exhibit Team Pride
          •    Take Calculated Risks
          •    Be Active Community Citizens
          •    Urgently Do the Right Thing
          •    Make Consultative Decisions




© 2007 Data Management & Warehousing                                          Page 23
                              White Paper - Data Warehouse Governance




References
The section below represents some useful resources for those considering building a data
warehouse solution.

      Web resources

Organisation                                                        Website
Data Management & Warehousing                                       http://www.datamgmt.com

DM Review –                                                         http://www.dmreview.com/
Data Warehouse Project Management
The Data Warehouse Institute –                                      http://www.tdwi.org/
Data Warehouse Governance
Data warehouse governance –                                         http://www.amazon.com/
Best practices at Blue Cross & Blue Shield of North Carolina
Data Warehouse Project Management                                   http://www.amazon.com/
The Seven Pillars of Data Warehouse Governance                      http://www.itweb.co.za/
The ePM Book                                                        http://www.epmbook.com/
Software Testing Resources                                          http://www.testingfaqs.org/
From Here to Agility: The Physics of Speed                          http://www.stickyminds.com/
Agile Manifesto                                                     http://agilemanifesto.org/
Agile Alliance                                                      http://www.agilealliance.org/
e-Programme                                                         http://www.e-programme.com/
Data Protection Law                                                 http://www.dpalaw.info/
Alistair Cockburn                                                   http://alistair.cockburn.us/

Wikipedia                                                           http://en.wikipedia.com



Acknowledgements
The author and Data Management & Warehousing would like to acknowledge and thank all
the clients and colleagues who have taken the time to review and comment on this document
prior to publication. Special thanks to Ron Ballard for the most detailed of reviews and proof
reading.



Copyright
© 2007 Data Management & Warehousing. All rights reserved. Reproduction not permitted
without written authorisation. References to other companies and their products use
trademarks owned by the respective companies and are for reference purposes only.




      © 2007 Data Management & Warehousing                                                 Page 24

				
DOCUMENT INFO
Shared By:
Tags:
Stats:
views:41
posted:4/24/2012
language:English
pages:24