Chief Skeptical Officer – A
New C-Level Role
Alan McSweeney
Why Does This Happen?
Not What Is
Wanted Or
Too Late What Was
The Big Idea Required/
Envisaged
Reduced
Functionality
Lack Of
Requiring
Integration
Manual
Workarounds
And Continue To
The Big Failure
Happen Time and
Again? Requires
Too Expensive Rework Or
Replacement
And What Can Be
Done to Stop It? High Cost Of
Not Scalable
And/Or Poor
Operation
Performance
October 18, 2011 2
Spectrum Of Failures
Performance and/or Specified
Operational Business Benefits
Problems and Savings Not Functionality
Delivered Delivered Does not
More Expensive Meet Business
to Operate Than Requirements
Planned
Significant
Rework
Project Late
Required
and/or Over
Budget
Solution Largely
Unused And/Or
Unusable
Complete Project
Success: Increasing Frequency/ Complete Project
On-time, On-budget
and Delivering
Probability of Failure: Cancelled,
Unused, Rejected
Specified Benefits Occurrence
October 18, 2011 3
What Is The Role Of The Chief Skeptical Officer?
• Role with end-to-end view and responsibility from idea
definition to delivery
• Provide positive skepticism around idea definition and
solution specification
• Be an independent voice
• Be an advocate for and champion of standards
• Maintain a repository of learning and standards
• Embed learning into the organisation
October 18, 2011 4
The Key Functions Of The Chief Skeptical Officer
Remove Arbitrariness,
Provide Independent Relativity And
Oversight of Decisions Subjectivity
Require Assertions To
Be Well Supported By
Evidence
Ensure Rational
Decision-Making
Avoid Systematic
Distortion and
Identify and Eliminate Misrepresentation
Behavioural And Other Ensure Evidence-Based
Biases On Decision- Judgements
Making Question Assumptions,
Attitudes Of
Track End-To-End View Knowledge, Opinions
From Concept To and Beliefs Stated As
Delivery Facts
October 18, 2011 5
The Journey From Idea To Successful Operation
Idea, Business
Need, Business
Benefits
Process
Definition and
Solution Design
Costing -
Implementation
and Operational
Solution
Implementation
and Delivery
Solution
Operation
Benefits Realised
October 18, 2011 6
The Journey From Idea To Successful Operation
Idea, Business
Need, Business
Benefits
But Promised
Process
Definition and
Benefits All Too
Solution Design Frequently Not
Costing - Delivered
Implementation
and Operational
Solution
Implementation
and Delivery
Solution
Operation
Benefits Realised
October 18, 2011 7
Frequently Too Many Handoffs On The Journey
From Idea To Successful Operation
Idea, Business Handoff and
Need, Business
Benefits Information Loss
Process Handoff and
Definition and
Solution Design Information Loss
Costing - Handoff and
Implementation Information Loss
and Operational
“Chinese Whisper” Solution Handoff and
Implementation Information Loss
Effect as Initial and Delivery
Concept Moves to
Implementation Handoff and
Solution
Operation Information Loss
Benefits Realised
October 18, 2011 8
What Causes the Disconnect From Idea To Delivery
Mi
sco
Idea, Business mm Too
Need, Business u n Ma
Benefits
ica ny
tio Ha
n, nd
Iss o
Process To ues ffs B
-En Le etw
Definition and d V ft U ee
Solution Design iew nre n S
or solv tage
En ed s,
Costing - d-T , A To
Implementation o-E ssu o O
and Operational
nd mp ften
Re tio Se
sp ns pa
on M ra
Solution sib ad te
ilit e, N Te
Implementation y o O ams
and Delivery ne ,
W
ith
Solution En
d-
Operation
Benefits Realised
October 18, 2011 9
What Can Go Wrong Along The Journey From Idea
To Successful Operation
Poorly Defined Need,
Benefits Exaggerated
Idea, Business
Need, Business
Benefits Inadequate Analysis,
Poor Or Incomplete
Process Design
Definition and
Solution Design Overly Optimistic Initial
and Operational Cost
Costing - Estimates
Implementation
and Operational
Poor Requirements Promised/Expected
Analysis, Poor Design, Solution
Implementation Benefits Not Delivered
Poor Management
and Delivery
Slow, Unreliable, Solution
Expensive To Operate, Operation
Not Integrated,
Requires Manual
Workarounds
Benefits Realised
October 18, 2011 10
Cost of Fixing Errors During Project Lifecycle
Errors/ gaps/
omissions become
significantly more Benefits Realised
expensive to fix at
later stages of the
solution Solution
Operation
Solution
Implementation
and Delivery
Costing -
Implementation
and Operational
Process
Definition and
Solution Design
Relative cost to remediate
Idea, Business errors at the end can be 50-
Need, Business 100 (or more) times more
Benefits
expensive than at the start
October 18, 2011 11
Role Of The Chief Skeptical Officer Is To Take End-To-End
View, Enforce Discipline and Perform Reviews
Idea, Business
Need, Business
Benefits Getting It Right
Here …
Process
Definition and
Solution Design … Reduces/
Ch Avoids Problems
ief
En Ske Costing - Here
su pt
re ica Implementation
Th l O and Operational
at
Th ffice
e S r’s
tar Ro Solution
t D le Implementation
eli Is T and Delivery
ve o
rs Ta
An ke
d C En
an d-T Solution
De o-E Operation
liv nd
er
Th View
eE T
nd o Benefits Realised
October 18, 2011 12
What Is The Chief Skeptical Officer Trying To Protect
You Against?
• Cognitive Bias – Poor or inaccurate judgements, illogical
interpretations and decisions, characterised by patterns of
behaviour
• Strategic Misrepresentation – Deliberate misrepresentation in
budgeting caused by distorted incentives
• Planning Fallacy – Systematic tendency to underestimate how long
it will take to complete a task even when there is past experience of
similar tasks over-running
• Optimism Bias – Systematic tendency to be overly optimistic about
the outcome of actions
• Focalism – Systematic tendency to become inwardly focussed and to
lose situational awareness and appreciation of wider context during
times of stress
October 18, 2011 13
Cognitive Bias - Types
• Many classifications and types of cognitive bias
• Can be very difficult to avoid because of their embedded
nature and emotional/irrational basis
− Decision-Making and Behavioural Biases - affecting belief
formation and business decisions
− Probability and Belief Biases - affecting way in which information
is gathered and assessed
− Attributional Biases - affecting the determination what was
responsible for an event or action
• Cognitive biases are very real and can have damaging
effects
October 18, 2011 14
Decision-Making And Behavioural Biases
• Relying too heavily on one piece of • Placing too much importance on one
information when making a decision - aspect - Focusing Effect
Anchoring • Looking to reduce a small risk to zero
• Believing things because many others rather than a greater reduction of a larger
believe the same – Bandwagon risk – Zero-Risk Bias
• Assigning greater weight to apparently • Rejecting new evidence that contradicts
dominant factors – Attention Bias an established paradigm – Semmelweis
• Interpreting information so as to that Effect
confirms preconceptions – Confirmation • Making decisions based to what is pleasing
• Seeing oneself as less biased than others – to imagine instead basing decisions on
Blind Spot evidence and rationality – Wishful
Thinking
• Strong preference for immediate payoffs • Assigning a higher value to disposal/loss
relative to later ones – Hyperbolic compared with cost of acquisition – Sunk
Discounting Cost Effect
• Greater preferences just because of • Viewing a harmful action as worse than an
familiarity - Exposure Effect equally harmful omission or inaction –
• Paying more attention and giving more Omission Bias
weight to the negative rather than the • Justifying increased investment based on
positive – Negativity Bias the cumulative prior investment despite
• Looking for information even when it new evidence suggesting that the decision
cannot affect action – Information Bias was wrong - Irrational Escalation
October 18, 2011 15
Probability And Belief Biases
• Excessive or inflated belief one's • Perceiving patterns where none exist
performance, ability – Overconfidence – Clustering
Effect • Selecting an options for which the
• Belief gaining plausibility through probability of a favorable outcome is
increasing repetition - Availability known over an option for which the
Cascade probability of a favorable outcome is
• Assigning greater weight to initial or unknown - Ambiguity Effect
recent events more than subsequent • Considering information to be correct
or later events – Serial Position Effect if it has any personal meaning or
• Assigning a lower probability to the significance – Subjective Validation
whole than the probabilities of the • Overestimating the likelihood of
parts – Subadditivity Effect positive rather than negative
• Avoidance of risk or the negative by outcomes – Valence Effect
pretending they do not exist – Ostrich • Failure to examine all possible
Effect outcomes when making a judgment –
• Judging future events in a more Attentional Bias
positive light than is warranted by
actual experience – Optimism Bias
October 18, 2011 16
Attributional Biases
• Where skilled underrate their abilities and unskilled
overrate their abilities – Dunning–Kruger Effect
• Defending the status quo – System Justification
• Overestimation of agreement – False Consensus Effect
October 18, 2011 17
Strategic Misrepresentation
• Deliberate misrepresentation in planning and budgeting caused by
distorted incentives
• Response to how organisations structure rewards and give rise to
motivations
• Systematic (and predictable) misrepresentation
− Deliberately understand costs to gain acceptance with understanding that
costs will increase
− Not willing to face reality of high costs
− Overstatement or understatement of requirements
− Competition for scarce funds or jockeying for position
− Inclusion of ideology into planning
• Underlying system and processes need redesign to eliminate
October 18, 2011 18
What Are The Key Disciplines The Chief Skeptical
Needs To Promote
• Disciplines and Skills
− Benefits Management and Realisation
− Capital Planning and Investment Management
− Business and Process Analysis
− Solution and Process Analysis and Architecture Design
• Practices
− Reference Class Forecasting
− Audits and Checklists
October 18, 2011 19
Reference Class Forecasting
• Technique to improve accuracy in plans and projections by
basing them on actual performance in a reference class of
comparable actions
• Based on knowledge about actual performance
− Analyse distributions and probability of cost overruns
− Compare the proposed project with the reference class
distribution to establish the most likely outcome
• Used to avoid Planning Fallacy, Strategic
Misrepresentation and Optimism Bias
October 18, 2011 20
Benefits Management And Realisation
• To increase the likelihood of success from IT and other investments,
organisations must identify the different causes of benefits before
developing any project implementation plan
• Types of IT projects with very different approaches to benefits
management
− Fixing or improving something that exists
• Resolve problem
• Improve integration
− Implementing a new initiative
• New system
• New processes
• Need to focus on the changes needed to achieve results and take full
advantage of new facilities offered rather than on just IT features
• Effective change management is crucial to achieving benefits
October 18, 2011 21
Achieving Benefits – Some Questions To Ask
• Why is there a need to improve?
• What improvements are needed? What improvements are possible or achievable?
Have the improvements been agreed by all stakeholders?
• What benefits will be realised by each stakeholder if the business objectives are
achieved? How can each benefit be measured?
• Who owns each of the benefits and will be accountable for its delivery?
• What business changes are needed to achieve each benefit? Have the explicit links
between each benefits and required business changes been identified?
• Who will be responsible for ensuring the business changes are made successfully?
• How and when can the changes be made? Who will make the changes? Does the
business have the ability and capacity to make the changes?
October 18, 2011 22
Benefits Management And Benefits Dependency
Network
• Benefits Dependency Network (BDN) is an approach to linking:
− Information Technology Enablers, Changes and New Capabilities – enabling
technologies and functions and facilities needed to support the realisation of
the identified benefits and to allow the necessary changes to be undertaken
− Business Changes – business activities and new ways of working that are
required to ensure that the desired benefits are realised
− Enabling Changes - prerequisites for achieving the business changes or that are
essential to bring the new system into effective operation
− Business Objectives - high level priorities in relation to the drivers, outcomes
and improvements to be delivered on completion of the project
− Business Benefits - outcomes of a change that are deemed to be positive by a
stakeholder and that are valuable to the organisation and are measureable
− Stakeholders - individuals or groups who will benefit from the project and are
either affected by or directly involved in making the changes needed to realise
the benefits
October 18, 2011 23
Types Of Changes
• Business Changes - Permanent changes to working
practices, processes, procedures, interactions and
relationships that will cause the benefits to be delivered
− Generally need new IT system to be in place
− May require enabling changes to be implemented
• Enabling Changes - Typically one-time changes that are
pre-requisites for making the business changes or are
necessary to bring the new system into effective operation
− Can be made in advance of system implementation
− Training
− Processes redesign
− New work practices
− Changes to job roles and responsibilities
October 18, 2011 24
Benefits Dependency Network
Provide for
Require Allow Enable Delivery of
Information
Technology
Enablers, Enabling Business Business Business
Changes and Changes Changes Benefits Objectives
New
Capabilities
• BDN provides a framework for explicitly linking the overall investment objectives
and the requisite benefits with the business changes which are necessary to
deliver those benefits and the essential IT functionality to both drive and enable
these changes to be made.
• BDN forms part of the benefits realisation plan
• Helps keep the focus on benefits realisation during the program execution
• Allows variations of the project or program to be assessed for their impact on
benefits realisation
October 18, 2011 25
Benefits Dependency Network
Means to Ways to Achieve Changes Results of Changes
Achieve
Changes
Information
Technology
Enablers, Enabling Business Business Business
Changes and Changes Changes Benefits Objectives
New
Capabilities
October 18, 2011 26
Benefits Dependency Network
Understand Why the Business Will Use the Information Technology System
What What the Why the
What Users What
Information Business Business
Do With Information
Technology Gains From Uses The
Information Technology
Capabilities Information Information
Technology Capabilities
and Features Technology Technology
Capabilities Achieve
Offer Capabilities Capabilities
Understand What Information Technology System is Needed to Deliver on Requirements
What What the
Information What the What the What the
Business
Technology Business Has Business Business
Needs to
Capabilities to Do Needs To See Wants
Provide
Are Required
October 18, 2011 27
Benefits Dependency Network
• Shows the link from the solution, through business activities,
outcomes and benefits to the organisation's overall drivers
• Used to confirm that the solution being introduced will actually
provide the results you are seeking
• Any function within the proposed solution that is not linked to a
benefit is potentially of doubtful value
• Functions with many link or links to key benefits can be identified
and given extra attention
• BDN is a complex approach that requires benefit identification,
assessment, validation and realisation maturity within the
organisation
• Imposes a rigour on the organisation in analysing benefits from
projects
October 18, 2011 28
Capital Planning And IT And Other Investment Core
Requirements
• Determine the scale, scope, and sources of funding for IT
and other areas
• Assign financial resources to competing activities within
the IT portfolio
• Establish a balance between capital expenditure (new
projects) and operating expenditure (running systems
delivered by past projects)
• Optimise the total cost of ownership
• Manage IT and other portfolios for value and not just cost
• IT and other areas need to implement a process for
justifying its costs and be seen to be taking these steps
October 18, 2011 29
Capital Planning And IT and Other Investment
Management
• Aligns IT and other investments to organisation strategy (scoring)
• Prioritises investments (ranking)
• Provides strategic criteria for investment analysis
• Conduct annual IT and other portfolio management reviews
• Provides recommendation to stop, slow, maintain or accelerate
program funding
• Identifies redundant/inefficient systems
• Integrates IT and other architectures within investments
• Ensures compliance with funding standards
October 18, 2011 30
Characteristics Of Credible Cost Estimates
• Clear identification of requirements of the ultimate deliverable
• Broad participation in preparing estimates
• Availability of valid data for performing estimates – historical,
experience, benchmarks
• Standardised and comprehensive estimate structure that includes all
possible sources of cost
• Provision for uncertainties – include known costs explicitly and allow
for unknown costs
• Recognition of inflation
• Recognition of excluded costs
• Independent review of estimates for completeness and realism
• Revision of estimates for significant changes in requirements
October 18, 2011 31
Challenges Of Developing Good Cost Estimates
• Requires detailed, stable, agreed requirements
• Agreed assumptions
• Access to detailed documentation and historical data for
comparison
• Trained and experienced analysts
• Risk and uncertainty analysis
• Identification of a range of confidence levels
• Adequate contingency and management reserves
October 18, 2011 32
Reasons for Good And Bad Cost Estimates
Ineffe
and U ctive Risk
Effect ncer
i ve
Unce Risk and Unfa Analy tainty
r ta Techn miliar s is
Ident
ificat Analy inty First- ology or
s is
Rang ion of a Time
Use
Probl
em
Confi e of De Acces s Getting
dence
Level Docu tailed s to D
m ata Unre
s and H entation Unre
Proje asonable
Adeq
ua istor Train Unre alistic or ct Bas
Conti
ngen te Data ical e
Exper d and
liable
Data Unre elin e
Mana cy an ien
geme d Detai
led, S Analy ced Assumalistic
Reser nt sts No o ption
ves Agree table, Comp r Limited s Overo
Requ d ariso ptimi
ireme Agr Avail n Data sm
n ts
Assum eed able New
ption Pr ocess
s e s Untra
Proje Inexp ined and
ct Ins er ie
t abilit
y Comp Analy nced
le sts
or Te x Project
chnol Unre
ogy alistic
Savin Project
gs
• Lost of reasons for and causes of inaccurate cost estimates
October 18, 2011 33
Sources of Risk and Uncertainty In Estimating Costs
• Lack of understanding of the project requirements
• Shortcomings of human language and differing
interpretations of meaning of project
• Behaviour of parties involved in the cost estimation
process
• Haste
• Deception
• Poor cost estimating and pricing practices
October 18, 2011 34
IT Investment
• IT is commonly seen as failing to deliver value for money
• Benefits and value must be actively managed for
• Realising and assessing business benefits from IT-enabled investments involves
more than simply assessing Total Cost of Ownership for IT-related projects and
managing the IT budget
• Key requirements
− Ability to get lifetime costs right
− Ability to define benefits correctly and effectively
− Ability to manage the benefits management process
− Ability to increase and sustain benefits management maturity
• Use existing methodologies and frameworks to implement key requirements
quickly
− ITIM
− Benefits Dependency Network
− ValIT
− Organisational change and commitment
• Effective benefits management enables organisations to clearly and consistently
articulate IT’s contribution to achievement of business objectives
October 18, 2011 35
Business and Process Analysis And Design Builds
Bridge From Business To Solution
Problem
Requirement Business Analysis:
Elicit Requirements
Current State Analyse
Communicate
Business Analysis Validate
and Solution
Design
Solution Design:
Translate
Requirements into Solution
Solution
Desired Future
State
• Business analysis is a key driver of business value
• Solution delivery start with business analysis
October 18, 2011 36
Weaknesses In Business Analysis Capabilities And
Competencies At The Root of Many Project Failures
Poor Size/Capacity/ Poor Strategic
Requirements Complexity Alignment
Poor Analysis Large Project, Inadequate Business
Practices Complex, Difficult Case, Undefined
Changes and Processes Problem/Need
Business Requirements Large Project Team and Business Benefits Business Needs
Not Captured Multiple Stakeholders Not Measured Not Met
Opportunities
Lost
Investment
Inadequate Resource Unproven Technology Inadequately Explored Wasted
Allocation and Solution Options
Prioritisation
Inadequate Business Dynamic, Changing Solution Design Not
Involvement Environment Aligned to Business Needs
Poor Focus on Uncertainly/ Poor Solution
Business Needs Ambiguity Design
October 18, 2011 37
Analysis-Related Causes Of Failures
Poor Size/Capacity/ Poor Strategic
Requirements Complexity Alignment
Poor Analysis Large Project, Inadequate Business
Practices Complex, Difficult Case, Undefined
Changes and Processes Problem/Need
Business Requirements Large Project Team and Business Benefits Business Needs
Not Captured Multiple Stakeholders Not Measured Not Met
Opportunities
Lost
Investment
Inadequate Resource Unproven Technology Inadequately Explored Wasted
Allocation and Solution Options
Prioritisation
Inadequate Business Dynamic, Changing Solution Design Not
Involvement Environment Aligned to Business Needs
Poor Focus on Uncertainly/ Poor Solution
Business Needs Ambiguity Design
October 18, 2011 38
Taking A Complete View of Systems And Processes Is
Essential to Solution Success
• Overall solution operates with a mix of automated and
manual processes in a structured or ad-hoc manner o
deliver the required results
• Understanding the overall set of processes and their
operation is crucial to successful results
• Need to see the entire picture to understand how a
solution should operate
− Systems/applications are just one part of this universe
• Unambiguous definition of processes is required
• Processes that are to be automated define the scope of
the development and implementation work
October 18, 2011 39
Complete View of Systems and Processes Is Essential
to Solution Success
External External
Manual Manual
Interaction Interaction
External Automated External
Component Process Component
Manual Manual
Process Process
System System
Component Component
Automated Automated
Process System Process
Manual Manual
Process Component Process
External External External
Manual Component Manual
Interaction Interaction
October 18, 2011 40
Solution Design and Implementation Sequence
Defines where the business
wants to go
Business Plan
Business need identifies
solutions that will allow
Business Need delivery of plan
Defines the benefits to be
Business Benefits achieved by the solution
Defines the detailed requirements
Requirements
of the solution
Definition
Process Design
Defines the processes that will be
implemented by the solution
Solution Architecture
and Design
Defines the solution design to
implement the processes
Technical and Detailed
Design
Creates a detailed technical design
for implementation
Implementation
Implements the detailed design
October 18, 2011 41
Consistent Approach to Business And Process
Analysis
• Adopt a consistent and robust framework in business analysis
− Enables effective benefits realisation through a solution which meets the
business need
• Key elements
− Establish enterprise standards, procedures and governance
− Standardise on infrastructure, analysis methods and operational procedures
− Develop competence and skills
• Key benefits
− Implement solutions that meet business needs
− Increase the ability of the business to adapt quickly to changes
− Reduce risk, complexity, redundancy and support complexity
− Align business and IT
− Enable re-use and faster time-to-market
− Present one face to the business (customer)
− Increase business value
October 18, 2011 42
More Information
Alan McSweeney
alan@alanmcsweeney.com
October 18, 2011 43