Agile Methodologies
and
Chaos Revisited
William (Bill) Myers
Cinergy Corporation
Information Builders Summit 2004 User Conference
May 2004
* The statements and opinions herein are those of the individual author. They are provided for informational
purposes only and should not be relied upon for decision making.
May, 2004 William (Bill) Myers 1
Introduction
• Audience & Flow
• Agile Methodologies
• Philosophy & Methods
– Management & Coding
• General Agile reporting
• Questions
– Follow-up questions at end
• Appendix
• Links
• Additional Details / starts at References slide
• Additional points
• Supplemental Handout
May, 2004 Page 2 William (Bill) Myers
What is Agility
• Ability to adapt
• to Changing circumstances
• to Changing requirements
• Impact
• Improves business climate
• Provides competitive edge
• Quick gratification
» Customers & Developers
» Faster Return on Investment
May, 2004 Page 3 William (Bill) Myers
Point of View - Concerning Risk
• Traditional Methods
• Risks for IT
– Errors
– Delivery on budget, on time, on scope
• Business risks generally ignored
• Agile Methods
• Business risks
– Competitive timeliness
– Effectiveness / the right thing
• IT risks
– Errors
– Budget, time, scope
May, 2004 Page 4 William (Bill) Myers
Agile Methodologies / Perspective
• How to Decide
• Relationships &
Responsibilities
• 80 / 20 rule
• Feature driven
• Customer stories
• Baseline high level
• Emerge/Converge
• Reduce
reporting risks
Note: Crystal Methodology
IS NOT RELATED to the reporting product
May, 2004 Page 5 William (Bill) Myers
What is Agile Development?
• Set of policies and conventions
• Controls risk
• Puts project into safety zone
• Agile Methodologies
• Changing environments
• Philosophies
– Products, Personal relationships, Timing, Documentation
• Methods
– Approach & Principles
– Project management
– Programming – low level practices
• Traditional Methodologies
• Rigid, known environments
• Static processes
• Structured Methods
– Waterfall project management & clones
May, 2004 Page 6 William (Bill) Myers
Agile Philosophies
http://www.agilealliance.org
• Agile Manifesto
– Individuals and interactions
– over processes and tools
– Working software
– over comprehensive documentation
– Customer collaboration
– over contract negotiation
– Responding to change
– over following a plan
Quote from consortium: “While there is value in the
items on the right (sub-topics above), we value the
items on the left (major topics) more”.
May, 2004 Page 7 William (Bill) Myers
General Agile Principles
• Deliver something useful to customer
• Nothing counts until you deliver software
• Deliver frequently
• Four to eight weeks
• Encourage collaboration
• “Active customer involvement is imperative!” – find a champion
• Daily meetings – very short
• Provide technical excellence
• Rely on the skills of your people
• Do the simplest thing possible
• Balance flexibility and structure
• Self-correcting teams
• Don’t let people suffer
• Install when business needs it & technically feasible
May, 2004 Page 8 William (Bill) Myers
Agile Manifesto - Highlights
• Nothing counts until you deliver software
• Deliver every four to eight weeks
• Direct contact - customers & developers
• Daily contact and communication
• Let champions drive projects
• Maintain a constant pace
• Agile processes enhance quality
• Teams self-correct
• Never forget the goal - software
May, 2004 Page 9 William (Bill) Myers
Adaptability Scale
Remember Sarbanes-Oxley
May, 2004 Page 10 William (Bill) Myers
Project Management Styles
• Command & Control
• Defined process
• Plan what you expect – make predictions
• Enforce the predictions
• “Change Control”
– to manage change
• Leadership – Collaboration
• Empirical process
• Start with a plan
• Change plan frequently
• Inspection & Adaptation
– To control change
May, 2004 Page 11 William (Bill) Myers
Waterfall Methods
and clones
• Process
• Sequential
• Analysis, design, construct, test, implement
• Rigid
• Paper intensive
• Good with 1960’s mainframes with cards
• When Useful
• Known processes
– Hardware & vendor software upgrades
• Little creativity required
May, 2004 Page 12 William (Bill) Myers
Many Agile Methods
• What methodology to use ?
• Crystal & Crystal Light ( helps with decision )
• Project management
• Scrum ( broadest, variable end-point )
• DSDM Model ( construction oriented )
• Lean Development (domain, 80/20 approach )
• Feature-Driven Development - FDD (most structured)
• Adaptive
• Extreme Programming – XP
• Programming oriented
• Extreme Programming - XP
» Customer Stories, programs, working conditions
• Agile reporting ( core & option )
May, 2004 Page 13 William (Bill) Myers
Agile Methodologies
• Agile Software Development Ecosystems
• Reference book & web site
• by Jim Highsmith
• http://www.jimhighsmith.com
• Tom DeMarco:
• Agility: 1
• everything else: 0
• Numerous books
• Highlights only
May, 2004 Page 14 William (Bill) Myers
Observations - Agile Conference
• Easy to go wrong
• Mis-interpretation, mis-implementation, missed point
• No silver bullet
• One size does not fit all
• Beware too much documentation
• Offload legal requirements from developers
• Always “on time, on scope, on budget”
• Leads to mediocrity ( no risks taken )
• Cultural change
• from - military Command & Control
• to – cooperative Leadership & Collaboration
• Productivity leaps
• 77 function points / month when industry average 2 pts / mo
• WebFocus fits agile development very well
May, 2004 Page 15 William (Bill) Myers
Reasons for Failure
• One Size Methodology Fits All
• Intolerant Methodology
• Embellished
• Do frivolous tasks
• Ignore necessary tasks
• Heavy
• Untried
• Tried Once
• Limited applicability
• Champion
• Failure to Tune
• Distractions
• Dates can’t be met
May, 2004 Page 16 William (Bill) Myers
Crystal Methodology – Theory
( NOT RELATED to Crystal Reporting )
• Self-adapting "shrink-to-fit," software
development methodologies
• Different set of policies & conventions for each project
• Projects are sensitive to people issues
• Crystal is people-centric
– other methodologies may be:
» process-centric
» architecture-centric
» tool-centric
• Keys
• Communication intensive
– Masses of paper do not improve communication
• Experiment based
• Requires feedback loops
• Theory of Methodologies
May, 2004 Page 17 William (Bill) Myers
Crystal – Communication
• Effectiveness of different methods
People are Non-Linear
First-Order Components
in Software Development
• Courtesy of Alistair Cockburn
• http://alistair.cockburn.us/crystal/articles/cpanfocisd/characterizingpeopleasnonlinear.html
May, 2004 Page 18 William (Bill) Myers
Crystal - Model
• Model
• Risk to end-user, Size, Complexity
– Tables present questions to ask
» To establish processes & controls
» When to change
• Weight is costly
– Heavy methods for large teams
– More ceremony for critical deliverables
– Avoid frivolous tasks
– Efficiency is expendable
» Non-bottleneck activities & Rarely used items
• Crystal Methodologies
• Reference books & web site
• Alistair Cockburn – Author
• http://alistair.cockburn.us/crystal
May, 2004 Page 19 William (Bill) Myers
SCRUM
• Principles
• Do the right thing (from customer’s view)
• Deal with reality, not artifacts or fixed plans
• Short iterations
– Usually four weeks
– Be prepared to implement at the end of an iteration
• Allow changes
• Keep everything visible
– Concentrate on time remaining - not time spent
• Highlights and pinpoints responsibilities
• Self organization
– Creates sense of obligation and teamwork
– Extensive collaboration
– Changes role of Project Leader
» Certification available
May, 2004 Page 20 William (Bill) Myers
SCRUM
• Broad application
• Iterative, lightweight project management method
• Software AND non-software projects
• Based on manufacturing Process Control theory
• Adaptable
– Can be implemented
• Beginning of a project
• Middle of a project
• Product development effort that is in trouble
• More information
• Ken Schwaber and Mike Beedle
• http://www.controlchaos.com/
May, 2004 Page 21 William (Bill) Myers
SCRUM – Concepts
Courtesy of Ken Schwaber
http://www.controlchaos.com/Scrumo.htm
May, 2004 Page 22 William (Bill) Myers
SCRUM
• Full visibility
• Priority list
• Items in development cycle ( sprint )
• Short, daily meetings ( 15 minutes )
• Work hours remaining in sprint
• Assignments, who & what
• Responsibility & Authority
• Anyone can add to priority list
• Business manages priorities
• Developers determine what can be done
• Project Leader removes impediments
• Removes developers from politics
May, 2004 Page 23 William (Bill) Myers
SCRUM - Impact
• Backlog
• Scope Creep – no such thing
• No request ever lost
» It may never get a high enough priority
• Business owner only one who can remove it
• Return on Investment
• Install option available
– Software completed to date available / immediate return
• Option to stop
– Save development costs
• Perfect for fixed price projects
– Reorder deliverables list – no cost
– Substitute deliverables with equal effort – no cost
– “Stop Development” clause easy to apply
– Big competitive advantage
May, 2004 Page 24 William (Bill) Myers
Scrum - Definitions
• Product Backlog
• Prioritized list of items
» Business Product Owner – sets priorities
» List of Functional requirements,
Nonfunctional requirements, Issues
» Contains preliminary estimate
» Changes constantly
» Items in a sprint: locked down
• Time-box
• Period of time that can not be exceeded
• Sprint
• Time-box of 30 days
• Sprint backlog defines delivery list
» Product item
» Internal support item (i.e. database, system doc)
» Work remaining & burndown graph
May, 2004 Page 25 William (Bill) Myers
Scrum - Players
• Scrum
• Product Owner
» Adds items to product backlog list
» Set priorities
• Team
» Determines sprint list
» Develops software
• Stakeholder
» Funded project, will use it, affected by it
» Requests enhancements
• ScrumMaster
» Responsible for Scrum process
» Removes impediments
• Other interested parties
• IT Management
» Manpower allocation
» Budgets & Billing
• Senior Business Management
» Best use of corporate resources
May, 2004 Page 26 William (Bill) Myers
Scrum - Milestones
Sprint
( 30 cal days ) Sprint
Sprint Planning Sprint Review
Retrospective
Meeting Daily Scrum Meeting
Meeting
( 8 Hours ) Meeting ( 4 hours )
( 3 hours )
( 15 minutes )
• Sprint Planning meeting ( two 4 hour segments )
» Product owner & Team: identifies backlog items to be
delivered in 30 day sprint / Team commits
» Team plans details in Sprint Backlog
• Daily Scrum Meeting ( 15 minutes )
» Team: Review prior day, identify impediments, plan day
• Sprint Review Meeting ( 4 hours )
» Team: demonstrates completed functionality
- to Product Owner and other interested parties
• Sprint Retrospective meeting ( 3 hours )
» Team: how to improve next sprint or
make more enjoyable
May, 2004 Page 27 William (Bill) Myers
SCRUM: database - goals
• Software development
• Collect new requests
• Manage product backlog
• Manage Scrum process
• Reduce administrative load on developers
» Reduce or eliminate most status reporting needs
• Support Management
• Provide data for manpower planning
• Provide status to everyone
» Priority in queue
» Currently being worked on
» Remaining effort
• Provide historical information
» Time tracking mechanism for billing
» Answer “What did you do for us last year”
May, 2004 Page 28 William (Bill) Myers
SCRUM – Process Flow
SCRUM
Backlog Planning Sprint Demonstration
User
Request new
feature
Product Owner
Business
Commits Put
Accept or business into
Prioritize
Reject resources Production
Backlog
request ?
Daily
Planning
Meeting
e-Mail Daily SCRUM Meeting: Yes
Development
result 1. Completed yesterday
2. To complete today Demonstrate
Team
Make
Determine what 3 Impediments Results
via Web Page high level
can be done
estimate Put into
by priority
(Team Commits) Production
Log time for Retrospective
billing only Meeting
Estimate
Hours Work
Application
e-Mail notification
remaining
Process
Request
Response
Accepted Backlog Sprint List
Charge
back
Result
Status Change
Projected
Reports
Work in Current Delivered Invoiced Status
Priorities Manpower
Process Assignments Products Hours Reports
Needs
REPORTS:
Timely - reports always show current status, but can show historical strips
Agile - reports are self-building, self-summarizing, include many filters
Visible - only a confidential project would be hidden William W Myers - June 2004
May, 2004 Page 29 William (Bill) Myers
SCRUM – Database Model
SCRUM Projects
ProjectType
Database Project_ID ProjectType
Model
Request Assignments People ProjectTeam
FromUser Project_ID
Project_ID
Person_ID Person_ID
UserReq_ID Feature_ID Person_ID
Project
Management
ProjMgt_ID
Features
Feature_ID Production
Module
Module_ID
Comments
Comment_ID
Hours
Module Sprints
Charged
Sprint
Charge_ID_ID Sprint_ID
Module_ID
Person_ID
Sprint_to_ Sprint_ID
Source
Feature
Comment_ID
Feature_ID Sprint
Features
Sprint_ID
Feature_ID
William W Myers - June 2004
May, 2004 Page 30 William (Bill) Myers
SCRUM: demo app – Priority Page
May, 2004 Page 31 William (Bill) Myers
DSDM Model
• Principles
• Iterative & incremental development
• Changes during development are reversible
• Requirements baselined at high level
– Minimum details to continue
• Testing throughout lifecycle
– Developers AND business users
• More Information
• Developed in UK
• Grew out of RAD approach
• http://www.dsdm.org/
May, 2004 Page 32 William (Bill) Myers
Lean Development Model
• Most strategic oriented of Agile methods
• Principles
• Domain instead of point solutions
• 80% today better than 100% tomorrow
• Everything is changeable
• Minimal is essential
• Needs determine the technology
• Best value for money
• More information
• http://www.poppendieck.com/
• http://www.leanconstruction.org/
May, 2004 Page 33 William (Bill) Myers
Adaptive Software Model
• Software development
• Complex adaptive system
• Like stock market or flock of birds
• Circumstances generate emergent capabilities
» Total skills exceeds sum of parts
» Only emergent capabilities tames complexity
• Software organizations:
Must create the proper circumstances
for emergence
of superior capabilities
May, 2004 Page 34 William (Bill) Myers
Adaptive Software Model
• Principles
– Leadership – Collaboration
– Speculate, collaborate, learn
– Drop militaristic Command and Control (stifles emergence)
– Cycle through phases many times, converging
– Tasks not listed in plan
– Components at end of plan are listed
– Mission directed, feature driven, iterative
time boxed, risk driven, change tolerant
May, 2004 Page 35 William (Bill) Myers
Feature-Driven Development (FDD)
• Process
– One-time
– Quick, high value, low dollar
• Develop overall domain model
• Build feature list
• Plan by feature
– Iterative
• Design by feature
• Two week cycles building components
• More information
• http://www.featuredrivendevelopment.com/
• http://www.nebulon.com/fdd/
May, 2004 Page 36 William (Bill) Myers
Extreme Programming - XP
• Project management
• Collect customer stories
• Estimate & prioritize
• Small iterative development cycles
• Development
• Test-first development
» Write test cases before coding
• Collective code ownership
• Refactoring
• Pair programming
• Daily standup meetings
• Many processes & rules
• More information
• http://www.extremeprogramming.org
• http://www.objectmentor.com/home
May, 2004 Page 37 William (Bill) Myers
Agile - Review
• Positives
• Software delivered
• No death marches
• Dangers
• Process itself becomes the goal
• Placebo effect
• Key point is missed
• SCRUM
• Immediately useful
• Becoming broadly popular
» Classes around the world
» Hundreds certified
May, 2004 Page 38 William (Bill) Myers
Reporting Population
Super-Users Strategic Users
(ad-hoc) ( historical cubes )
Tactical Users
(core & options )
• Super User
» Technical skills
» Understands their piece of the data
» May be tactical or strategic
• Strategic User
» Executive or analyst
» Needs highly summarized and graphed snapshots
» Limited technical skills
• Tactical User
» Needs up-to-minute detailed data to do their job
» Limited technical skills
May, 2004 Page 39 William (Bill) Myers
Normal Report Development Process
• Demand that customer predict the future
• Forecast list of needed reports
• Requires visualization / no data available
• Data may support 20,000,000 variations
• Design signoff in blood
• Early code freeze
• Rigid Change Control
• More signoffs
• Complain about scope creep
• Accuse users of changing their mind
• List all changes as bugs
• Falsely implies a defective product Requirements List
May, 2004 Page 40 William (Bill) Myers
Agile Reporting
• Bounded Chaos
• Random actions inside of the boundary
• Self Building
• Developer builds 90%
• User checks options
• Program finishes writing itself
• Program creates report
• Broad Scope
• Can replace millions of fixed reports
• Requirements can be hazy up front
May, 2004 Page 41 William (Bill) Myers
Agile Reporting - Definitions
• Report (definition)
• Any output where user can not change data in database
• More than printer
• Core & Options
• Core – on every report, fixed
• Options – added to report based on user’s selections:
columns & totals
• Filters
• Limits input data
• Medium
• Web page, paper, excel, pivot table, PDF file, lists,
graphs, etc
• Measures
• Changes the core - unusual
May, 2004 Page 42 William (Bill) Myers
IBI Sample - ggorder2.fex
• As Installed
• Units by Product
• For one date
• Agile Modification
• Add &sort1
• Standard front-end
• Add dollar value
» Immaterial to agile
May, 2004 Page 43 William (Bill) Myers
Core & Option - Web Page
• Shows total sales of each product by state & sales rep
• Checking no boxes: one number only
May, 2004 Page 44 William (Bill) Myers
Agile ggorder2.fex
• Core only
• Boxes checked
• None
May, 2004 Page 45 William (Bill) Myers
Agile ggorder2.fex
• Same fex
• No changes
• Boxes checked
• Vendor Name
• Product
May, 2004 Page 46 William (Bill) Myers
Agile ggorder2.fex
• Same fex
• No changes
• Boxes checked
• Product
– Totals
• Store Name
May, 2004 Page 47 William (Bill) Myers
Agile ggorder2.fex
• Same fex
• No changes
• Boxes checked
– State, City,
Store Name (totals),
Vendor Name,
Product
• Options
– 362,889 formats
– ONE program
– Risks managed
May, 2004 Page 48 William (Bill) Myers
Agile Reporting – Core & Options
• Core
– Units Ordered
• Options
– State
– Contact
– Product
– Package
– Totals on:
Contact, State
May, 2004 Page 49 William (Bill) Myers
Core & Option Executive Demo
• IBI Sales Prototype
• Used Production databases
• Not standard IBI demo
» Specs and Core & Option HTML were supplied
• 4 Days development
• Clear to all
• Reports
• Simple list – Volumes & Dollars (agile core & option)
• Delta – Change since yesterday (agile core & option)
• Complex Delta – Changes of averages
• 3-D Graphs
• Pies with drill downs
May, 2004 Page 50 William (Bill) Myers
Exclusions
• Power not demonstrated
• Combined databases
• Reporting Cubes
• Scheduling & Mailing
• Super-user interface
May, 2004 Page 51 William (Bill) Myers
Risk Mitigation
• End-User reporting
– Normal recommendation
– Risks
– Mitigation for average user
– Super-users
May, 2004 Page 52 William (Bill) Myers
Normal Reporting Recommendation
• Reporting is simple, use…
– MS Access
– Excel
– Pivot Tables
– WebFocus
– Crystal Reports
– Etc
• Training
– Help
– Manuals
– Limited classes
May, 2004 Page 53 William (Bill) Myers
Risks
• Complexity
– Database structure
– Queries
• All tests against production databases
– Overload servers & network
• Over-reporting Deal Price
– Double counting data
• Under-reporting
Deal Price
– Not including data
May, 2004 Page 54 William (Bill) Myers
Mitigation
• Target the skill sets
– Developers
• Handle complexity
• Insure accuracy
• Insure efficiency
• Identify possible options
• Set up “Core & Option” reports
– End Users
• Decide on core & options during development
• Test during development
• Understand the results
May, 2004 Page 55 William (Bill) Myers
Products & Languages
• Effort
• Easy
• Varying levels of difficulty
– Brute force –vs- fits vendor’s product well
• Power
• Is agile
• Appears to be agile
• Partly agile
• Beware of pretty –vs- power
May, 2004 Page 56 William (Bill) Myers
Regular File Structure - WebFocus
Pre-load Standard Reporting Program
( fex )
Relationships
Variables
Lists Get
Report Caster Menu Item
Core & Options
( fex ) ( fex )
( HTML )
Options Pre-set
Pre-set
Filters Options, Filters
Options, Filters
Output Medium Output Medium
Output Medium
Report
( fex )
HTML
PDF Files
Excel
May, 2004 Page 57 William (Bill) Myers
Core & Option Structure - WebFocus
• Build JavaScrips
• Options & Template
• Options – SQL database
• Template & Fields – text file
• Hidden Fields fairly static
May, 2004 Page 58 William (Bill) Myers
Report ( fex )
DEFINE FILE TRADES
Cost/P12.2CB = VOLUME * Price;
Value/P12.2 = VOLUME * STRIKE_PRICE;
Net/P12.2 = Value - Cost;
END
TABLE FILE TRADES
SUM VOLUME AS 'Volume'
Cost
Value
&sort1
-* ***** add other BY statements here *****
END
May, 2004 Page 59 William (Bill) Myers
The End
( References )
• http://www.agilealliance.org - Agile Alliance
• http://www.jimhighsmith.com - Agile overview
• http://alistair.cockburn.us/crystal - Crystal
• http://www.controlchaos.com/Scrumo.htm - Scrum
• http://www.dsdm.org - DSDM
• http://www.poppendieck.com - LEAN
• http://www.leanconstruction.org - LEAN
• http://www.santafe.edu - Complex Adaptive
• http://www.featuredrivendevelopment.com - FDD
• http://www.nebulon.com/fdd - FDD
• http://www.extremeprogramming.org - EX (Extreme)
• http://www.objectmentor.com/home - EX (Extreme)
May, 2004 Page 60 William (Bill) Myers
Agile Manifesto (1)
Follow these principles for Developing Software:
• Satisfy customers
with valuable software
• The highest priority is to satisfy the customer
through early and continuous delivery of valuable
software.
• Welcome change requests at any time
• Making changes quickly can be a competitive
advantage
• Welcome changing requirements, even late in
development. Agile processes harness change for
the customer's competitive advantage.
May, 2004 Page 61 William (Bill) Myers
Agile Manifesto (2)
Follow these principles for Developing Software:
• Deliver working software frequently
• Don’t let people suffer
• Deliver working software frequently, from a couple
of weeks to a couple of months, with a preference
to the shorter timescale.
• Daily Contact
• Customers & developers
• Avoid intermediaries
• Business people and developers must work
together daily throughout the project.
May, 2004 Page 62 William (Bill) Myers
Agile Manifesto (3)
Follow these principles for Developing Software:
• Use Champions
• Build projects around motivated individuals. Give
them the environment and support they need, and
trust them to get the job done.
• Team Communication
• Avoid long or large meetings
• The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.
May, 2004 Page 63 William (Bill) Myers
Agile Manifesto (4)
Follow these principles for Developing Software:
• Working software is primary measure
• Models, documentation, use cases do not count
• Working software is the primary measure of
progress
• Maintain a constant pace indefinitely
• No Death Marches
• Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.
May, 2004 Page 64 William (Bill) Myers
Agile Manifesto (5)
Follow these principles for Developing Software:
• Quality enhances agility
• Late and post-implementation changes easier
• Continuous attention to technical excellence and
good design enhances agility.
• Self-organizing teams increase quality
• Volunteers produce better results
• Members look out for and help each other
• The best architectures, requirements, and
designs emerge from self-organizing teams.
May, 2004 Page 65 William (Bill) Myers
Agile Manifesto (6)
Follow these principles for Developing Software:
• Simplicity
• Do less – get more
• The art of maximizing the amount of work not
done -- is essential.
• Teams self-correct
• Not continuously – will drive people nuts
• At regular intervals, the team reflects on how to
become more effective, then tunes and adjusts its
behavior accordingly.
May, 2004 Page 66 William (Bill) Myers
File Naming Standards - WebFocus
Function Program Name Suffix
Pre-load Scr_Status_h.fex h
Get Core & Options Scr_Status.htm
Report Scr_Status.fex
Drill Down Scr_Status_dr1.fex dr(+)
Report Caster Scr_Status_c1.fex c(+)
Menu Item Scr_Status_m1.fex m(+)
• Application prefix + Name + Suffix
– Drilldown can be called from many reports
– Name is first report written
• Program field names – UpperLower_Case
May, 2004 Page 67 William (Bill) Myers