Risk Base Testing Strategy by koh14466


Risk Base Testing Strategy document sample

More Info
									Banking	and	Capital	Markets

Balancing	quality,	risk	and	cost—	
Test	profiling
The	past	decade	has	seen	institutions	          Not	only	because	of	the	dollar	value,	         Figure 1: Risk based testing
move	away	from	the	fallacy	that	you	must	       but	because	the	business	objectives	
and	can	test	everything.	Why?	Because	          and	risks	will	drive	the	testing	strategy	
it	can’t	be	done,	it	costs	too	much	to	         and	associated	estimates.	Which	means	
try	and	it	doesn’t	produce	the	intended	        testing	shouldn’t	be	an	afterthought	in	
benefits	in	quality.                            refining	estimates.	This	cost	needs	to	be	
                                                driven	by	the	business	objectives	and	
On	a	large	project	with	a	long	timeline,	       value	to	the	organization	instead	of	what	
there	are	many	moving	parts.	Let’s	use	         IT	thinks	it	should	test.
the	code	base	to	illustrate	an	example,	if	
you	are	trying	to	test	all	components,	by	      Most	of	the	testing	groups,	in	the	
the	time	you	are	part	way	through,	you	         majority	of	companies,	size	projects	
likely	have	changed	the	code,	so	you	           based	on	the	functional	requirements	
have	to	restart	testing	many	times	over.	       for	the	project.	Few	go	down	the	
The	retesting	materially	increases	overall	     path	of	assessing	functional	risks	of	
cost	of	the	initiative.	Typically	testing	      the	applications	developed	and	even	          Risk                                  Cost
makes	up	to	40%	(or	more)	of	the	total	         fewer	will	determine	the	risks	based	on	
development	costs.	On	a	$10m	project,	          probability of failure,	e.g.	an	assessment	
$4m	is	spent	on	testing	without	the	            conducted	based	on	information	                When	impact	of	failure is	quantified,	
benefit	of	superior	quality.	                   provided	by	the	development	teams	             companies	look	into	reputational	risk,	lost	
                                                and	the	impact of failure	–	assessment	        business,	lost	productivity,	maintenance	
Of	all	the	disciplines	required	to	deliver	a	   conducted	based	on	information	                costs,	and	all	of	these	factors	translate	
successful	project,	testing	represents	one	     provided	by	the	business	department	for	       the	business	risk	into	dollars.	Many	
of	the	largest	cost	saving	opportunities.	      the	product.                                   of	these	organizations	have	powerful	
                                                                                               Project	Management	Offices	(PMO)	that	
So what is risk based testing?                  The	companies	that	do	these	                   manage	and	monitor	risk	closely.	Mature	
It’s	the	balance	of	risk	and	cost	to	get	       assessments	understand	that	technical	         organizations	use	the	risk	calculations	
the	quality	you	need	instead	of	what	you	       risk	is	dependent	on	factors	like:	            below	to	quantify	the	contingency	budget	
hope	you	are	getting	(Figure	1).                code	sanity,	developers’	competency,	          and	some	even	use	the	profile	to	plan	
                                                technical	context,	architectural	              slack	into	the	schedule.	The	downstream	
Most	projects	start	with	a	good	idea	           complexity,	environment	configuration	         benefit	of	which	is	a	more	predicable	
supported	by	a	business	case	for	a	             and	tools	usage.	In	order	to	measure	          project	budget	and	schedule.
product	that	will	provide	competitive	          risk,	developers	look	into	risk	
advantage,	replace	a	technology,	or	            measurement	such	as	lines	of	code,	            Traditional	risk	variables	are:
provide	better	services	to	customers.	          branches,	paths,	complexity,	changes	in	
Given	the	relative	cost	of	testing	to	the	      source	code	and	inspect	them	regularly	        •	Probability	of	Failure:	P(i)
total	cost	of	ownership,	you	have	to	           and	exercise	due	diligence	in	managing	        •	Impact	of	Failure:	I(i)
consider	testing	in	your	business	case.	        source	control.
                                                                                               Risk, R(i): R(i) = P(i) x I(i)

                                                                                                         PricewaterhouseCoopers	LLP				11
                                                                                               The	primary	method	used	in	the	Test	
                                                                                               Profile	Level	Assessment	is	through	
                                                                                               a	questionnaire.	It	assesses	the	risk	
                                                                                               inherent	in	any	testing	deliverable.		As	
                                                                                               with	most	risk	assessments,	it	attempts	
                                                                                               to	gauge	what	the	impact	of	these	risks	
                                                                                               would	be	and	the	likelihood	of	them	
                                                                                               to	occur	(impact	and	probability).	The	
                                                                                               two	main	sources	for	completing	the	
                                                                                               questionnaire	are	any	relevant	system/
                                                                                               project	documentation	as	well	as	input	
                                                                                               from	system/project	subject	matter	
                                                                                               experts	(Figure	2).		

                                                                                               The	Test	Profile	Level	Assessment	is	
                                                                                               the	responsibility	of	the	project,	and	is	
                                                                                               normally	performed	by	the	test	manager.	
                                                                                               This	assessment	is	a	recommendation	
                                                                                               and	can	be	adjusted	to	fit	the	needs	of	
                                                                                               the	project.	For	example,	an	increase	
                                                                                               in	the	test	level	may	be	warranted	if	the	
                                                                                               project	has	a	very	high	business	profile.

                                                                                               In	summary	there	are	four	levels:	L1,	
                                                                                               L2,	L3,	and	less	complex	testing	for	
The	importance	of	these	risk	                 If	we	measured	risk	thoroughly	using	            L4.	At	all	levels,	some	test	phases	may	
measurements	is	useful	during	testing	        the	method	described	above	do	we	                be	combined	(e.g.	system,	systems	
execution	prioritization	when	risk	base	      have	enough	information	for	the	testing	         integration	or	business	acceptance	
testing	is	employed.	                         assessment	to	build	an	accurate	testing	         testing).	
                                              profile	for	the	project?	What	else	is	there?
Risk-based testing (RBT)	prioritizes	the	                                                      The	Testing	Profile	Level	Assessment	
features	and	functions	to	be	tested	based	    Functional	testing	risk	assessment	              includes	the	sections	noted	in	Figure	2.
on	the	risk	they	represent.	In	this	way	      assesses	the	application	functional	
the	execution	begins	with	high	risk	test	     requirements	at	best,	but	what	we	have	          Base	Score
cases	and	continues	testing	using	the	        yet	to	identify	are	the	risks	for	the	project.
risk	score.	However,	what	most	of	the	risk	                                                    This	part	of	the	assessment	takes	into	
base	testing	proponents	are	overlooking	      Test Profile Level Assessment (TPLA)             consideration	the	fundamental	nature	
is	the	importance	of	unit	testing,	as	a	      is	a	tool	that	determines	an	efficient	          of	the	system.	This	score	doesn’t	vary	
mandatory	prerequisite	for	successful	risk	   balance	between	quality	and	cost	on	             much.
base	testing	and	significant	reduction	of	    the	one	hand,	and	risk	on	the	other.	It	
system	integration	test	(SIT)	execution.	     helps	determine	the	smartest	way	to	             The	sections	within	Base	Score	focus	on:
The	risk	you	may	take	without	knowing	is	     mange	risk	and	provides	information	on	
deploying	code	in	production	that	was	in	                                                      •	 The	financial	characteristics	of	the	
                                              the	usage	of	testing	resources	in	a	given	
fact	never	tested!                                                                                system	

12				Banking	Review:	Spring	2010
•	 How	the	customer	is	affected	by	the	       many	differences	in	the	results	of	this	                Test	Profile	Level	Assessment	Final	
   system	                                    assessment	from	project	to	project.                     Score	Includes	the	following	levels
•	 The	technical	and	operational	nature	
                                              The	sections	within	Project	Score	focus	                • L1 High Risk: all	phases	of	testing	
   of	the	system	
                                              on	the	following:	                                        and	all	testing	artifacts	are	mandatory;	
                                                                                                        could	merge	system	and	SIT	testing.
Project	Score                                 •	 Size	of	the	delivery	
                                              •	 Technical	complexity	of	the	delivery                 • L2 High/Medium Risk: all	phases	
This	part	of	the	assessment	takes	into	                                                                 of	testing	and	all	testing	artifacts	are	
consideration	the	nature	of	the	project’s	    •	 Project	structure	                                     mandatory,	development	integration	
deliverables,	how	the	deliverables	will	                                                                testing	could	be	excluded;	also	
                                              •	 Nature	of	the	project	
be	developed	and	implemented,	as	well	                                                                  system	and	SIT	testing	could	be	
as	the	make-up	of	the	project.	There	are	                                                               merged.

Figure 2: Testing Profile Level Assessment

    Base Scoring—Financial Score             •	   Total	amount	of	external	customer	account	balances
                                             •	   Impact	producing	erroneous	financial	results	
                                             •	   System	rate	in	terms	of	transaction	volume
                                             •	   Average	value	of	each	transaction

    Base Scoring—Customer Score              •	   External	customer	impact	of	unavailable	or	erroneous	financial	results	
                                             •	   Sensitivity	of	the	data	in	the	system
                                             •	   Importance	of	the	external	customer	base	of	this	system	to	the	organization	
                                             •	   External	customers	interface	to	this	system

    Project Scoring—Size Score               •	 The	size	of	the	project	team	for	this	deliverable
                                             •	 The	size	of	the	deliverable,	either	in	terms	of	line	of	code	(LOC)	or	new	functionality
                                             •	 Impact	of	this	system/product	to	the	business	areas

    Project Scoring—Technical                •	   Test	environment	configuration	and	resemblance	with	production
    Complexity Score                         •	   Number	of	new	interfaces	to	be	delivered
                                             •	   The	nature	of	the	new	technological	infrastructure	to	be	used
                                             •	   The	nature	of	the	new	functionality	being	delivered?

    Project Scoring—Project Complexity       •	   Where	will	development	take	place?
    Score                                    •	   Knowledge	of	the	vendor(s)	involved	in	the	project
                                             •	   Requirements	development	methodology
                                             •	   Level	of	expertise	of	the	project	team
                                             •	   Type	of	change	management/version	control	being	used

    Project Scoring—Project Impact           •	   Impact	of	the	delay	in	implementing	this	project	on	other	projects	
    Score                                    •	   System	implementation	details
                                             •	   Fallback	plans	in	place	in	the	event	of	system	failure?
                                             •	   How	would	a	delay	or	a	defect	in	the	project	affect	the	financial	institution’s	compliance	with	
                                                  regulatory,	legal	and/or	tax	requirements?

                                                                                                                 PricewaterhouseCoopers	LLP				13
• L3 Medium/Low Risk:	all	phases	of	               • L4 Low Risk:	development	integration	                           development	integration	testing	and	
  testing	and	all	testing	artifacts	are	             testing	is	not	required.	Test	strategy	                         system	testing	could	be	excluded;	
  mandatory.	Development	integration	                is	not	required,	the	test	plan	would	                           also	system,	SIT	and	business	
  testing	and	system	testing	could	be	               suffice.	Depending	on	the	project,	                             acceptance	testing	could	be	
  excluded;	also	system	and	SIT	testing	             a	shorter	version	of	the	test	plan	                             merged	(Figure	3).
  could	be	merged.                                   could	be	used.	In	some	cases	
                                                                                                                  This	exercise	should	help	you	define	
  Figure 3: TPLA Customization                                                                                    your	testing	approach	and	determine	
                                                                                                                  how	you	will	move	testing	forward	into	
                                                          		 L1	Test	Strategy	+Test	Plans	for	each	stage
                                                                                                                  the	strategy	phase.	The	next	step	is	to	
                              Full Testing
                               Process                                                                            establish	the	strategy	that	will	define	
                                                          		 L2	Light	Test	Strategy	+	Test	Plans	for	each	stage   what	you	will	test	and	measure	quality	
    Testing	Profile	Level	                                                                                        based	on	the	value	it	brings	to	the	
                                                          			L3	Test	Plans	for	each	stage                         organization	and	intended	business	
                             Light Testing                                                                        case.
                                                          			L4	Light	Test	Plan	(Testing	Workbook)

                                                                                                                   We will expand on the testing
                                                                                                                   approach and strategies as
 Figure 4: Testing Risk Assessment Path                                                                            outlined in Figure 4 for future
                                                                                                                   articles. If you have any
                                                                                                                   suggestions on upcoming
                                     Project	initiation                          Project	Charter                   themes, please contact:

    Business Team                                                                                                  Daniela	Medeleanu	
  Business	Team	Impact
                               Testing	Profile	                       Testing	                 Testing	            416	869	8760		
   Development Team             Assessment                           Approach                  Strategy            daniela.medeleanu@ca.pwc.com
   Probability	of	failure

                                                                                                                   Paulette	McLeod		
                                Project	Risk	                                                                      416	869	2371		

14				Banking	Review:	Spring	2010

To top