EXERCISE A– Testing Project
Post delivery defect leakage goal: This is the % of the testing defects identified in our organization(excludes client found defects)/total defects. It should be greater than 99%.
Effort and Schedule goals & formula:
Effort PCB Schedule PCB
Goal(Upper Limit) 15% Goal(Upper Limit) 5%
Goal(Lower Limit) 0% Goal(Lower Limit) -5%
Effort : (Actual Hours-Estimated Hours)* 100 / (Estimated Hours)
Schedule: ((Actual End Date-Actual Start Date)-(Estimated End Date- Estimated Start Date))*100 / (Estimated End Date-Estimated Start Date+1)
Phase Organization Goal Formula
RDD Completion of test preparation >=0.75 Review defects/Size
TDD Completion of test execution >= 1.25 Review defects/Size
Defects distribution across different testing cycles:
Usually 60% defects are found in the 1st test cycle, 90% of defects found by the 2nd test cycle, remaining in the last test cycle.
Note: Goals are usually tighter by 25% of the mean. While totaling defects use the weighted defects i.e. Severity 1 = 10, Severity 2 = 5, Severity 1 =1
Case Study 1:
Setting Goals
You are an independent testing team and the scope of work is system testing. There is another team Denver which has done similar projects within the same financial area. Their data is as follows;
Denver Data
Post delivery defect leakage 95%
Effort Variance UCL 50, LCL -40
Schedule Variance UCL 3, LCL 0
RDD .25
TDD .50
Size 600 Test Case Points
However this project is more complex and has many more interfaces to test. This would be data driven test where many SQL queries would have to be written to extract and validate the data. The size is 1000 Test Case points. You have planned 3 test cycles. Within each test cycle you have planned for 3 test rounds.
Q.1. Set the post delivery defect leakage, effort/schedule variance, RDD and TDD goals. How many defects would you expect at end of test preparation and test execution.
How many defects you would expect at end of each test cycle? Make your assumptions and identify the risks.
A.1.
Project Data Goals
Post delivery defect leakage 95%
Effort Variance UCL 50, LCL -40
Schedule Variance UCL 3, LCL 0
RDD .75, 750 defects
TDD 1.25,1250 defects, 1st Cycle – 750, 2nd cycle -375
Size 600 Test Case Points 1000
At end of test Planning
At end of test preparation the effort, schedule data looks like this;
Planned Actual
Effort 1200 man hrs 1500 man hrs
Schedule 10th June 2004 12th June 2004
Schedule Variance UCL 3, LCL 0
Review defects 450 defects
During your test planning phase some testers resigned and you had to quickly recruit new associates who joined towards the end of test planning phase.
The main business functionalities are adjustments, auto login, customer service transactions, BT related functionality, credit card payment and maintaining batch items. Many queries on the functionality were raised to the client. Mean time to resolve 70% of the queries was 2 days. 30% of the queries took more than 6 days to resolve.
The distribution of queries, test cases and reviews across functionality is;
Function/Module No of Queries Review defects Test Cases
Adjustments 18 65 245
Maintaining Batch Transactions & Create new batch 73
40 136
BT adjustments 27 9 234
Convenience Cheques 23 38 23
Customer Service Transactions 6 13 129
Letters & Rates 40 24 68
Auto Login 9 34 56
Credit card payments 23 55 143
Unpaid Tem Fee 45 19 111
Fee Tallies 12 34 98
Miscellaneous Adjustments 8 38 189
Refund Credit Balance 10 34 245
Other 47 136
Distribution of defects category wise across testware is as follows;
Function/Logic (incorrect testware, not as per functional requirements) ;187 defects
Suggestions ; 50
Test Coverage ; 213 defects
Q 2 What analysis would you do and what actions would you take? What goals would you re-visit? Make your assumptions and identify the risks.
End of Test Execution
At end of cycle 1 - The effort variance is 60%. It took a long time for the client to decide whether the defect raised was really a defect. The client had included some new functionality to test. During test execution the link to the client test environment was down.
Defect Trend Test Cases Defects – Severity
Stages Planned Executed Severity 1 Severity 2 Severity 3 Suggestion Total
Smoke Test 50 40 18 37 5 2
Cycle 2 1500 592 23 76 13 5
Cycle 3 1500 789 15 49 12 2
Cycle 4 1500 1979 25 38 11 4
Cycle 5 1979 1979 7 23 9 9
Cycle 6 1979 1979 2 7 0 0
Staging 2746 2768 39 99 31 5
Total 129 329 81 27
Rejected 34 22 35 5
Defect Reported Status Open Fixed Closed Deferred Reject Later Duplicate Total
Severity 1 2 9 34 32 34 2 16
Severity 2 20 20 170 52 22 11 34
Severity 3 3 6 13 4 35 8 12
Suggestion 5 5 10 5 2
Total
Q3. Plot the chart on test cases planned vs executed, % executed and defect trend for all severity levels? Plot the chart of the defect status, find out the % distribution of defect status? What analysis would you do? What actions you should take? What goals would you re-visit?
Defect Trend Test Cases Defects – Severity
Stages Planned Executed Severity 1 Severity 2 Severity 3 Suggestion Total
Smoke Test 50 40 18 37 5 2 370
Cycle 2 1500 592 23 76 13 5 623
Cycle 3 1500 789 15 49 12 2 407
Cycle 4 1500 1979 25 38 11 4 451
Cycle 5 1979 1979 7 23 9 9 194
Cycle 6 1979 1979 2 7 0 0 55
Staging 2746 2768 39 99 31 5 916
Total 129 329 81 27 3016
Rejected 34 22 35 5 485
Rejected % 26% 7% 43% 19% 16%
Defect Reported Status Open Fixed Closed Deferred Reject Later Duplicate Total
Severity 1 2 9 34 32 34 2 16 129
Severity 2 20 20 170 52 22 11 34 329
Severity 3 3 6 13 4 35 8 12 81
Suggestion 5 5 10 5 2 27
Total 25 40 222 98 96 23 62 665
A3 Find out why the defects were rejected, plot defects across test cases and functionality & re-estimate. Find out how many defects did not have corresponding test cases. Set up the defect resolution time with the client. In fact this should have been done during the planning phase. Do a causal analysis to find out the reasons for the rejected defects. For the client added functionality, re-estimate the testing effort and also find out if the test cases have been added for the newly added functionality
EXERCISE B–Testing Project
You are executing an independent system testing project. For system test you have planned 4 cycle of 2 weeks each:
Week Week 1 Week 3 Week 5 Week 7
Cycle Cycle 1 Cycle 2 Cycle 3 Cycle 4
Build Build 1 Build 2 Build 3 Build 4
Functionality being added 2 5 3 0
Build_1 was delivered on time i.e. first day of week 1. This was tested till end of second week. Defects found in Cycle 1 were 50. But Build 2 was delivered in week 4 instead of week 3 and that too was a broken build. After discussing this problem with the client, they have decided to give a working build which will have the functionality of Build 2and 3 together in week 5.
Q1. How will you plan testing of build which will have functionality of Build 2 and 3?
A1. In this scenario you will have 3 Cycles instead of 4 and you will add another cycle in Week 9.
Comments :
Post a Comment