Software Testing & QA

C Langugage interview Qestions and Answers

2009-11-03T19:09:00.000-08:00

1) Name the four basic data types in “C” language?
The four basic data types in “c” language are as follows
(i) Char
(ii) Int
(iii) Float
(iv) Double

2) Describe at least five different format specifiers?
%d: -An integer whole number
%f: -a floating point number
%c: -a single character
%p: -a machine memory address

3) Define and explain scanf () function?
The Scanf () function can be used to get input into a program and it requires two arguments. First a format specifier defines the type of data to be entered, then the name of the variable in which the input will be stored. This scanf () function is responsible for giving input into the program.

4) What are the maximum and minimum possible ranges of values for long and short type?
If the int variable is created by default as a ‘long’ type it typically will have a possible range of values from a maximum of +214748347 and a minimum of -2147483648. For ‘short’ type these are the maximum and minimum values +327676 and minimum -32768.
(While answering this question you can specify the approximate value raised to power).

5) What exactly is a ‘variable scope’, ‘local variables’ and ‘global variables’?
The extent to which a variable is accessible in a program is called the ‘variable scope’.
Variables declared internally inside a function are known as ‘local’ variables.
Variables declared externally outside a function are known as ‘global’ variables.

6) What are signed values?
When an int variable is declared it can by default contain either positive of negative integer values. These are known as ‘signed’ values. The range of positive values is determined by your system.

7) Explain about register variables relative to compiler?
A register variable declaration informs the compiler that the specified variable will be heavily used by the program. The intention is for the compiler to place register variables in the machine register of the computer to speed access times. Local internal variables can only be declared as register variables.

8) State some significant uses of arrays in C programming?
One of the most significant uses of arrays in c programming concerns with their ability to store strings of text. Each element in an array of the char data type can store a single character. Adding the special null character escape sequence in the arrays final element promotes the array to string. It can be referenced using the array name and can be displayed using the %s format specifier.

9) What is enum used for and state its format?
The enum keyword provides a handy way to create a sequence of integer constants in a concise manner. Optionally the declaration can include a name for the sequence after the enum keyword. The constant names follow with a comma and placed within braces. The name assigned can be used to call the function again at later stage. The constants can be assigned any individual value but the following constant will always increment it by one.

10) Explain about the constants which help in debugging?
A #if directive test can be offered with #else and #else if directives. This allows conditional branching of the program to run sections of the code according to the result. Constants defined with a #define directive can be undefined with the #undef directive. The #ifdef directive has a companion directive #ifndef. These commands can be useful when debugging problem code to hide and unhide sections of the program.

11) Define and explain about! Operator?
The logical operator! NOT is a unary operator that is used before a single operand. It returns the inverse value of the given operand so if the variable “c” had a value of true then! C would return value of false. The not operator is very much useful in C programs because it can change the value of variables with successful iterations. This ensures that on each pass the value is changed.

12) What is operator precedence?
Operator precedence defines the order in which C evaluates expressions.
e.g. in the expression a=6+b*3, the order of precedence determines whether the addition or the multiplication is completed first. Operators on the same row have equal precedence.

13) Explain about the functions strcat() and strcmp()?
This function concatenates the source string at the end of the target string. Strcmp() function compares two strings to find out whether they are the same or different. The two strings are compared character by character until there is a mismatch or end of one of the strings is reached, whichever occurs first. If in case two strings are identical, a value of zero is returned. If there is no matches between two strings then a difference of the two non matching values are returned according to ASCII values.

14) Why do we use structures?
We use structures for different purposes which are
1) Construct individual arrays
E.g. 1) one array for storing names, another array for storing prices, and another for storing number of pages.
2) Use a structure variable.
A structure contains a number of data types grouped together. These data types may or may not be of the same type.

15) State the difference between sprint () and sscanf() Functions?
The sprint() function works similar to the printf() function except for one small difference, instead of sending the output to the screen as printf() does, this function writes the output to an array of characters. Once str has been built, its contents can be displayed on the screen.

16) How do you write data to low level disk I/O?
In low level disk I/O, data cannot be written as individual characters, or as strings or as formatted data. There is only one way data can be written to read in low level disk I/O functions as buffer full of bytes. Writing a buffer full of data resembles the fwrite() function.

17) Define these functions fopen(), fread(), fwrite() and fseek()?
Fopen(): - On opening a file, a pointer is set up which points to the first record in the file.
Fread(): - The pointer moves to the next record while reading a file. It always reads the file where the pointer is pointed.
Fwrite(): -The pointer moves to the beginning of the next record on closing the file the pointer is deactivated. This pointer writes the record where the pointer is currently places.
Fseek(): - Function lets us move the pointer from one record to another.

18) State the difference between Unions and structures?
Both structures and unions are used to group a number of different variables together. But while a structure enables us treat a number of different variables stored at different places in memory, a union enables us to treat the same space in memory as a number of different variables.

19) What are the different styles present in WS_OVERLAPPEDWINDOW?
The different styles present in windows are WS_OVERLAPPED, WS_CAPTION, WS_THICKFRAME, WS_MINIMIZEBOX, and WS_MAXIMIZEBOX.
All these macros are #defined in the ‘windows.h’ header file.

20) What are the rules for constructing real constants?
Real constants are often called Floating point constants. The real constants could be written in two forms fractional form and exponential form.
These are the following rules while constructing real constants
1) A real constant must have at least one digit
2) It must have a decimal point
3) It could be wither positive or negative
4) Default sign is positive
5) No commas or blanks are allowed within a real constant.

Oracle Database Interview Questions - 2

2009-10-29T20:39:00.000-07:00

1. Explain the difference between a hot backup and a cold backup and the benefits associated with each.

A hot backup is basically taking a backup of the database while it is still up and running and it must be in archive log mode. A cold backup is taking a backup of the database while it is shut down and does not require being in archive log mode. The benefit of taking a hot backup is that the database is still available for use while the backup is occurring and you can recover the database to any point in time. The benefit of taking a cold backup is that it is typically easier to administer the backup and recovery process. In addition, since you are taking cold backups the database does not require being in archive log mode and thus there will be a slight performance gain as the database is not cutting archive logs to disk.

2. You have just had to restore from backup and do not have any control files. How would you go about bringing up this database?

I would create a text based backup control file, stipulating where on disk all the data files where and then issue the recover command with the using backup control file clause.

3. How do you switch from an init.ora file to a spfile?

Issue the create spfile from pfile command.

4. Explain the difference between a data block, an extent and a segment.

A data block is the smallest unit of logical storage for a database object. As objects grow they take chunks of additional storage that are composed of contiguous data blocks. These groupings of contiguous data blocks are called extents. All the extents that an object takes when grouped together are considered the segment of the database object.

5. Give two examples of how you might determine the structure of the table DEPT.

Use the describe command or use the dbms_metadata.get_ddl package.

6. Where would you look for errors from the database engine?

In the alert log.

7. Compare and contrast TRUNCATE and DELETE for a table.

Both the truncate and delete command have the desired outcome of getting rid of all the rows in a table. The difference between the two is that the truncate command is a DDL operation and just moves the high water mark and produces a now rollback. The delete command, on the other hand, is a DML operation, which will produce a rollback and thus take longer to complete.

8. Give the reasoning behind using an index.

Faster access to data blocks in a table.

9. Give the two types of tables involved in producing a star schema and the type of data they hold.

Fact tables and dimension tables. A fact table contains measurements while dimension tables will contain data that will help describe the fact tables.

10. . What type of index should you use on a fact table?

A Bitmap index.

11. Give two examples of referential integrity constraints.

A primary key and a foreign key.

12. A table is classified as a parent table and you want to drop and re-create it. How would you do this without affecting the children tables?

Disable the foreign key constraint to the parent, drop the table, re-create the table, enable the foreign key constraint.

13. Explain the difference between ARCHIVELOG mode and NOARCHIVELOG mode and the benefits and disadvantages to each.

ARCHIVELOG mode is a mode that you can put the database in for creating a backup of all transactions that have occurred in the database so that you can recover to any point in time. NOARCHIVELOG mode is basically the absence of ARCHIVELOG mode and has the disadvantage of not being able to recover to any point in time. NOARCHIVELOG mode does have the advantage of not having to write transactions to an archive log and thus increases the performance of the database slightly.

14. What command would you use to create a backup control file?

Alter database backup control file to trace.

15. Give the stages of instance startup to a usable state where normal users may access it.

STARTUP NOMOUNT - Instance startup

STARTUP MOUNT - The database is mounted

STARTUP OPEN - The database is opened

16. What column differentiates the V$ views to the GV$ views and how?

The INST_ID column which indicates the instance in a RAC environment the information came from.

17. How would you go about generating an EXPLAIN plan?

Create a plan table with utlxplan.sql.

Use the explain plan set statement_id = 'tst1' into plan_table for a SQL statement

Look at the explain plan with utlxplp.sql or utlxpls.sql

18. How would you go about increasing the buffer cache hit ratio?

Use the buffer cache advisory over a given workload and then query the v$db_cache_advice table. If a change was necessary then I would use the alter system set db_cache_size command.

19. Explain an ORA-01555

You get this error when you get a snapshot too old within rollback. It can usually be solved by increasing the undo retention or increasing the size of rollbacks. You should also look at the logic involved in the application getting the error message.

20. Explain the difference between $ORACLE_HOME and $ORACLE_BASE.

ORACLE_BASE is the root directory for oracle. ORACLE_HOME located beneath ORACLE_BASE is where the oracle products reside.

Well, we have gone through the first 25 questions as I would answer them during an interview. Please feel free to add your personal experiences to the answers as it will always improve the process and add your particular touch. As always remember these are "core" DBA questions and not necessarily related to the Oracle options that you may encounter in some interviews. Take a close look at the requirements for any job and try to come up with questions that the interviewer may ask. Next time we will tackle the rest of the questions. Until then, good luck with the process.

Oracle Database Interview Questions - 1

2009-10-29T20:37:00.001-07:00

Differentiate between TRUNCATE and DELETE
What is the maximum buffer size that can be specified using the DBMS_OUTPUT.ENABLE function?
Can you use a commit statement within a database trigger?
What is an UTL_FILE.What are different procedures and functions associated with it?

Difference between database triggers and form triggers?
What is OCI. What are its uses?
What are ORACLE PRECOMPILERS?
What is syntax for dropping a procedure and a function? Are these operations possible?
Can a function take OUT parameters. If not why?
Can the default values be assigned to actual parameters?
What is difference between a formal and an actual parameter?
What are different modes of parameters used in functions and procedures?
Difference between procedure and function.
Can cursor variables be stored in PL/SQL tables.If yes how. If not why?
How do you pass cursor variables in PL/SQL?
How do you open and close a cursor variable.Why it is required?
What should be the return type for a cursor variable.Can we use a scalar data type as return type?
What is use of a cursor variable? How it is defined?
What WHERE CURRENT OF clause does in a cursor?
Difference between NO DATA FOUND and %NOTFOUND
What is a cursor for loop?
What are cursor attributes?
Difference between an implicit & an explicit cursor.
What is a cursor?
What is the purpose of a cluster?
How do you find the numbert of rows in a Table ?
Display the number value in Words?
What is a pseudo column. Give some examples?
How you will avoid your query from using indexes?
What is a OUTER JOIN?
Which is more faster - IN or EXISTS?
When do you use WHERE clause and when do you use HAVING clause?
There is a % sign in one field of a column. What will be the query to find it?
What is difference between SUBSTR and INSTR?
Which datatype is used for storing graphics and images?
What is difference between SQL and SQL*PLUS?
What is difference between UNIQUE and PRIMARY KEY constraints?
What is difference between Rename and Alias?
What are various joins used while writing SUBQUERIES?

Mediation Process in Telecom

2009-10-29T09:41:00.000-07:00

A billing mediation platform is a system used to convert datatypes of a certain type to other datatypes, usually for billing purposes. They are used mostly by telephone companies, who typically need to process UDRs (Usage Detail Records). In call scenarios it is known as CDR (Call Detail Record), and among broadband carriers it's referred to as IPDR.

This CDR/UDR data formats will consist of data like NPX,NPA,Call Duration,peak time flag,call length and all this data may be represented in binary formats.Mediation Manager will read this data and converts into common normalized format.

Billing Systems and all other Downstreams systems will convert again this data to component[its own] understandable format.

Billing mediation platforms get their name from their behavior; they "mediate" between a variety of other systems. In the typical telephone company scenario, the upstream systems (those providing data to the mediation platform) are network elements, such as telephone switches, and the downstream systems (those receiving data from the mediation platform) perform accounting, auditing, archiving, or bill-generation functions. The mediation system collects, collates and prepares data for consumption by the downstream systems, which often accept data only in a limited set of formats.

Typically a mediation platform is used for the following tasks:

1. Collection and validation of CDRs
2. Filtering out of non billing-relevant CDRs
3. Collating
4. Correlation of different input sources CDRs
5. Aggregation of partial CDRs related to the same call
6. Format change and CDRs normalization
7. Business transformation of data

In a telecom billing scenario, mediation is the first step after receiving a CDR. The mediated CDR is forwarded to a rating engine, which calculates the charge associated with the CDRs.In today's world Rating Engines are more becoming necessary for the telecom billing system to meet the growing variant customer needs for different services.[citation needed]

Despite the name, not all of the data transferred via billing mediation platforms is actually used for billing purposes. For instance, the mediation software might generate traffic volume statistics based on the number and origin of the records passing through it. Those statistics could then be used for capacity planning, as part of a network monitoring procedure, or for any other business intelligence applications.

Sophisticated Billing Mediation software from various providers serves end to end functionality for Telecom Operators. Mediation Software Performs various operation from Collection to Downstream Distribution to Modules like Retail Billing, Interconnect Settlement, Business intelligence, Fraud Detection, Revenue Assurance, Test Call Generation. Following list of activities provides an insight on Mediation software activities

1. Collection and Archive
2. Decoding/Encoding
3. CDR Normalization (Common Format)
4. Filtering
5. Conversion
6. Validation
7. Record Enrichment (Using Complex Reference Data)
8. Duplicate Record Detection
9. Aggregation or Correlation
10. Buffering
11. Cloning
12. Sorting
13. Downstream Format Mapping
14. Header and Trailer generation
15. Downstream Distribution
16. Error Messaging and Alarms
17. Auditing and Reports
18. Reconciliation
19. Reference Data Configuration
20. Provisioning services for the subscription.

Complementary to Billing Mediation functions, comprehensive mediation platforms also provide functionality dedicated to Service Provisioning (the two areas frequently intermix as services configured and used by the end customer result in usage data records generation in the network).

Mediating between the systems is not the only job that Mediation Platform can do. Actually this can be used as a provisioning agent. The basic provisioning commands can be configured within the mediation system and whenever we get a request for the system which does the provisioning, the request can be converted into a file , in which mediation can append the service provisioning commands and send it to HLR for activating any request. This of course , load dependent but can come very handy when there is a crisis in the other system.

At core Mediation involves data transfer between various systems with or without modification of data starting Network elements to OSS/BSS systems.Mediation platform for Telecom Practice supports various systems

* Retail Billing
* Wholesale Billing - National and International
* Network Traffic Management
* Data Warehousing
* Reconciliation system
* Fraud Management
* Provisioning Feed to Sub-systems

Telecom operators offer Voice,video,data,fax and internet services to subscribers and partners on various product lines.Mediation products are tuned to provide solutions for complex business challenges.

Some of the main suppliers of Billing Mediation solutions are the following:

Ericsson (Multi Mediation). InTec (Inter-mediatE). Comverse (ComverseONE). Ventraq (EventDynamics).

LoadRunner Interview questions and Answers -2

2009-10-27T10:19:00.001-07:00

Q42:What is the difference between HTML mode and URL mode?

HTML based mode, records script for every user action that is performed during recording (hmmm…sounds like QTP) while URL based mode records each and every browser request to the server and resources received from the server. Confused? ok, HTML based mode does recording as you perform clicks and doesn’t give you inside information like what is happening behind the recording while URL based mode records each and every step and emulate Javascript code.

From the point1) above you can guess, HTML mode would have less correlation to do while URL mode has much more complex correlation requirements.
HTML mode is smaller and is more intuitive to read as the statements are inside the functions corresponding to the user action performed. In the case of URL based, all statements gets recorded into web_url()
HTML mode is recommended for browser applications while URL mode is recommended for non-browser applications.
Lastly, don’t get the impression that I am advocating for HTML mode . URL mode can be of real help when you want to have control over the resources that need to be or need not to be downloaded, since you have each and every statement in-front of you (point 1)

Q43:What is the difference between concurrent users and simultaneous users?

All the vusers in a particular scenario are called Concurrent vusers. They may or may not perform the same tasks. On the other hand simultaneous vusers is more to do with rendezvous points. When we set rendezvous points we instruct the system to wait till a certain no of vusers arrive so that they all can do a particular task simultaneously. These vusers performing the same task at the same time are called Simultaneous vusers.
For example in a Yahoo Mail application: Suppose a scenario consists of 100 vusers with 3 tasks – 1) Login, 2) Check no of unread mails 3) Logout. Vusers at 1) + 2) + 3) will be called as concurrent vusers as they are part of same scenario performing some task but if have set a rendezvous point so that say 25 vuser perform the 2) task at the same time these 25 vusers would be termed as simultaneous vusers.

Q44:What is the difference between process and thread?

Process is defined as the virtual address space and the control information necessary for the execution of a program while Threads are a way for a program to split itself into two or more simultaneously running tasks. In general, a thread is contained inside a process and different threads in the same process share some resources while different processes do not.
Source
In terms of Loadrunner, when we run Vuser as a process, LoadRunner creates 1 process called mmdrv.exe per Vuser. So if we have 10 Vusers, we will have 10 mmdrv.exe processes on our machines.
while when we run Vuser as a thread, LoadRunner creates 1 thread per Vuser. So if we have 10 Vusers, then we will have 1 process with 10 threads running inside it if the limit is 10 threads per process.
Running Vuser as a thread is more memory efficient that running Vuser as a process for obvious reasons that less memory resources are utilized when we run them as thread. I read somewhere that running as a process has an advantage that system becomes more stable. Now how is that stability achieved? I am not sure…Can someone clarify this point to me?
Note that while running Java Virtual User, JVM’s are loaded per process and will follow the same principle above.

Q45:What are the new features in LoadRunner 9.5?

Ans:

-->LoadRunner noobs always struggle finding out the protocol behind a given application. It seems HP has noted the fact and have come out with a cool new feature called Protocol Advisor. It sits inside VUGen. Protocol Advisor helps you to determine an appropriate protocol for recording a Vuser script. The Protocol Advisor scans your application for elements of different protocols and displays a list of the detected protocols. These protocols can provide a starting point for finding the optimal protocol for your application.
-->Now you have the facility to export the Test Result from VUGen scripts to HTML and open Quality Center defects directly from the report.
-->HP Service Test is now fully integrated with LoadRunner. You simply need to buy a license for service test to use its capabilities. [for the uninitiated, HP Service Test automates the testing process for services (SOA). Read more about it in this document.
-->LoadRunner 9.5 supports Windows Vista SP1. (LoadRunner 9.1 only supported generators while LR9.5 provides support for both generator as well as controller). To use LR9.5 on Vista, make sure that User Account Control (UAC) is enabled for users with User privileges.
-->LoadRunner 9.5 provides support upto .NET Framework 3.5 version and supports .NET and Web Services protocols.
-->LoadRunner is now integrated with a 3rd party software Shunra, to provide you access to WAN emulation. Read the complete list of features here.
Now you can define a Post Collate Command inside the Controller that it will run directly after it collates the results of a scenario run.
-->LR controller has a new utility called Host Security Manager utility that will allow you to create secure communication channels between the LoadRunner Controller and load generators.
-->There is a new Analysis API that will allow you to create an analysis session from test results, analyze raw results of an Analysis session, and extract key session measurements for external use.
-->LR analysis now supports SQL Server 2005.
-->LoadRunner 9.5 no longer supports Windows 2000.
-->LR 9.5 is compatible with Quality Center(QC) 9.2, 10.0 and QTP 9.5 and 10.0 versions.

Q46: What is the purpose of using HP - LoadRunner?
In real world scenario, it is not possible to create situation involving say one thousand users using a system simultaneously so as to test the behavior of the system under such stressful conditions. LoadRunner can create such a situation.
LoadRunner artificially simulates several thousand users - which are called Virtual Users. These artificial / digitally created users are simultaneously forced to operate on the same task, thereby loading the system in a way it is expected to be loaded by real human users in actual practice.
With LoadRunner we can simulate situation with hundreds or thousands of artificial users & we can impose definite, consistent and repeatable loads on the system thereby stressing it from end-to-end. This way we can emulate several business processes & production conditions, which a deployed application is going to encounter.
LoadRunner accurately measures, monitors, and analyzes a system’s performance and functionality.

Q.47: What are the essential capabilities we look in a typical application performance-testing Tool?
Essential capabilities of an application performance testing tool are that:
1) It must be able to test a system which combines many software applications and hardware platforms.
2) It must be able to determine the suitability of a server for any given application.
3) It must be able to test the server before the necessary client software has been developed.
4) It must be able to emulate an environment where multiple clients interact with a single server application.
5) It must be able to test an application under the load of tens, hundreds, or even thousands of potential users.

LoadRunner interview questions and Answers -1

2009-10-27T08:28:00.000-07:00

Q1.How to get current system time?

A. Method#1: This function is developed to usein Mercury Load Runner peformance tool.

This main use of this functions to return the current system time at any given

point of time while load runner script is running.This functiona can be used to

report transaction times , script starti time and end time.

long get_secs_since_midnight(void)
{
char * curr_hr; /* pointer to a parameter with current clock hr */
char * curr_min; /* pointer to a parameter with current clock min*/
char * curr_sec; /* pointer to a parameter with current clock sec */
long current_time, /* current number of seconds since midnight */
hr_secs, /* current hour converted to secs */
min_secs, /* current minutes converted to secs */
secs_secs; /* current number of seconds */

curr_hr = lr_eval_string("{current_hr}>");
curr_min = lr_eval_string("{current_min}");
curr_sec = lr_eval_string("{current_sec}");

hr_secs = (atoi(curr_hr)) * 60 * 60;
min_secs = (atoi(curr_min)) * 60;
secs_secs = atoi(curr_sec);

current_time = hr_secs + min_secs + secs_secs;

return(current_time);
}

Method#2:
lr_save_datetime( Today is H M S DATE_NOW curr_time );
lr_output_message(lr_eval_string( {curr_time} ));

Method#3:
ou can utilise the parameter value for "Date/Time" where Vugen can extract the date for you depending on the format you want. This is much simpler in writing codes.

Method#4:

typedef long time_t;

Action()

{

time_t t;

lr_message("Time in seconds since 1/1/70: ldn" time(&t));

lr_message("Formatted time and date: s" ctime(&t));

return 0;

}

Q2:why we insert the rendezvous point while running the scenario.

Ans#1 : It is an interrupt point to stop current VUser script execution untill remaining VUsers scripts also executed upto that point.It maintains ' Load Balancing ' while running VUsers script.

Ans#2:Rendezvous word is come from Greek. Renduvous means Meeting Point .

consider one scenario that contain 4 Actions.

1. Download the site

2. Login

3. Cash withdraw

4. Logout.

100 Vusers perfrom the actions one by one. but we wants emulate the heavy load at Case withdraw action. So we will insert rendezvous point at Cash withdraw action. Vusers will wait at cash withdraw action once all the users will arive at cash withdraw then controller release all users simultaneously.

Q3:How do you identify the scenarios to be tested in performance testing for web application?

Ans:soon it will be answer(this a frequent question from many companies)

Q4:What is load testing?

- Load testing is to test that if the application works fine with the loads that result from large number of simultaneous users, transactions and to determine weather it can handle peak usage periods.

Q5:What is Performance testing?

- Timing for both read and update transactions should be gathered to determine whether system functions are being performed in an acceptable timeframe. This should be done standalone and then in a multi user environment to determine the effect of multiple transactions on the timing of a single transaction.

Q6:Did u use LoadRunner? What version?

- Yes. Version 8.xx or 9.xx(now a days,use it as 9.xx)
Explain the Load testing process? -
Step 1: Planning the test. Here, we develop a clearly defined test plan to ensure the test scenarios we develop will accomplish load-testing objectives. Step 2: Creating Vusers. Here, we create Vuser scripts that contain tasks performed by each Vuser, tasks performed by Vusers as a whole, and tasks measured as transactions. Step 3: Creating the scenario. A scenario describes the events that occur during a testing session. It includes a list of machines, scripts, and Vusers that run during the scenario. We create scenarios using LoadRunner Controller. We can create manual scenarios as well as goal-oriented scenarios. In manual scenarios, we define the number of Vusers, the load generator machines, and percentage of Vusers to be assigned to each script. For web tests, we may create a goal-oriented scenario where we define the goal that our test has to achieve. LoadRunner automatically builds a scenario for us. Step 4: Running the scenario.
We emulate load on the server by instructing multiple Vusers to perform tasks simultaneously. Before the testing, we set the scenario configuration and scheduling. We can run the entire scenario, Vuser groups, or individual Vusers. Step 5: Monitoring the scenario.
We monitor scenario execution using the LoadRunner online runtime, transaction, system resource, Web resource, Web server resource, Web application server resource, database server resource, network delay, streaming media resource, firewall server resource, ERP server resource, and Java performance monitors. Step 6: Analyzing test results. During scenario execution, LoadRunner records the performance of the application under different loads. We use LoadRunnerÃ¢â‚¬â„¢s graphs and reports to analyze the applicationÃ¢â‚¬â„¢s performance.

Q7:When do you do load and performance Testing?

- We perform load testing once we are done with interface (GUI) testing. Modern system architectures are large and complex. Whereas single user testing primarily on functionality and user interface of a system component, application testing focuses on performance and reliability of an entire system. For example, a typical application-testing scenario might depict 1000 users logging in simultaneously to a system. This gives rise to issues such as what is the response time of the system, does it crash, will it go with different software applications and platforms, can it hold so many hundreds and thousands of users, etc. This is when we set do load and performance testing.

Q8:What are the components of LoadRunner?

- The components of LoadRunner are The Virtual User Generator, Controller, and the Agent process, LoadRunner Analysis and Monitoring, LoadRunner Books Online.

Q9:What Component of LoadRunner would you use to record a Script? - The Virtual User Generator (VuGen) component is used to record a script. It enables you to develop Vuser scripts for a variety of application types and communication protocols.

Q10:What Component of LoadRunner would you use to play Back the script in multi user mode?
- The Controller component is used to playback the script in multi-user mode. This is done during a scenario run where a vuser script is executed by a number of vusers in a group.

Q11:What is a rendezvous point?

- You insert rendezvous points into Vuser scripts to emulate heavy user load on the server. Rendezvous points instruct Vusers to wait during test execution for multiple Vusers to arrive at a certain point, in order that they may simultaneously perform a task. For example, to emulate peak load on the bank server, you can insert a rendezvous point instructing 100 Vusers to deposit cash into their accounts at the same time.

Q12:What is a scenario? - A scenario defines the events that occur during each testing session. For example, a scenario defines and controls the number of users to emulate, the actions to be performed, and the machines on which the virtual users run their emulations.

Q13:Explain the recording mode for web Vuser script? - We use VuGen to develop a Vuser script by recording a user performing typical business processes on a client application. VuGen creates the script by recording the activity between the client and the server. For example, in web based applications, VuGen monitors the client end of the database and traces all the requests sent to, and received from, the database server. We use VuGen to: Monitor the communication between the application and the server; Generate the required function calls; and Insert the generated function calls into a Vuser script.

Q14:Why do you create parameters?

- Parameters are like script variables. They are used to vary input to the server and to emulate real users. Different sets of data are sent to the server each time the script is run. Better simulate the usage model for more accurate testing from the Controller; one script can emulate many different users on the system.

Q15:What is correlation? Explain the difference between automatic correlation and manual correlation?

- Correlation is used to obtain data which are unique for each run of the script and which are generated by nested queries. Correlation provides the value to avoid errors arising out of duplicate values and also optimizing the code (to avoid nested queries). Automatic correlation is where we set some rules for correlation. It can be application server specific. Here values are replaced by data which are created by these rules. In manual correlation, the value we want to correlate is scanned and create correlation is used to correlate.

Q16:How do you find out where correlation is required? Give few examples from your projects?

- Two ways: First we can scan for correlations, and see the list of values which can be correlated. From this we can pick a value to be correlated. Secondly, we can record two scripts and compare them. We can look up the difference file to see for the values which needed to be correlated. In my project, there was a unique id developed for each customer, it was nothing but Insurance Number, it was generated automatically and it was sequential and this value was unique. I had to correlate this value, in order to avoid errors while running my script. I did using scan for correlation.

Q17:Where do you set automatic correlation options?

- Automatic correlation from web point of view can be set in recording options and correlation tab. Here we can enable correlation for the entire script and choose either issue online messages or offline actions, where we can define rules for that correlation. Automatic correlation for database can be done using show output window and scan for correlation and picking the correlate query tab and choose which query value we want to correlate. If we know the specific value to be correlated, we just do create correlation for the value and specify how the value to be created.

Q18:What is a function to capture dynamic values in the web Vuser script?

- Web_reg_save_param function saves dynamic data information to a parameter.

Q19:When do you disable log in Virtual User Generator, When do you choose standard and extended logs?

- Once we debug our script and verify that it is functional, we can enable logging for errors only. When we add a script to a scenario, logging is automatically disabled. Standard Log Option: When you select
Standard log, it creates a standard log of functions and messages sent during script execution to use for debugging. Disable this option for large load testing scenarios. When you copy a script to a scenario, logging is automatically disabled Extended Log Option: Select
extended log to create an extended log, including warnings and other messages. Disable this option for large load testing scenarios. When you copy a script to a scenario, logging is automatically disabled. We can specify which additional information should be added to the extended log using the Extended log options.

Q20: How do you debug a LoadRunner script?

- VuGen contains two options to help debug Vuser scripts-the Run Step by Step command and breakpoints. The Debug settings in the Options dialog box allow us to determine the extent of the trace to be performed during scenario execution. The debug information is written to the Output window. We can manually set the message class within your script using the lr_set_debug_message function. This is useful if we want to receive debug information about a small section of the script only.

Q21 : How do you write user defined functions in LR? Give me few functions you wrote in your previous project?

- Before we create the User Defined functions we need to create the external
library (DLL) with the function. We add this library to VuGen bin directory. Once the library is added then we assign user defined function as a parameter. The function should have the following format: __declspec (dllexport) char* (char*, char*)Examples of user defined functions are as follows:GetVersion, GetCurrentTime, GetPltform are some of the user defined functions used in my earlier project.

Q22:What are the changes you can make in run-time settings? - The Run Time Settings that we make are: a) Pacing - It has iteration count. b) Log - Under this we have Disable Logging Standard Log and c) Extended Think Time - In think time we have two options like Ignore think time and Replay think time. d) General - Under general tab we can set the vusers as process or as multithreading and whether each step as a transaction.

Q23:Where do you set Iteration for Vuser testing?

- We set Iterations in the Run Time Settings of the VuGen. The navigation for this is Run time settings, Pacing tab, set number of iterations.
How do you perform functional testing under load? - Functionality under load can be tested by running several Vusers concurrently. By increasing the amount of Vusers, we can determine how much load the server can sustain.

Q24:What is Ramp up? How do you set this?

- This option is used to gradually increase the amount of Vusers/load on the server. An initial value is set and a value to wait between intervals can be
specified. To set Ramp Up, go to ‘Scenario Scheduling Options’

Q25:What is the advantage of running the Vuser as thread?

- VuGen provides the facility to use multithreading. This enables more Vusers to be run per
generator. If the Vuser is run as a process, the same driver program is loaded into memory for each Vuser, thus taking up a large amount of memory. This limits the number of Vusers that can be run on a single
generator. If the Vuser is run as a thread, only one instance of the driver program is loaded into memory for the given number of
Vusers (say 100). Each thread shares the memory of the parent driver program, thus enabling more Vusers to be run per generator.

Q26 : If you want to stop the execution of your script on error, how do you do that?

- The lr_abort function aborts the execution of a Vuser script. It instructs the Vuser to stop executing the Actions section, execute the vuser_end section and end the execution. This function is useful when you need to manually abort a script execution as a result of a specific error condition. When you end a script using this function, the Vuser is assigned the status "Stopped". For this to take effect, we have to first uncheck the option “Continue on error in Run-Time Settings.

Q27: What is the relation between Response Time and Throughput?
- The Throughput graph shows the amount of data in bytes that the Vusers received from the server in a second. When we compare this with the transaction response time, we will notice that as throughput decreased, the response time also decreased. Similarly, the peak throughput and highest response time would occur approximately at the same time.

Q28:Explain the Configuration of your systems?

- The configuration of our systems refers to that of the client machines on which we run the Vusers. The configuration of any client machine includes its hardware settings, memory, operating system, software applications, development tools, etc. This system component configuration should match with the overall system configuration that would include the network infrastructure, the web server, the database server, and any other components that go with this larger system so as to achieve the load testing objectives.

Q29 : How do you identify the performance bottlenecks?

- Performance Bottlenecks can be detected by using monitors. These monitors might be application server monitors, web server monitors, database server monitors and network monitors. They help in finding out the troubled area in our scenario which causes increased response time. The measurements made are usually performance response time, throughput, hits/sec, network delay graphs, etc.
If web server, database and Network are all fine where could be the problem? - The problem could be in the system itself or in the application server or in the code written for the application.

Q30:How did you find web server related issues?

- Using Web resource monitors we can find the performance of web servers. Using these monitors we can analyze throughput on the web server, number of hits per second that
occurred during scenario, the number of http responses per second, the number of downloaded pages per second.

Q31 : How did you find database related issues?

- By running Ã¢â‚¬Å“DatabaseÃ¢â‚¬Â monitor and help of Ã¢â‚¬Å“Data Resource GraphÃ¢â‚¬Â we can find database related issues. E.g. You can specify the resource you want to measure on before running the controller and than you can see database related issues

Q32:Explain all the web recording options?

Q33:What is the difference between Overlay graph and Correlate graph?

- Overlay Graph: It overlay the content of two graphs that shares a common x-axis. Left Y-axis on the merged graph showÃ¢â‚¬â„¢s the current graphÃ¢â‚¬â„¢s value & Right Y-axis show the value of Y-axis of the graph that was merged. Correlate Graph: Plot the Y-axis of two graphs against each other. The active graphÃ¢â‚¬â„¢s Y-axis becomes X-axis of merged graph. Y-axis of the graph that was merged becomes merged graphÃ¢â‚¬â„¢s Y-axis.

Q34:How did you plan the Load? What are the Criteria?

- Load test is planned to decide the number of users, what kind of machines we are going to use and from where they are run. It is based on 2 important documents, Task Distribution Diagram and Transaction profile. Task Distribution Diagram gives us the information on number of users for a particular transaction and the time of the load. The peak usage and off-usage are decided from this Diagram. Transaction profile gives us the information about the transactions name and their priority levels with regard to the scenario we are deciding.

Q35:What does vuser_init action contain?

- Vuser_init action contains procedures to login to a server.

Q36:What does vuser_end action contain?
- Vuser_end section contains log off procedures.

Q37:What is think time? How do you change the threshold?

- Think time is the time that a real user waits between actions. Example: When a user receives data from a server, the user may wait several seconds to review the data before responding. This delay is known as the think time. Changing the Threshold: Threshold level is the level below which the recorded think time will be ignored. The default value is five (5) seconds. We can change the think time threshold in the Recording options of the Vugen.

Q38:What is the difference between standard log and extended log?

- The standard log sends a subset of functions and messages sent during script execution to a log. The subset depends on the Vuser type Extended log sends a detailed script execution messages to the output log. This is mainly used during debugging when we want information about: Parameter substitution. Data returned by the server. Advanced trace.

Q39:Explain the following functions:

- lr_debug_message - The lr_debug_message function sends a debug message to the output log when the specified message class is set. lr_output_message - The lr_output_message function sends notifications to the Controller Output window and the Vuser log file. lr_error_message - The lr_error_message function sends an error message to the LoadRunner Output window. lrd_stmt - The lrd_stmt function associates a character string (usually a SQL statement) with a cursor. This function sets a SQL statement to be processed. lrd_fetch - The lrd_fetch function fetches the next row from the result set.
Throughput - If the throughput scales upward as time progresses and the number of Vusers increase, this indicates that the bandwidth is sufficient. If the graph were to remain relatively flat as the number of Vusers increased, it would
be reasonable to conclude that the bandwidth is constraining the volume of
data delivered.
Types of Goals in Goal-Oriented Scenario - Load Runner provides you with five different types of goals in a goal oriented scenario:
The number of concurrent Vusers
The number of hits per second
The number of transactions per second
The number of pages per minute
The transaction response time that you want your scenario
Analysis Scenario (Bottlenecks): In Running Vuser graph correlated with the response time graph you can see that as the number of Vusers increases, the average response time of the check itinerary transaction very gradually increases. In other words, the average response time steadily increases as the load
increases. At 56 Vusers, there is a sudden, sharp increase in the average response
time. We say that the test broke the server. That is the mean time before failure (MTBF). The response time clearly began to degrade when there were more than 56 Vusers running simultaneously.

Q40:What is correlation? Explain the difference between automatic correlation and manual correlation?

- Correlation is used to obtain data which are unique for each run of the script and which are generated by nested queries. Correlation provides the value to avoid errors arising out of duplicate values and also optimizing the code (to avoid nested queries). Automatic correlation is where we set some rules for correlation. It can be application server specific. Here values are replaced by data which are created by these rules. In manual correlation, the value we want to correlate is scanned and create correlation is used to correlate.

Q41:Where do you set automatic correlation options?

- Automatic correlation from web point of view, can be set in recording options and correlation tab. Here we can enable correlation for the entire script and choose either issue online messages or offline actions, where we can define rules for that correlation. Automatic correlation for database, can be done using show output window and scan for correlation and picking the correlate query tab and choose which query value we want to correlate. If we know the specific value to be correlated, we just do create correlation for the value and specify how the value to be created.

Q42:What is a function to capture dynamic values in the web vuser script?
- Web_reg_save_param function saves dynamic data information to a parameter.

What is the difference between two formulae mentioned?

2009-09-30T09:51:00.000-07:00

Where the different processes go is dependent upon the rquirements.
(Logon+do_something +logoff)* 20 is different to
logon+(do_something )*20+logoff. It all depends upon whether people logon to
do a specific process and leave or whether they repeat it. Only your SME can
tell you the characteristics needed.

2009-09-22T04:31:00.000-07:00

QA documentation

2009-09-15T05:30:00.000-07:00

TABLES

Table 3-1. Comparison of Requirements
1. OVERVIEW

This study was funded by the United States Nuclear Regulatory Commission (NRC) under interagency agreement RES-91-003 between the NRC and the National Institute of Standards and Technology (NIST). As requested by NRC, this report presents a review of Part 2.7, "Quality Assurance Requirements of Computer Software for Nuclear Facility Applications," ASME NQA- 2a-1990 [ASMENQA2], and a review plan for the analysis of software documentation. The results of this study are presented to NRC for consideration and are not endorsed by NRC.

While this study was conducted relative to the assurance of software for safety systems in nuclear power plants, the results and recommendations contained in this report are applicable to other critical software safety systems. This analysis of documentation and review processes resulted in identifying the issues and tasks involved in software quality assurance (SQA). It also revealed that because most standards are generic (i.e., do not address the needs of a specific industry), tailoring of standards to specific domains must be done to ensure a complete safety evaluation.

This review is intended to provide NRC reviewers with guidelines for evaluating software used in nuclear power plant safety systems. These evaluations are based on software documentation. Utilities submit vendor products to NRC for approval at the end of product development, and sometimes during development. NRC may conduct audits during the development process. NRC reviewers evaluate these products based on the documentation, reviews, and reports supplied by the vendor. They check for compliance with product plans, safety requirements, and with recommended standards and guidelines for software development and assurance. NRC may be asked to review other types of software products, which may be only one component of a system, or NRC may examine an entire system, consisting of software and other components. The recommendations in this report apply only to the software documentation aspects of the safety evaluation.

The purpose of this report is to aid the NRC reviewers during safety evaluations. However, this report also includes an appendix which discusses formal reviews conducted at milestones (e.g., at completion of a software lifecycle phase or when necessary to discuss a serious concern). The formal review is conducted by the purchaser and developers of software; this type of review is also appropriate within a utility when it develops the software. Other types of reviews can also be conducted by individuals who prepare reports on products for discussion at the formal review meetings or who, as in the case of NRC reviewers during a safety evaluation, examine the product, its technical documentation and project documentation.

Although the frame of reference for this study includes only safety systems in nuclear power plants, this report includes recommendations for documentation and reviews in a broader context to ensure completeness. While some software safety systems may be small, other computer systems in the nuclear industry may be larger, may have more complexity and interfaces, and may access or generate large databases. NRC reviewers should have a comprehensive understanding of a complete quality program, regardless of the size and characteristics of the product under review. Depending on the particular software product, NRC reviewers may decide that not all the recommendations in this report are appropriate or may add additional safety- related review items to improve the coverage of safety issues.

Section 2 of this report includes background information on the uses of [ASMENQA2], documentation, and reviews. Section 3 details the contents of [ASMENQA2] and cites its omissions. Section 4 identifies the necessary types of documents, and provides suggestions for their content. Section 5 provides recommendations for NRC reviewers conducting safety evaluations. Appendix A contains definitions of quality attributes. Finally, Appendix B provides information on the formal review process (which may assist NRC during safety evaluations).

1.1. Acronyms

Acronyms used in this report (other than those used to refer to documents) are listed below.

O&M Operation and Maintenance
PMP Project Management Plan
SCM Software Configuration Management
SCMP Software Configuration Management Plan
SDD Software Design Description
SQA Software Quality Assurance
SQAP Software Quality Assurance Plan
SRS Software Requirements Specification
SV&V Software Verification and Validation
SVVP Software Verification and Validation Plan
SVVR Software Verification and Validation Report
2. BACKGROUND INFORMATION

[ASMENQA2] may be used by the vendor to ensure quality in the product, or by the NRC reviewer in safety evaluations. When [ASMENQA2] is the designated governing document, the vendor is not obligated to perform any SQA activity not specifically identified, even if recognized in other standards or documented practices.

The vendor reviews documentation from a proactive perspective and must adjust processes and software products based on review findings. For the vendor, an SQA plan (SQAP) must provide significant detail, or reference detailed requirements. For consistency within the nuclear industry, the SQAP should be based on a rigorous standard. If NRC is monitoring development of a product, then NRC roles are also proactive.

When NRC reviewers evaluate a completed product for approval for use within a nuclear power plant, the NRC reviewer is reactive. The reviewer must understand the development and assurance processes that were applied to the product. To do this, the reviewer examines documentation of the processes to seek evidence that the vendor has fulfilled all the applicable requirements for software in safety systems for nuclear power plants. The reviewer will make this determination based in part on standards and practices adopted or recommended for use by the nuclear industry.

[ASMENQA2] addresses requirements for lifecycle activities, documentation, reviews, and several SQA activities such as records management and procurement procedures. Vendors may use [ASMENQA2] to plan the software quality program. For this reason, this report identifies additional practices, documentation and reviews that may be applicable.

NRC reviewers should be aware of the total spectrum of issues for software quality. Thus, this report provides general recommendations for documentation and review. In some instances, these recommendations may not be applicable to software safety systems in nuclear power plants. NRC should recommend that vendors justify why a particular document, specific contents of a document, or an activity, are omitted. The justification may be a simple statement (e.g., database documentation is not provided because there is no database).

Documentation serves as a communication medium informing all involved persons of their responsibilities and the technical specifications related to the product. Documentation also serves as a communication medium for those who build the product, monitor the project, and review the products and processes of the project. Reviewers during development will examine product documentation to learn how the product is being built, while the reviewer at final product evaluation examines how the product was built. The reviewer during development examines plans and interim products to determine whether or not the planned assurance and development activities are sufficient and the evolving product is meeting its assurance criteria. The reviewer at final product evaluation will attempt to verify that appropriate engineering practices were used, assurance activities were properly applied, problems found during development and validation have been corrected, and the resulting product meets its specifications.

This report provides recommendations for reviews by NRC and provides discussion on reviews by the vendor. In vendor reviews, the primary purpose is to make immediate decisions based on the results of the review. NRC is not present for these reviews. However, in response to a utility's request, NRC may attend reviews or examine project documentation at some point during development; in this case NRC's role is proactive. Descriptions of these reviews are included in this report to aid NRC reviewers' understanding of procedures that should have been performed during development. For the safety review by NRC, the procedures and types of questions are different than those of the formal reviews conducted by the vendors. However, understanding and the results of the development (formal) reviews may be important aids to NRC reviewers.

[ASMENQA2], its analysis, and all recommendations in this report are based on the waterfall lifecycle model. Readers should be aware that the recommendations in this report may not be totally applicable when other lifecycle models, CASE tools, modern development methodologies, and state-of-the-art concepts for documentation (e.g., hypertext) are used.

3. CONTENTS OF ASME'S QUALITY ASSURANCE STANDARD

[ASMENQA2] has been tailored from [IEEE1012] and [IEEE7300], standards on software verification and validation plans (SVVPs) and SQAPs, respectively. [ASMENQA2]'s title indicates that it has been written specifically to provide SQA requirements for developers in the nuclear industry. However, only one section addresses issues specific to nuclear facilities (SV&V section mentions (un)intended functions). Table 3-1 contains recommended SQA activities and documentation and a comparison of [ASMENQA2] and [IEEE7301] relative to these recommendations. [IEEE7301], the most current version of the IEEE SQAP standard, is used because it is more complete than previous versions. While Table 3-1 indicates that both standards address most topics, it does not show the extent of the requirements for each topic. This section provides an overview of [ASMENQA2]'s general contents.

[ASMENQA2] states that while any software lifecycle model may be used, the model must encompass the activities associated with the model of [IEEE1012] for requirements, design, implementation, test, installation and checkout, and operation and maintenance (O&M). [ASMENQA2] addresses other lifecycle processes, which are referred to as assurance activities in [NIST204, NUREG5930]. One feature of [ASMENQA2] that may be confusing is the fact that it runs through the lifecycle three times. First, it describes the activities of each lifecycle phase. It then lists the documentation for each phase. Finally, it addresses the reviews of each lifecycle phase.

[ASMENQA2] identifies lifecycle activities and some assurance activities but does not necessarily require documentation of these activities. [IEEE7301] identifies basic SQA documentation requirements without requiring lifecycle activities. However, since [IEEE7301] is a standard describing the content of SQAPs, it is acceptable that it does not explicitly describe SQA activities. [ASMENQA2] cites some documentation requirements from [IEEE7301], identifies lifecycle activities, and specifies reviews on some development products. It does not specify reviews of all lifecycle activities.

If a standard requires an activity, the standard should contain requirements for the documentation, review, and monitoring of the activity. The inclusion of statements such as the following would be useful: each lifecycle activity shall be documented; reports shall be prepared on each activity's implementation; an SQA review shall verify that the lifecycle activity plan is acceptable and was accomplished (or if not, why not); tracking mechanisms shall provide information to project management and reviewers.

[ASMENQA2] and the IEEE standards are intended for specific projects, and therefore do not address a vendor's quality management program. A plan based on [ASMENQA2] may reference the vendor's general quality management program (e.g., one based on [ISO9000]).

Requirements [ASMENQA2] [IEEE7301]
Assurance Activities
SCM
SV&V

yes
yes

N/A
N/A
SQA Plan Elements
organization for quality responsibilities
identification of applicable software products
standards, practices, conventions
metrics
reviews
audits
documentation
error reporting and corrective action
access control
procurement/supplier
records management
training
tools, techniques, methodologies
risk management

yes
yes
yes
no
yes
no
yes
yes
yes
yes
yes
no
yes
no
yes
yes
yes
yes
yes
yes
yes
yes
no
yes
yes
yes
yes
yes
Required Documentation
Project Management Plan
SQA Plan
SCM Plan
Configuration Identification Documents
Configuration Change Control Documents
Configuration Status Accounting Documents
SCM Audits and Reviews Documents
SV&V Plan
SV&V Reports (includes Test Reports)
Software Requirements Specification
User's Manual
Software Design Description
Source Code Documents
Installation and Checkout Document
O&M Manual
Maintenance Documents

no
yes
no
no
yes
yes
no
yes
yes
yes
yes
yes
no
no
no
yes
no
yes
yes
yes
yes
yes
no
yes
yes
yes
yes
yes
Stds & Procedures Manual
no
no
no
Required Reviews
Management Reviews
Software Requirements Review
Software Design Review
Source Code Review
Test Report Review
Development Documentation Review
Installation & Checkout Review

none
yes
yes
no
no
yes
no
SVVP, SCMP
yes
yes, 2 reviews
no
no
no
no
Table 3-1. Comparison of RequirementsProject Management

Project Management

Project management is clearly indicated as a basis for software development maturity [SEI91], but neither [ASMENQA2] nor [IEEE7301] require a project management plan (PMP). The argument for its omission may have been that the PMP specifies SQA activities; thus an SQAP would not include requirements for a PMP. However, an SQA standard, especially one that has already been tailored for a particular use, should be complete in identifying all the documentation and activities that comprise a quality project. When NRC reviews the products during development, the NRC reviewers should review the PMP to assess potential capability of the vendor to satisfy the requirements, and ensure that the plans for development were reasonable to begin with. NRC reviewers at final product evaluation should also examine the project management documentation to ensure that the project has been well-managed and that all essential elements of the project have been addressed.

Software Quality Assurance

[ASMENQA2] does not address measurement, the use of metrics for measuring quality. Measurement of the software development process and product is valuable to the developing organization. NRC reviewers should be interested in the vendor's reasons for changing processes and technical specifications. The SQAP should identify what measurements are to be taken, when they are to be examined, by whom, their purpose (e.g., to determine if there is a problem, to evaluate effect of change in development process, to examine effectiveness of testing, inspection or other analysis techniques) and who has authority to make changes in any of the processes.

Information on measurement in the SQAP is helpful to NRC reviewers. For example, periodic checks of test results (e.g., number of errors found) may reveal an error rate outside the control limits. A reviewer may discover that this is due to the fact that inspections were eliminated when the project fell behind schedule. When NRC is involved during development, actions can be recommended to correct the situation. At final product review, if the NRC reviewers have access to problem report data, they can question the reason for the higher error rate and may use this information to make judgments about the fitness of the final product.

[ASMENQA2] does not address risk management. Since the nuclear community in general must perform system hazard analysis this may be acceptable. However, a citation for risk management in a "software" standard may assure that software risks, or potential hazards, are to be identified from the system hazard analysis reports, and that these risk areas receive extra attention from the SQA activities.

[ASMENQA2] does not specify required audits. The audits identified in [IEEE7301] are conducted during development, in a contractual environment. NRC has the option to conduct audits during the development process, and at final safety evaluation. At the audits, many of the typical audit questions will be asked at this time, but from a reactive perspective.

[ASMENQA2] does not address training staff to conduct SQA activities. Again, [ASMENQA2] is intended to be a plan for a specific project; training may be covered in the vendor organization's general quality management program. For completeness, however, the SQAP may cite any special training for the project. With this knowledge from the plan, NRC reviewers can form expectations regarding the product quality.

Software Configuration Management

[ASMENQA2] identifies several major software configuration management (SCM) activities and requires documentation of some of these activities. For configuration change control, [ASMENQA2] states that "changes to software shall be formally documented," and for configuration status accounting, "the information that is needed to manage a configuration shall be documented." However, there is no requirement for an SCM plan (SCMP) or other SCM documents (e.g., documentation of SCM audits and reviews). For any project involving more than one person, an SCMP is needed to communicate how SCM will be implemented. The absence of SCM documentation can become significant when a reviewer at final product evaluation tries to identify whether the software products under evaluation are actually those products which were verified and validated.

Software Verification and Validation

[ASMENQA2]'s documentation requirements for SV&V include most of the requirements from [IEEE1012], but does not describe them in detail. SV&V documents such as the SVVP and SV&V reports are not specifically mentioned, although some contents of these documents are identified (e.g., description of SV&V tasks, criteria for accomplishing these tasks, results of execution of SV&V activities, test results). Requirements for the content of SV&V documentation should be more detailed in order to ensure that the intent of the requirements are fulfilled. SV&V must be planned, managed, and documented even for small projects.

Software Test

Testing is part of the SV&V process. Preparation and planning for the execution of testing occurs throughout the lifecycle, while the execution itself occurs more or less in a phase. Although [IEEE1012] provides detailed requirements for each major category of software testing (component, integration, system, acceptance), [ASMENQA2] treats all testing generically. This can be a problem because each of the major test types has different objectives. Reviewers during development need to verify that the objectives are appropriate and sufficient for product assurance, and reviewers at final product evaluation need to verify that those objectives have been met.

Acceptance test, which is the final set of validation activities for a product, is not addressed by [ASMENQA2]. In some environments, acceptance tests are planned and performed by the customer. In other environments, the vendor, with customer approval, plans and conducts acceptance tests. NRC should recommend to the nuclear industry that documentation for acceptance test (e.g., plans, procedures, results) should be submitted in the package for NRC review of safety features.

Software Requirements

[ASMENQA2] has tailored [IEEE1012] for a specific use. A major weakness of [IEEE1012] and [IEEE7301] is their lack of emphasis on the relationship of software to the system in which it resides and to software safety and computer security requirements. This report attempts to address these concerns.

[ASMENQA2] is directed at the software only. Software safety systems may be embedded in hardware, and the typical software requirement specification (SRS) may not define how the software fits into the total system. This information should be provided in the SRS. While it is acceptable that the concept phase of [IEEE1012] is not addressed in [ASMENQA2], some information may need to be provided that usually is discussed before deciding to build a product (in the concept phase). For example, some functions of a software safety system may have previously been performed by non-software components. The rationale for making the transition from hardware to software could be important in helping NRC reviewers assess the completeness of the software requirements and whether they have addressed all potential hazards impacting software.

[ASMENQA2] identifies the minimum content of the SRS which includes a description of functionality, performance, external interfaces, design constraints, and quality attributes. Requirements for safety, computer security, and user documentation are missing. Requirements specific to software safety need to be identified (e.g., trace to test cases, trace through the system documentation) to assure adequate attention to these requirements throughout assurance activities.

Software Design and Implementation

[ASMENQA2] specifies general design requirements; however, no consideration is given to security or safety. [ASMENQA2] does not address source code documentation.

Installation and Checkout

[ASMENQA2] requires documentation of the approval of the software for operational use.

Operation and Maintenance

[ASMENQA2] requires that all modifications made to the software during the O&M phase should be documented.

Development Documentation Review

[ASMENQA2] requires review of development documentation, but does not define what development documentation includes. A vendor can easily claim compliance to this requirement by reviewing almost any documentation, but the intent of the requirement may not be fulfilled. Section 4 discusses development documentation.

4. DOCUMENTATION RECOMMENDATIONS

Project documentation may include many kinds of documents (e.g., plans, task reports, development products, problem reports, phase summary reports). Project size, criticality (i.e., the severity of the consequence of failure of the system), and complexity are some features that may affect the amount of documentation a project should need. For example, the design documentation may consist of a single document describing both the system architecture and the detailed modules or it may consist of separate documents for the architecture and subsystems. The purpose of this section is not to specify how many documents should be required. Rather, this section identifies the information content needed for any project and the timeliness of requirements so that the information can be used by the vendor, the utility, and the NRC reviewers. Because the NRC reviewers cannot determine the characteristics of the software product without substantial technical specifications, project plans, and reports, NRC should specify the technical products of the vendor that the utility must provide NRC.

This report expands upon the documentation requirements in [ASMENQA2]. While minimum content requirements are provided for several documents (which include the [ASMENQA2] requirements), some documents may be presented together as a single document. Each of the planning documents may be kept simple and short, but the basic information identified in the recommendations should be addressed. Issues not addressed include the mechanics of document preparation, the medium of the documents (e.g., paper, disk), and requirements for modifying or updating documents. The impact of modern methodologies (e.g., information engineering) on documentation is also not addressed.

4.1. Project Management Documentation

PMP's should include project organization, methods, tools, reporting, assurance activities, cost and scheduling estimates. Some of this information may be included in the SQAP. The vendor may have a generic PMP but every project will have some specific information which must be provided. The following recommendations for a PMP are based on [IEEE1058]. PROJECT MANAGEMENT PLAN

Introduction.
Project Overview. Summary of the project and its goals, the product, major work activities, major work products, major milestones, required resources, and master schedule and budget. Statement of requirements for assurance (e.g., SQAP, SCMP, SVVP).
Project Deliverables. Description of all items used by the developer and items delivered ultimately to the customer.
Evolution. Procedures for reviewing and producing updates to the PMP.
References, Definitions, and Acronyms.
Project Organization.
Process Model. Relationships between project activities. Specification of timing of major milestones, baselines, reviews, work products, project deliverables, and sign-offs. Definition of project initiation and termination activities.
Organization Structure. Description of the internal management structure including authority, responsibility, and communication.
Organizational Boundaries and Interfaces. Project boundaries with other entities (customer, subcontractors, other system components). Interfaces of SCM, SQA and SV&V.
Project Responsibilities. Description of major project activities and who has responsibility for each activity.
Managerial Process.
Management Objectives and Priorities. The philosophy, goals, and priorities for management activities such as reporting mechanisms; priorities among requirements, schedule, and budget; risk management procedures; and, procedures relating to existing software.
Assumptions, Dependencies, and Constraints. Assumptions the project is based upon, events on which the project is dependent, and constraints the project is to be conducted under.
Risk Management. Project risk factors including contractual, technological (e.g., ability of technology to support the safety requirements), size/ complexity/criticality, personnel acquisition and retention, and customer acceptance risks. Mechanisms for tracking the risk factors and implementing contingency plans.
Monitoring and Controlling Mechanisms. Tools and techniques used in monitoring and controlling adherence to the PMP including reporting mechanisms, report format, information flows, and review and audit mechanisms.
Staffing Plan. Types and numbers of personnel assigned, required skill levels, and training necessary.
Technical Process.
Methods, Tools and Techniques. Methods, tools, and techniques used to conduct project activities. References to standards, policies and procedures used.
Software Documentation (may be included here or in SQAP). The documentation requirements, milestones, baselines, reviews, and sign-offs.
Project Support Functions. Reference to (or include here) the SCMP, SQAP, and SVVP. Responsibilities, resource requirements, schedules, and budget for the SCM, SQA, and software verification and validation (SV&V).
Work Packages, Schedule and Budget.
Work Packages. Description of the work packages (i.e., task grouping) for the project activities.
Dependencies. Interdependencies among work packages and dependencies on external events.
Resource Requirements. Estimates of project resources including numbers and types of personnel, computer time, support software, computer hardware, office and laboratory facilities, travel and maintenance requirements.
Budget and Resource Allocation.
Schedule.
Additional Components.
Reference to additional items such as subcontractor management, security for the project, independent V&V, training, facilities, installation, or product maintenance plans.
4.2. Software Quality Assurance Documentation

The recommendations for the SQAP are listed in Table 3-1 and are discussed in section 3.

4.3. Software Configuration Management Documentation

There are several SCM documents that should be produced. The SCMP should describe how the SCM activities (configuration identification, configuration control, status accounting) will be conducted and documented. The other documents report on the SCM activities. The SCMP may exist as a generic plan which can be referenced by any project. Even on a small project it is easy to lose track of the relationship between tested software modules and modules in which software units have been changed. The following recommendations for an SCMP are based on [IEEE828]. [IEEE828] is used not only because of its merits but also because it is serving as the base document for an international standard in ISO/IEC JTC1 SC7 on Software Engineering.

SOFTWARE CONFIGURATION MANAGEMENT PLAN

Introduction. Overview of the SCM process, the purpose and scope of the SCMP.
Management. Procedures for establishing SCM organization, and for assigning tasks/responsibilities to all units in the organization.
SCM Activities. Includes configuration identification, configuration control, configuration status accounting and report, and audits/reviews. Include in the description how these activities are to be implemented, which units are responsible for which activities, etc.
Tools, Techniques, and Methodologies. Tools and methodologies to be used for implementing SCM activities.
Records Collection and Retention. Procedures for identifying, assembling, filing, storing, maintaining, and disposing of SCM documentation.
SCMP Maintenance. How the SCMP will be reviewed, updated, or revised while in use.
The following recommendations for the other SCM documents are based on [IEEE828].

CONFIGURATION IDENTIFICATION DOCUMENTS. Items that comprise a baseline and the procedures for identification of configuration items.

CONFIGURATION CHANGE CONTROL DOCUMENTS. Changes to the software. A description of the change, the rationale for the change, and the identification of affected baselines.

CONFIGURATION STATUS ACCOUNTING DOCUMENTS. Information that is needed to manage a configuration item including the following information: status of specifications; status of proposed changes; status of product versions or revisions; reports of approved changes; and, reports of the implementation of installed updates or releases.

SOFTWARE CONFIGURATION MANAGEMENT AUDITS AND REVIEWS DOCUMENTS.

4.4. Software Verification and Validation Documentation

For large or long term projects, the SVVP and the detailed test documentation should be separate documents. For small projects, while the initial planning and SVVP should be completed during the requirements phase, additional information regarding the SV&V tasks, test designs, case, and procedures may be added to the SVVP later. There are several types of SV&V documents that should be produced. The following recommendations for an SVVP are based on [IEEE1012].

SOFTWARE VERIFICATION AND VALIDATION PLAN

Objectives. The objectives of performing SV&V should be stated. Sometimes thorough SV&V on every software requirement, design feature, and line of code may not be possible and specific objectives must be selected (e.g., to show that critical performance objectives have been met; to examine software interfaces to the system, especially during failures external to the software).
Management Activities. Review of V&V Plan, review of SV&V results, monitoring of SV&V activities, policies for iteration of tasks, etc.
Schedule of SV&V Tasks. Includes milestones and completion dates.
Organizations. Organizations involved in the SV&V activities and specific responsibilities.
Communication. Communication procedures between design team members, code team members, SV&V team members, and others involved in the project.
Summary of Resources. Staffing, tools, facilities, and finances needed to perform SV&V tasks, etc.
Facilities. Planning of facilities, especially tools for test activities (e.g., simulation, test case generation) that require acquisition, staff training, and test. Planning and providing for test facilities may take as long as software development itself, so adequate time should be allotted. Delayed facility planning may result in inadequate test strategies and execution.
Testing. Distinction among the different types of tests, their objectives, and documentation.
Lifecycle SV&V. Detailed descriptions of SV&V tasks throughout lifecycle. Identifies the following:
SV&V tasks and objectives for each lifecycle phase (e.g., the specific safety objectives as a focus of each task).
Methods and criteria used in performing the SV&V tasks.
Inputs and outputs required for each task.
Schedule for SV&V tasks.
Resources for performing SV&V tasks.
Assumptions concerning schedule, resources, or approach.
Responsibilities for performing the tasks.
SV&V Reporting. Description of how results of implementing the SVVP will be documented.
Anomaly Reporting and Resolution. Description of method for reporting and resolving anomalies, and tracking anomaly resolution.
Policy for Reiterating a SV&V Task. Description of criteria used to determine the extent to which a SV&V task should be reperformed when input is changed.
Policy for Deviating from the Approved Plan. A good reviewer during final product evaluation may pick up some differences between planned and actual activities and ask many questions about the deviation. If procedures were in place and were followed, vendor may be able to demonstrate that the deviation did not detract from quality of the software, or its SV&V.
Configuration Control Procedures. Configuration management procedures on the documentation examined by SV&V tasks; if this information is included only in a SCMP, then reference to the SCMP should appear in the SVVP.
The following recommendations for other SV&V documents are based on [IEEE1012].

TASK REPORTS. Report on the status, interim results, or final results of a single well-defined task performed during a lifecycle phase. May include information about the relative efficiencies and difficulties of certain tasks, which may lead to adjustments in the development process, such as choice of methodologies or tools, reallocation of SV&V resources, alteration of schedules, etc.

SOFTWARE VERIFICATION AND VALIDATION PHASE SUMMARY REPORTS. Summary of interim or final results of all SV&V tasks performed during a lifecycle phase. Should be issued for each lifecycle phase. If SV&V tasks are performed by different organizations, the phase summary report will help to consolidate the evaluations of the different organizations.

ANOMALY REPORTS. The location, impact, cause, criticality, and possible recommendations for each anomaly. Should be generated throughout the lifecycle. Procedures for reporting anomalies might include using key words to characterize each anomaly, severity scales to assess its significance, and procedures for reviewing, approving, and dispatching of the reports.

SOFTWARE VERIFICATION AND VALIDATION FINAL REPORT. A summary of all SV&V tasks, summary of task results, summary of anomalies and resolution, assessment of overall software quality, and recommendations. This summary may be issued at the end of Installation and Checkout Phase or at conclusion of SV&V effort. It can be used as the baseline for maintenance, to help improve the development process for the next project, and to call attention to outstanding unresolved anomalies.

4.4.1.Test Documentation

Although test plans for component, integration, and system testing are usually separate documents, for small projects they may be presented together in one document. In these cases, the document should contain separate, complete sections for each type of testing, since the objectives and test strategies for each type of testing will still differ. If more than one organization is responsible for testing (e.g., developer performs component and integration tests, but an independent agent performs system tests), then separate documentation should be maintained. Such an arrangement should be specified in the SVVP. The following recommendations for test documentation are based on [NASA2100], [DOD2167A], and [FUJII].

TEST PLANS. Includes: objectives and requirements; basic strategy (e.g., how each type of testing will satisfy specific objectives; an example may be to state what design modules will undergo control flow analysis or data flow analysis); location and schedule of briefings, formal testing, and debriefings; facilities/tools; staff/resources; tentative schedule; review of test cases and procedures; configuration management, procedures for releasing test results; data reduction and analysis; and, review of test results.

TEST DESIGNS/CASES. Includes: objectives of classes of tests (e.g., logic tests, coverage, boundary tests); traceability to requirements; constraints; initialization; termination; inputs; expected outputs; criteria for evaluating results; and, interfaces exercised.

TEST PROCEDURES. Description of all test execution procedures including: equipment preparation; support software preparation; and, detailed test procedures (test scenario description; step by step instructions to perform test; actions required by equipment; action to perform in case of error; data reduction/analysis procedures).

TEST REPORT. Summarizes test execution and results. Includes: test records (date, time, location of test; test team members, witnesses; problems encountered, recovery procedures attempted, outcome of attempt); test results; and, evaluation and recommendations.

4.5. Requirements Documentation

4.5.1.Software Requirements Documentation

There may be functionality in the system whose sole purpose is to prevent a safety hazard from occurring. Such functions need special attention throughout the software development, especially in its documentation. The same holds for computer security requirements; if computer security is not an issue for a product, the SRS should include a statement to that effect. In an SRS that requires a quality attribute, a quantified description of that attribute should be included in the SRS; for characteristics of quality attributes see [SOFTENG]. The following recommendations for an SRS are based on [NASA2100], [ASMENQA2], NIST180, [IEEEP1059], [IEC880], and [IEEE830].

SOFTWARE REQUIREMENTS SPECIFICATION

The purpose of the SRS is to provide a description of the software. However, the SRS should also describe the relationship of the software component with the rest of the system. The description may include a graphical representation and an overview description of the services the software provides for the system. However, this information is not sufficient for any reviewer to judge the capability of the software to fulfill the overall system requirements.

Functionality. Functions the software is to perform, and how inputs are transformed into outputs.
Functions. Description of all operations, equations, mathematical algorithms, logical operations, etc. Safety and security functions should be flagged.
Inputs. Definition of inputs: specify sources of inputs, types, formats, units of measure, timing, and ranges of valid inputs, including accuracies and tolerances, disposition of illegal values, error messages.
Outputs. Definition of outputs: specify destinations of outputs, units of measure, timing, range of valid outputs, including accuracies and tolerances, disposition of illegal values, error messages.
Data Handling. Requirements for database or dictionary.
Performance. Specification of each performance requirement in testable and quantitative terms. The following should be addressed:
Timing and sizing requirements.
Sequence and timing of events, including user interaction tolerances.
Throughput and capacity requirements.
Error detection, isolation, and recovery requirements for data and processes.
Fail-safe requirements.
Quality Attributes. Non-time-related requirements that software must meet such as usability, efficiency, reliability, maintainability, and portability.
Design/Implementation Constraints. Constraints imposed externally (e.g., technical standards, marketing department), hardware limitations (e.g., use of a specific compiler, limits on memory), and by project requirements (e.g., safety features).
External Requirements.
Interfaces (to other systems within or outside nuclear power plant). User, hardware, software, and communications interfaces. The purpose, requirements (e.g., performance, safety, security), and implementation constraints for each interface. For example, requirements for user interface may include screen descriptions and report descriptions.
Environment. Operating conditions (e.g., plant commissioning, refueling).
Safety Requirements.
Identification of the stable modes and error conditions of the software.
Software requirements identified from hazard analysis.
User Interactions (e.g., requirements for signals that indicate safety problems and for information that notifies user of the status of other safety functions).
Security Requirements. Includes access limitations to the system (e.g., log-on procedures and passwords), data protection and recovery methods. These requirements should be prioritized. Security issues may also be identified by hazard analyses.
User Documentation. Description of requirements for content of user documentation.
Man-Machine Specifications. Description of requirements for appropriate and timely signals between system and operator, automatic system actions to prevent danger or damage, etc.
4.5.2.User Documentation

[ASMENQA2] identifies the minimum content for user documentation including a description of the following: user interaction with the software, training, input and output specifications and format, system limitations, anticipated errors and user response, and user and maintenance support. However, requirements concerning security and safety (e.g., threats, how to override problems, how to respond to warning signals) are not specifically addressed. The following recommendations for user documentation are based on [NASA2100], [ASMENQA2], and [DOD2167A].

USER'S MANUAL

Overview of Purpose and Functions. The purpose and main capabilities of the software, and its overall operation in terms of functions, options, and restrictions and limitations.
Installation and Initialization. The procedures for installing, tailoring, and initiating the software including the following:
Equipment setup.
Power-on and power-off.
Bootstrap and load.
Initiation commands.
Initialization of files, variables, or other data.
Tailoring, reconfiguration, adaptation.
Restart. Procedures for restarting software after initialization failures, abnormal termination.
Termination. Procedures for terminating operation, and determining whether normal termination has occurred.
Recovery Steps. Recovery procedures in terms of: check points; collection of failure data; restoring files; and, restoring devices to operational mode.
Inputs.
System inputs. Description of system inputs to the software that may occur while the software is in use and may affect the software's interface with the user (e.g., inputs from a remote sensor). Include applicable attributes of the input such as format, frequency, effect on the software state or mode, allowable range, and units of measure.
User inputs. Description of user inputs including commands, data, and option selection.
Outputs. Description of expected outputs of the software including results of functions, and error messages.
Potential Hazards. Description of user response to unexpected events or problems that could cause serious hazards if user does not respond; description of activities that could cause the need for user response or override.
Functions and their Operations. Description of each function in terms of: purpose of function; step-by-step procedures for execution; and, related functions.
Error Messages. Error messages output by the software, the meaning of each error message, and the corresponding corrective actions to be taken.
4.6. Design and Implementation Documentation

In general, design rules will also be implemented by the source code (e.g., the modularity features and complexity of the design should transfer to the source code). It is important to ensure that the vendor's plans for modularity and complexity are stated in the design documentation and the modularity and complexity of the design are not lost during implementation. This report addresses design and implementation/source code in two separate documents.

4.6.1.Design Documentation

The following recommendations for a design document are based on [NASA2100], [DOD2167A], [APPGUIDE], P1059, and [ASMENQA2].

SOFTWARE DESIGN DESCRIPTION

Architectural Design.
The role of the software within the system and the relationships between this software and other software components.
Assumptions about the environment including operating system, user interface, program, data management, data interchange, graphics, and network services, especially assumptions on which safety functions and computer security needs may be based.
Architectural Design Description. The design of the software including:
Logical or functional decomposition.
Description of the modules including mention of the safety and computer security functions.
Design of the modules in terms of execution control and data flow including flagging safety and security functions.
Design requirements/constraints on the modules.
Relationships and interactions between the modules.
A schema for the allocation of functions to modules.
Logical data design - conceptual schema.
Entity/data identification and relationships.
Timing and sequencing.
Implementation constraints.
Each state and mode the software operates in and the modules that execute in each state and mode.
Execution control and data flow between modules in the different states and modes.
Memory and processing time allocation to the modules.
External Interface Design. Allocation of the software's external interface requirements to modules. The design of each interface in terms of:
Information description.
Initiation criteria.
Expected response.
Protocol and conventions.
Error identification, handling and recovery.
Queuing.
Implementation constraints.
Requirements relative to safety and computer security.
Software requirements allocation to modules.
Database description. Reference should be made to the document describing the database, its platform, dependencies, etc., or described it here.
Human interaction. Screens, types of alert mechanisms.
Tracking to any other systems in the plant that have safety functions.
Detailed Design.
Modules Design and Traceability to Architectural Design. The design of the software into its modules and traceability between the architectural design modules and software modules. The traceability with flags on safety features and computer security requirements. Descriptions of the modules should include:
Inputs and outputs.
Functions.
Data descriptions and relationships.
Diagrams.
Control and signal flow.
Error handling/messages.
Interfaces between modules.
Packaging details (placement of modules).
Flags on safety, computer security functions.
Detailed Design of Modules. Design information necessary to code the modules. The information should include:
Detailed design to the lowest level.
Functions or operations.
Algorithms.
Specific data definitions including data conversions.
Local and global data.
Parameters for initiation and adaptation.
Logic flow (control flow; timing variations; priority assignments; interrupt priorities and handling).
Error detection and handling.
Physical data design (internal schema; query language; access method; key, record, and data element definition and structure).
Device interface.
Interrupts and signals.
Limitations that restrict performance of modules.
External Interface Detailed Design. The traceability of the external interface design of the software into its modules. A description of each external interface including:
Type and purpose.
Description of the data transmitted across the interface including purpose, source and destination, data type, data representation, size, units of measure, limit/range, accuracy, precision/resolution.
Messages transmitted and the assignment of data elements to each message.
Priority of interfaces and messages transmitted across them..
Protocol (fragmentation and reassembly of messages; error control and recovery procedures; synchronization; flow control; data transfer rate and minimum transfer rate; routing, addressing and naming conventions; transmission services; status, identification, notification and other reporting features; security).
Description of user responses to system/software safety/security violations.
Coding and Implementation Notes. Information such as stubs for incremental development and use of compiler options.
Information On Planning, Designing, Executing Tests.
4.6.2.Source Code Documentation

A source code manual should be developed for a project, unless it already exists within the vendor organization, in which case it should be referenced by the SQAP. The source code manual, which may be very general, is used by the programming staff, and possibly also by reviewers when evaluating a product.

SOURCE CODE MANUAL. Description of the standards and practices for coding, including rules on issues such as the format of the code, maintaining the modularity, complexity, data flow of the design, and commenting the code (e.g., explanations of inputs and outputs). Deviations from the rules stated in the manual should be justified in the source code where deviation occurs and in the accompanying documentation.

The purpose of supporting documentation for source code is to describe the source code and enable traceability to requirements, design, and test cases. This documentation is especially useful to the development staff, to new staff members, and to the maintenance staff. Examples of information that might be included in this documentation are the following:

SUPPORTING DOCUMENTATION FOR SOURCE CODE

Explanation of deviations from the design.
Explanation of deviations from the source code manual.
Explanation of deviations from the programming language standard (e.g., the use of a dialect of the language).
Charts showing the relationship of code modules to design components, or a listing of subroutines that support one design module. This is helpful if the design is at a higher level than the code.
Interface information, if the design is at higher level than code.
Identification of the most critical modules for safety and security.
Comments within modules.
Description of how "controlled practices" are used (e.g., use of techniques to offset risks of using pointers) [NIST190].
4.7. Test Documentation

Test documentation is a subsection of section 4.4. Software Verification and Validation.

4.8. Installation and Checkout Documentation

During installation and checkout the software system is installed and tested to ensure that it performs as required in its operational environment. Documentation for this phase should include the approval of the software for operational use and may include the user's manual, if installation and checkout is to performed by the end-user, rather than the vendor. See section 4.5.2 for content of the user's manual. The following is a minimum content for an installation and checkout document based on [IEEEP1059].

INSTALLATION AND CHECKOUT DOCUMENT

Instructions for installation at each site, which accurately and completely represent the actual configuration and are appropriate for the expertise level of the expected installer.
SV&V activities during installation and checkout (see also section 4.4. V&V).
Schedule for installation, if software is to be installed at a large number of sites.
Procedural flowchart or checklist.
4.9. Operation and Maintenance Documentation

O&M documentation should describe procedures (e.g., preparation, monitoring, fault recovery) executed to operate the software, as well as all modifications made to the software during this phase. The following recommendations for the operational procedures manual is based on [NASA2100] and [DOD2167A].

OPERATIONAL PROCEDURES MANUAL

System Preparation and Set-up Procedures.
Power on and off.
Initiation. Description of the following:
Equipment setup and the procedures required for pre-operation.
Procedures necessary to bootstrap the system.
Procedures for loading software and data.
Procedures for initializing files, variables, or other parameters.
Commands typically used during initiation.
Standard Operating Procedures. Detailed description of standard operating procedures including:
Input and output procedures. Description of the following:
Input and output media (e.g., magnetic tape, disk, or cartridge). Explain procedures for reading and writing on these media.
Operating system control language.
Operator procedures for interactive messages and replies (e.g., which terminal to use, passwords, log on/off).
Monitoring procedures. Description of procedures for monitoring software during operation, including:
Applicable trouble and malfunction indications.
Evaluation techniques for fault isolation.
Conditions requiring computer system shutdown.
On-line intervention, abort, and user communications.
Off-line Routine Procedures.
Daily operating procedures (e.g., system back-ups, and logs for maintenance).
Standard safety and security procedures.
On-demand procedures (e.g., in response to user requests).
Fault and Recovery Procedures. Description of automatic and manual procedures to be followed in case of a fault or abnormal condition in the software. Describe the immediate actions and subsequent recovery procedures for every anticipated trouble condition.
Emergency Procedures. Description of procedures to be conducted in emergencies, including:
Procedures for critical system failures.
Environmental emergency procedures (e.g., fires, hurricanes, or thunderstorms).
Safety or security emergency procedures.
Diagnostic Features.
Diagnostic Procedures. Identification of each procedure and its purpose. Also description of the following:
Hardware, software, or firmware necessary to execute the procedures.
Step-by-step instructions for executing the procedure.
Diagnostic messages and the corresponding required action.
Diagnostic Tools. Description of each tool and its application.
The following recommendations for maintenance documents is based on [EWICS2] and [IEEEP1219].

MAINTENANCE DOCUMENTS. Description of changes made to the software as a result of corrective, adaptive, or perfective maintenance. The modifications should be documented as appropriate in new documents or in addenda to development documents (e.g., SRS, SDD). Various documents are needed to record, track, and implement maintenance. The following are examples of documents that may be used to perform maintenance.

Anomaly Report. Issued by the operations staff to initiate corrective maintenance.
Modification Feasibility Request. Request issued by the user management to carry out a feasibility study on a proposed modification. Initiates perfective or adaptive maintenance.
Feasibility Report. Issued to user management to report on results of the feasibility study.
Validated Modification Request. Issued by user management either to reject the proposed modification or to proceed.
Maintenance Action Report. Issued to the user management after completion of the required preventive maintenance detailing the work done.
Revalidation Request. Request to retest the system after completion of the required preventive maintenance, or implementation of a modification.
Revalidation and Configuration Management Report. Issued to user management after the necessary revalidation is performed, so user can decide if system should be returned to service.
4.10.Reporting

The purpose of reporting is to communicate to others information on specific deficiencies in the software or to summarize specific tasks or groups of tasks performed during development (e.g., SQA activities and SV&V activities). These reports should be used during formal reviews that assess the current product and development activities, as well as to improve the development process for future projects. Reports should be generated as necessary (e.g., for each anomaly or at the completion of each major task). There is no set number of required reports; however, the vendor should avoid producing an overabundance of unnecessary reports. Examples of necessary reports for an activity during a particular lifecycle phase include task reports and anomaly reports. Examples of similar reports can be found in section 4.4.

5. RECOMMENDATIONS FOR REVIEW AT SAFETY EVALUATION

This section of the report provides recommendations for the NRC review of the safety features of the product. In order to evaluate the product, the reviewers examine all documentation associated with the product and may test the product. Test activities are outside the scope of this study, and are not discussed in this report. Instead, this report focuses on how to evaluate the product based on its documentation. This discussion deals strictly with the evaluation of the software component of the safety system, and not the whole system, although NRC reviewers may examine all components during the safety evaluation. However, the principles identified in this section can be adapted for system reviews, with the addition of system-oriented details.

At the safety evaluation, reviewers are responsible for assessing that the software meets its software quality objectives (e.g., safety, functionality, reliability, maintainability, reviewability) [SOFTENG]. To perform this evaluation, the reviewers need sufficient information about the product requirements, its development, and its overall quality. At a minimum, the reviewers should have available the following: system requirements/description, system hazard analysis report, project planning documentation (e.g., SQAP, SCMP, SVVP, PMP), development documentation (e.g., SRS, SDD, source code), reports of assurance activities, project reports, and formal review reports.

The reviewers will also need to evaluate the installation package, which consists of installation procedures, installation medium (e.g., magnetic tape), test case data used to verify installation, and expected output from the test cases. In some instances, the product may already be installed in the utility. NRC should request documentation on the results of installation and acceptance testing.

O&M documentation should be provided, even though it does not relate directly to the development of the product. The reviewer should check that the operational procedures manual is adequate to ensure that the user will be able to operate the software product safely and correctly. In some instances, the reviewer may be re-evaluating a previously-approved product, which has undergone extensive maintenance. For these reviews, the vendor should provide the maintenance documentation, which describes all modifications made to the original product during the O&M phase.

An assumption about the NRC review process is that the reviewers may examine some documentation prior to the site visit, where the bulk of the review occurs. The site may be either that of a vendor or utility, depending on the project. If the review occurs at the vendor's site, the reviewer may request interviews with vendor personnel as needed. The review process involves primarily the review of documentation, but not at the same level of detail as the formal reviews conducted during the development process. NRC reviewers focus on how the product was built, on the results of assurance activities, and the tracking of a selected set of inputs through the system (a string check), rather than a detailed examination of the product. NRC reviewers, while interested in the answers to the checklist questions used in formal reviews, are more concerned with how the vendor responded to the findings of the formal reviews. NRC should request results of all vendor/utility assurance activities, and responses to problems. The NRC reviewers may use sections 4 and Appendix B of this report as baselines for examining the documentation. While the reviewers may perform some dynamic or static analyses on the code, discussion of these activities is outside the scope of this task.

The overall purpose of the safety evaluation review (from the software perspective) is to assess how well the product meets its software quality objectives. Some general types of questions regarding the product, such as the following, guide the reviewers:

How thoroughly has the vendor analyzed the safety of critical functions of the software?
How well has the vendor established the appropriateness of the functions, algorithms, and knowledge on which the software is based?
How carefully has the vendor implemented the safety and performance requirements of the software?
If any function of the safety system was performed by previous systems (either hardware, firmware, or software), is the software substantially equivalent to the predicate device?
The types of questions listed in the remainder of this section provide guidelines for the safety evaluation and are not a complete list. At any point, the reviewer may find it necessary to explore a topic in more detail, and may refer to the checklists in Appendix B. The checklists for this section are based on [HHSFDA], [IEC880], [EWICS2], [ANS104], and [PARNAS].

Requirements and Design

Does the SRS identify safe and unsafe operating states?
Does the SRS identify potentially unsafe operational procedures?
Does the SRS identify measures for monitoring potentially unsafe situations?
What are the safety features and the rationale for them?
Are the safety requirements consistent with the hazard analysis?
Do the safety requirements address both environmental influences and random component failures?
Have the different classes of users been considered?
Are the enforced constraints (legal, company-related) included in the safety criteria?
Does the SRS describe constraints between hardware and software?
Were the functional requirements analyzed for intrinsic hazards?
Are functional requirements classified according to safety criticality?
Does the SRS identify requirements for redundant controls between electro-mechanical and software controls?
Are there requirements for self supervision of the software? Do they meet safety requirements?
Is there modularity in the design?
Have critical components been isolated?
Software Development Methodologies

Were formal methods used? If so, how much experience does the vendor have with formal methods?
Were design reviews performed and documented?
How was failure analysis of the software conducted? What methods were used (e.g., FMECA)?
If automated tools were used, were they thoroughly tested for adequacy before use?
Was a high level language with a tested translator used? Installation Package
Does the package contain sufficient materials to permit rebuilding and testing of the installed program?
Are the installation procedures clearly understandable? Are the format and content of the tape properly identified?
Can the program be rebuilt from the installation package?
Do the test cases produce results identical with the expected outputs?
Operational Procedures Manual

Does the manual adequately describe standard operating procedures, as well as emergency procedures, and fault and recovery procedures?
Does the manual adequately describe diagnostic procedures and tools?
Maintenance

Are all modifications made to the software during maintenance documented?
Is there a formal procedure to start maintenance when problems are found?
Is there a standard procedure to treat all modification requests?
Is there a common modification request document for all disciplines?
Is there a formal change control procedure?
Are modification tools (e.g., diagnostic tools, configuration management tools) addressed, if any are used?
Assurance Activities

Project Management:

Were reviews of project status and assurance task reports conducted?
Based on metrics data, were any processes found to be outside statistical control limits? If so, what actions were taken?
SV&V:

What analysis techniques were used (e.g., control flow analysis, inspection)?
Were the test strategies consistent with the requirements and were they adequate to evaluate the design?
Were the test strategies sufficient to determine whether the software is safe and effective?
How were the safety features of the software tested and analyzed?
Do the results of testing and analysis demonstrate conformance to requirements established at each level (i.e., module, integration, system)?
Do test results show that safety related functions were tested adequately and that they perform reliably?
SCM:

What SCM facilities were used?
At what point was the first SCM baseline established?
What evidence is there that the software delivered is the software that has been verified and validated?
SQA:

What metrics were used, if any?
What actions were taken in response to the metrics results?
Do SQA reports indicate that standards were appropriately followed?
What error analysis techniques were used? How did vendor respond to the findings?
Were software design and code inspections performed? If so, were the identified problems corrected?
Were technical reviews held, and problems identified in the reviews satisfactorily resolved in the product?
6. SUMMARY

[ASMENQA2] addresses activities for software lifecycle phases from software requirements to software maintenance, as well as some assurance activities, but does not necessarily require documentation or review of all these activities. Some SCM documentation is identified, but there is no requirement for an SCMP. Requirements for quality management, project management, and SQA elements including measurement, risk management, SQA audits and training are also not addressed. Although [ASMENQA2] does require documentation of many lifecycle and assurance activities (e.g., SVVP, SQAP, SCM documentation), the requirements are either very general or incomplete.

Because [ASMENQA2] is based on IEEE's first SQA standard, IEEE Std 730-1984, which was written six years before [ASMENQA2] and has since been revised, it fails to address current issues, such as software safety and computer security. The omission of requirements for project management, an SCMP, and metrics, as stated above may also be attributed to this fact. Since [ASMENQA2] does not specify unique assurance requirements for safety and security, this report identifies documentation requirements that emphasize these issues, as well as other important issues that are not addressed by [ASMENQA2]. This report also identifies review issues concerning software properties that may affect the safety of the total system.

One feature of [ASMENQA2] that may be confusing is its order of presentation. First, it describes the activities of each lifecycle phase. It then lists the documentation for each phase. Finally, the reviews of each lifecycle phase are addressed. Although no preference for any particular format is specified, one which may be more logical is to provide only one section for each lifecycle phase or assurance activity, and to specify the requirements for the activity, its documentation, and its review all in the same section.

Another feature of [ASMENQA2] that may be of concern is its lack of requirements specific to nuclear applications. Although its title indicates that it is an SQA standard intended for nuclear facility applications, [ASMENQA2] does not seem to be tailored for this purpose. Rather, it resembles a general SQA standard that could be applied to any software system.

For each software life cycle activity, this report provides recommended content for documentation of the activity. The activities include not only the development phases from software requirements to software maintenance but also project management, SCM, SV&V (including testing), and SQA.

While this report concentrates on activities for the software lifecycle, it also identifies some issues that are of special interest when the software lifecycle is embedded within the system lifecycle. One issue concerns information from the system requirements phase that are essential for the software development phases. Another issue concerns the scope and definition of software verification and validation within system verification and validation.

This report distinguishes between the formal reviews conducted during development and those conducted by NRC. The generic procedures for formal development reviews are provided in Appendix B, with detailed lists of questions for specific reviews. For these reviews, it is assumed that their purpose is to enact changes to the development process and product under review. For reviews conducted by NRC, it is assumed that these concentrate more on how the product was built, on the results of assurance activities, and the tracking of a selected set of inputs through the system (a string check), rather than a detailed examination of the product. Hence, questions for NRC reviewers are at a much higher level of abstraction than for development reviews, but the NRC reviewers may request the results of the vendor's formal reviews and may also use the detailed checklists to support their reviews as appropriate.

Although the required scope for this study includes only safety systems in nuclear power plants, the recommendations for documentation and reviews are presented in a broader context to ensure completeness. While the recommendations may be applied to systems of varying size, the purpose in expanding the scope is to provide NRC reviewers with a comprehensive understanding of what comprises a quality project. The important issue is not the exact number of documents, but rather the content of the different types of documentation. For small projects, the documents may be combined, but the necessary content should be retained. Thus, depending on the product under review, NRC may decide that not all the recommendations are appropriate and may add additional safety-related review items to improve the coverage of safety issues.

7. REFERENCES

[ANS104]
ANSI/ANS-10.4-1987, "Guidelines for the Verification and Validation of Scientific and Engineering Computer Programs for the Nuclear Industry," American Nuclear Society, May 13, 1987.
[APPGUIDE]
Application Portability Profile (APP) The U.S. Government's open System Environment Profile OSE/1 Version 1.0, The National Institute of Standards and Technology, NIST SP 500-187, 1991.
[ASMENQA2]
ASME NQA-2a-1990 Part 2.7, "Quality Assurance Requirements for Nuclear Facility Applications," The American Society of Mechanical Engineers, November 1990.
[BIRRELL]
Birrell, N. and Ould, M., A Practical Handbook For Software Development, Cambridge University Press, 1985.
[DOD2167A]
DOD-STD-2167A, "Defense System Software Development," United States Department of Defense, 29 February 1988.
[DUNN]
Dunn, Robert, Software Quality: Concepts and Plans, Prentice Hall, 1990.
[EWICS2]
Redmill, F. J. (ed.), Dependability of Critical Computer Systems 2, The European Workshop on Industrial Computer Systems Technical Committee 7 (EWICS TC7), Elsevier Science Publishers LTD, 1989.
[FUJII]
Fujii, Roger U., Dolores R. Wallace, Taz Daughtrey, "Software Verification and Validation Seminar," IEEE Standards Seminar, 1990.
[HHSFDA]
"Reviewer Guidance for Computer Controlled Medical Devices Undergoing 510(k) Review," Office of Device Evaluation, Center for Devices and Radiological Health, Food and Drug Administration.
[IEC880]
IEC 880, "Software for Computers in the Safety Systems of Nuclear Power Stations," International Electrotechnical Commission, 1986.
[IEEEP1059]
IEEE Std P1059, "(DRAFT 6.3) IEEE Guide for Software Verification and Validation Plans," The Institute of Electrical and Electronics Engineers, Inc., December 26, 1991.
[IEEEP1219]
IEEE Std P1059, "Standard for Software Maintenance," The Institute of Electrical and Electronics Engineers, Inc., January 31, 1992.
[IEEE610]
ANSI/IEEE Std 610.12-1990, "Glossary of Software Engineering Terminology," The Institute of Electrical and Electronics Engineers, Inc., 1990.
[IEEE828]
ANSI/IEEE Std 828-1983, "IEEE Standard for Software Configuration Management Plans," The Institute of Electrical and Electronics Engineers, Inc., 1984.
[IEEE830]
ANSI/IEEE Std 830-1984, "IEEE Guide for Software Requirements Specifications," The Institute of Electrical and Electronics Engineers, Inc., 1984.
[IEEE1012]
ANSI/IEEE Std 1012-1986, "IEEE Standard for Software Verification and Validation Plans," The Institute of Electrical and Electronics Engineers, Inc., November 14, 1986.
[IEEE1028]
ANSI/IEEE Std 1028-1988, "IEEE Standard for Software Reviews and Audits," The Institute of Electrical and Electronics Engineers, Inc., 1989.
[IEEE1058]
ANSI/IEEE Std 1058.1-1987, "IEEE Standard for Software Project Management Plans," The Institute of Electrical and Electronics Engineers, Inc., 1988.
[IEEE7300]
ANSI/IEEE Std 730-1984, "IEEE Standard for Software Quality Assurance Plans," The Institute of Electrical and Electronics Engineers, Inc., 1984.
[IEEE7301]
ANSI/IEEE Std 730.1-1989, "IEEE Standard for Software Quality Assurance Plans," The Institute of Electrical and Electronics Engineers, Inc., October 10, 1989.
[ISOSC7]
ISO/IEC(JTC1)-SC7, "Information Technology Software Life-Cycle Process," August 30, 1991.]
[ISO9000]
ISO 9000, "International Standards for Quality Management," International Standards Organization, ISO Central Secretariat, Casepostale 56, CH-1211, Geneve 20, Switzerland, May 1990.
[NASA2100]
NASA-STD-2100-91, "NASA Software Documentation Standard - Software Engineering Program," National Aeronautics and Space Administration, July 29, 1991.
[NIST190]
NIST Special Publication 500-190, "Proceedings of the Workshop on High Integrity Software; Gaithersburg, MD; Jan. 22-23, 1991," U.S. Department of Commerce/National Institute of Standards and Technology, August 1991.
[NIST204]
NIST Special Publication 500-204, "High Integrity Software Standards and Guidelines," U.S. Department of Commerce/National Institute of Standards and Technology, September 1992.
[NUREG5930]
NUREG/CR-5930, "High Integrity Software Standards and Guidelines," U.S. Nuclear Regulatory Commission, September 1992.
[PARNAS]
Parnas, D., "Active Design Reviews: Principles and Practice," IEEE International Conference on Software Engineering, London, 1985, p. 132-136.
[SEI91]
Paulk, Mark C., Bill Curtis, and Mary Beth Chrissis, "Capability Maturity Model for Software," Software Engineering Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, 15213, CMU/SEI-91-TR-24, ESD-TR-91-24, August, 1991.
[SOFTENG]
"Standard for Software Engineering of Safety Critical Software," Draft, Rev. 0, Ontario Hydro, December 1990.
[STARTS]
STARTS Purchasers Group, "The STARTS Purchasers' Handbook," NCC Publications, October 1987.
APPENDIX A. DEFINITIONS OF QUALITY ATTRIBUTES

The following definitions include direct quotes from [IEEE610] and [SOFTENG].

compatibility
(1) The ability of two or more systems or components to perform their required functions while sharing the same hardware or software environment. (2) The ability of two or more systems or components to exchange information.
completeness
The degree to which all of the software's required functions and design constraints are present and fully developed in the SRS, SDD, and code.
consistency
The degree of uniformity, standardization, and freedom from contradiction among the documents or parts of a system or component. See also TRACEABILITY.
correctness
(1) The degree to which a system or component is free from faults in its specification, design, and implementation. (2) The degree to which software, documentation, or other items meet specified requirements. (3) The degree to which software, documentation, or other items meet user needs and expectations, whether specified or not. (4) The ability of the SRS, SDD, and code to describe or produce the specified outputs when given the specified inputs, and the extent to which they match or satisfy the requirements.
feasibility
The degree to which the requirements, design, or plans for a system or component can be implemented under existing constraints. modifiability Quality attribute that refers to the characteristics of the SRS, SDD, and code which facilitate the incorporation of changes.
modularity
The degree to which a system or computer program is composed of discrete components such that a change to one component has minimal impact on other components.
predictability
The degree to which the functionality and performance of the software are deterministic for a specified set of inputs.
robustness
The degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental conditions.
structuredness
The degree to which the SDD and code possess a definite pattern in their interdependent parts. This implies that the design has proceeded in an orderly and systematic manner (e.g., top-down), has minimized coupling between modules, and that standards control structures have been followed during coding resulting in well structured software.
testability
(1) The degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met. (2) The degree to which a requirement is stated in terms that permit establishment of test criteria and performance of tests to determine whether those criteria have been met.
traceability
(1) The degree to which a relationship can be established between two or more products of the development process. (2) The degree to which each element in a software development product establishes its reason for existing (e.g., the degree to which each element in a bubble chart references the requirement that it satisfies). (3) The ability to trace the design decision history and reasons for these changes. See also CONSISTENCY.
understandability
The degree to which the meaning of the SRS, SDD, and code are clear to the reader.
verifiability
The degree to which the SRS, SDD, and code have been written to facilitate verification using both static methods and testing.
APPENDIX B. FORMAL REVIEWS OF THE DEVELOPER

The purpose of these reviews is to enact changes to the development process and product under review. For reviews conducted by NRC, the purpose is to concentrate more on how the product was built, on the results of assurance activities, and on tracking of behavior of a selected string, or slice, through the product rather than a detailed examination of the product. Hence, questions for NRC reviewers are at a much higher level of abstraction than for development reviews, but the NRC reviewers may use the detailed checklists.

Some standards that address SQA assume a contractual environment or an environment that can be treated contractually [IEEE7301, ISOSC7]. One of the requirements that generally appears in such standards is that both the developer and the customer must attend/conduct the formal reviews. An environment that is treated contractually is one in which the developer's organization may have several principal organizations within it (e.g., marketing, design, systems engineering, programming, SQA) and those persons developing the software may be responsible for formal reviews with other members of the organization. Typical formal reviews that are conducted between developer and customer or among organizations of the developer are discussed in this section of the report. NRC reviewers should be familiar with these formal review procedures, their results, and the follow-up results of specific product reviews. NRC should recommend that the utilities provide (or have available for) NRC the results of formal reviews and of follow-up activities. This is especially important because NRC reviewers at safety evaluation may want to check whether or not formal reviews were performed (see section 5), and may use review reports to examine how well the product was built.

Formal reviews are conducted at the end of each lifecycle phase or at the end of the planning period on the results or products of the phase or period. They may also be conducted when a serious problem or concern arises. This appendix to the report summarizes the key factors of management and technical reviews (document evaluation), discusses the role assurance activities take in the formal review process, and provides sample checklists for the technical/development reviews. For completeness all the technical/development reviews listed in Table 3-1 are included, even though [ASMENQA2] does not address them all.

B.1. The Review Process

Management reviews formally evaluate a project plan or project status relative to that plan. Management reviews have two purposes. The first is to ensure the adequacy and completeness of each planning document (e.g., PMP, SQAP, SCMP, SVVP, and Test Plans) for meeting project requirements. The second is to ensure that project activities are progressing according to the planning documents, identify the need for corrective action to the plan or the project, and ensure proper allocation of resources. All problems are documented. The results of the management reviews are summarized in a management review report which are auditable and traceable to and from the appropriate planning documents.

In contrast, the formal technical review examines the product, and the results of any assurance activities already conducted on the product. The purpose of technical reviews is to evaluate the software elements (e.g., SRS, software design description (SDD)) to ensure conformity to its specifications, compliance of the development of the software elements with its plans, and the integrity of changes to the software elements. The results of the technical reviews are summarized in technical review reports which are auditable and traceable to and from the appropriate planning documents. Success of a technical review requires that all participants carefully examine the inputs to the technical review prior to the review meeting. Section B.1.1 presents an outline of the management review process; section B.1.2 presents an outline of the technical review process. The descriptions of management and technical reviews are based on [IEEE1028].

In both the management and technical reviews, experts on specific topics (e.g., design experts for design reviews) should be present to lend their expertise to the review.

B.1.1.Management Reviews

Responsibilities. The review leader performs the administrative functions of the review and issues the management review report. Review team members are expected to be prepared for the meeting and ensure that the review objectives are met.

Inputs. Objectives of the review; list of issues to discuss; the specific planning document; current project schedule and cost data; reports from previously completed reviews; reports on project resources; and, data on complete or in progress software elements.

Entry Criteria. The need for conducting management reviews is specified in the appropriate project planning documents. Other management reviews may also be conducted upon request. The management review is performed when the review leader establishes/confirms the review objectives and determines that all necessary documents are available.

Procedures. The review leader and project manager plan for the review by identifying the review team, scheduling a time and place for the review, and distributing all inputs to the review team. An overview of the project is conducted for the review team by a qualified project member. Each review team member studies the inputs and prepares presentations for the review team. The management review consists of the review team:

Assessing adequacy and completeness of the planning documents (initial review).
Determining whether or not the project status corresponds to the planning document under review, and recording any deviations.
Determining whether or not factors not originally considered are constraining the planning document under review.
Listing issues and recommendations to be addressed by upper management and/or others who affect the project.
Recommending actions to follow the review, and authorization for additional reviews and audits.
Identifying other issues that need to be addressed.
Exit Criteria. The management review is complete once the review objectives have been addressed and the management review report has been issued.

Output. The management review report identifies: the project; the review team; review inputs; review objectives; action items; list of issues and recommendations; and, recommendations for additional reviews and information necessary to complete them. (The procedures for closing action items should be part of a plan.)

B.1.2.Technical Reviews

Responsibilities. The review leader performs the administrative functions of the review and issues the technical review report. The recorder documents the findings, decisions, and recommendations of the review team. Review team members are expected to be prepared for the meeting and ensure that the review objectives are met. Recommendations made by the review team should be such that management can act on them quickly. Management is responsible for responding to recommendations promptly.

Inputs. Objectives of the review; software element being reviewed; the software element's specifications; results of any assurance activities; any other necessary documentation.

Entry Criteria. The need for conducting technical reviews is specified in the project planning documents. Other technical reviews may also be conducted upon request. The technical review is performed when the review objectives are established, the individuals responsible at the review for the software element are prepared for the review, and the review leader determines the software element is sufficiently complete. Preparation may require considerable time spent in privately examining the inputs.

Procedures. The review leader plans for the review by identifying the review team, schedules a time and place for the review, and distributes all inputs to the review team. An overview of the project is conducted for the review team by a qualified project member. Each review team member studies the software element and related materials. The technical review consists of the review team:

Determining whether or not the software element corresponds to the specifications and standards to which it must adhere, and recording any deviations.
Listing issues, recommendations, and responsible individuals for resolving the issues.
Identifying other issues that need to be addressed.
Documenting the meeting, deficiencies found it the software element, and recommendations for management (the review leader determines whether or not an additional review should be performed on reworked software elements).
Exit Criteria. The technical review is complete once the review objectives have been addressed and the technical review report has been issued.

Output. The technical review report identifies: the review team; the software element; review inputs; the software element's unresolved deficiencies; list of management issues; action items; and, recommendations for unresolved issues deficiencies.

V&V activities should be performed prior to the formal review of the product of each of the lifecycle phases. The reviewer during development checks that these activities have been performed, and uses the resulting reports to answer the questions in the checklists below.

B.2. Checklists for Formal Reviews

Formal reviews may include reviewing SQA, SV&V, and SCM results in order to examine the product. This review may also help detect whether or not these activities were performed in accordance with their respective plans. The checklists below provide a guideline for reviewing both the product and plans for assurance activities. These checklists are not necessarily complete; at any time the reviewer during development may need to expand upon a given topic. In all checklists, negative answers require further examination by the reviewers.

B.2.1.Software Requirements Review

The following checklist contains questions a reviewer during development ask at the software requirements review based on [IEEE1028], [STARTS], [SOFTENG], [EWICS2], [IEEEP1059], [HHSFDA], [BIRRELL], and [ANS104].

Compatibility

Do the interface requirements enable compatibility of external interfaces (hardware and software)?
Completeness

Does the SRS contain everything listed in the corresponding documentation content? Does it include all requirements relating to functionality, performance, constraints, safety, etc.?
Does SRS include all user requirements (as defined in the concept phase)?
Do the functional requirements cover all abnormal situations?
Have the temporal aspects of all functions been considered?
Are the time-critical functions identified and the time criteria for them specified? Do these include the maximum and minimum times for their execution?
Does SRS define those requirements for which future changes are anticipated?
Are all normal environmental variables included?
Are the environmental conditions specified for all operating modes (e.g., normal, abnormal, disturbed)?
Consistency

Is there internal consistency between the software requirements?
Is the SRS free of contradictions?
Are the specified models, algorithms, and numerical techniques compatible?
Does SRS use standard terminology and definitions throughout?
Is SRS compatible with the operational environment of the hardware and software?
Has the impact of software on the system and environment been specified?
Has the impact of the environment on the software been specified?
Correctness

Does the SRS conform to SRS standards?
Are algorithms and regulations supported by scientific or other appropriate literature?
What evidence is there that shows vendor has applied the regulations correctly?
Does the SRS define the required responses to all expected types of errors and failure modes identified by the hazard analysis?
Were the functional requirements analyzed to check if all abnormal situations are covered by system functions?
Does SRS reference desired development standards?
Does the SRS identify external interfaces in terms of input and output mathematical variables?
Are the requirements for the man-machine interface adequate?
What is the rationale for each requirement? Is it adequate?
Is there justification for the design/implementation constraints?
Feasibility

Will the design, operation, and maintenance of software be feasible?
Are the specified models, numerical techniques, and algorithms appropriate for the problem to be solved? Are they accepted practice for nuclear power plants? Can they be implemented within the imposed constraints?
Are the quality attributes specified achievable individually and as a group?
Modifiability

Are requirements organized so as to allow for modifications (e.g., with adequate structure and cross referencing)?
Is each unique requirement defined more than once? Are there any redundant statements?
Is there a set of rules for maintaining the SRS for the rest of the software lifecycle?
Robustness

Are there requirements for fault tolerance and graceful degradation?
Traceability

Is there traceability from the next higher level spec (e.g., system concept/requirements and user needs as defined in concept phase, and system design)?
Does the SRS show explicitly the mapping and complete coverage of all relevant requirements and design constraints defined in the concept phase, by such means as a coverage matrix or cross-reference?
Is SRS traceable forward through successive development phases (e.g., into the design, code, and test documentation)?
Are safety functions or computer security functions flagged?
Understandability

Does every requirement have only one interpretation?
Are the functional requirements in modular form with each function explicitly identified?
Is there a glossary of terms?
Is formal or semiformal language used?
Is the language ambiguous?
Does the SRS contain only necessary implementation details and no unnecessary details? Is it over specified?
Are the requirements clear and specific enough to be the basis for detailed design specs and functional test cases?
Does the SRS differentiate between program requirements and other information provided?
Verifiability/Testability

Are the requirements verifiable (i.e., can the software be checked to see whether requirements have been fulfilled)?
Are mathematical functions defined using notation with precisely defined syntax and semantics?
Is there a verification procedure defined for each requirement in the SRS?
B.2.2.Software Design Review

Reviewers should be able to determine whether or not all design features are consistent with the requirements. Numerical techniques and algorithms should be appropriate for the problem to be solved. The program design needs to be partitioned in a manner consistent with the problem to be solved. And, the program should meet the requirements. The following checklist contains questions a reviewer during development may ask at the software design review based on [SOFTENG], [IEEE1028], [IEEEP1059], and [ANS104].

Completeness

Are all the items listed in section 4.6.1. addressed in the SDD?
Are the SRS requirements fulfilled?
Is there enough data (logic diagrams, algorithms, storage allocation charts, etc.) available to ensure design integrity?
Are algorithms and equations adequate, accurate, and complete?
Are requirements for the support and test software and hardware to be used in the development of the product included?
Does the design implement required program behavior with respect to each program interface?
Are all program inputs, outputs, and database elements identified and described to the extent needed to code the program?
Does the SDD describe the operational environment into which the program must fit?
Are all required processing steps included?
Are all possible outcomes of each decision point designated?
Does the design take into account all expected situations and conditions?
Does the design specify appropriate behavior in the face of unexpected or improper inputs and other anomalous conditions?
Does the SDD reference all desired programming standards?
Consistency

Are standard terminology and definitions used throughout the SDD? Are the style of presentation and the level of detail consistent throughout the document.
Does the design configuration ensure integrity of changes?
Is there compatibility of the interfaces?
Is the test documentation compatible with the test requirements of the SRS?
Is the SDD free of internal contradictions?
Are the models, algorithms, and numerical techniques that are specified mathematically compatible?
Are input and output formats consistent to the extent possible?
Are the designs for similar or related functions consistent?
Are the accuracies and units of inputs, database elements, and outputs that are used together in computations or logical decisions compatible?
Correctness

Does the SDD conform to design documentation standards?
Does the design perform only that which is specified in the SRS unless additional functionality is justified?
Is the test documentation current and technically accurate?
Is the design logic sound -- will the program do what is intended?
Is the design consistent with documented descriptions and know properties of the operational environment into which the program must fit?
Do interface designs agree with documented descriptions and known properties of the interfacing elements?
Does the design correctly accommodate all inputs, outputs, and database elements whose format, content, data rate, etc. are not at the discretion of the designer?
Feasibility

Are the specified models, algorithms, and numerical techniques accepted practices for use within nuclear power plants?
Can they be implemented within the constraints imposed on the system and on the development effort?
Are the functions as designed implementable within the available resources?
Modifiability

Does the design use information hiding as follows:
The modules are organized such that changes in the requirements only require changes to a small number of modules.
Functions and data structures likely to changes have interfaces insensitive to changes in individual functions.
The design partitions data structure access, database access and I/O access from the application software by the use of access programs (globally accessible data is not used).
Functionality is partitioned into programs to maximize the internal cohesion of programs and to minimize program coupling.
Does each program have a single function?
Modularity

Is there a schema for modularity, e.g., model-based?
Is the design structured so that it comprises relatively small, hierarchically related programs or sets of programs, each performing a particular, unique function?
Does the design use specific criteria to limit program size?
Predictability

Does the design contain programs which provide the required response to identified error conditions?
Does the design schedule computer resources in a manner that is primarily deterministic and predictable rather than dynamic?
Does the design contain a minimum number of interrupts and event driven software? Is justification given for uses of these features?
Is plausibility checking performed on the execution of programs to uncover errors associated with the frequency and/or order or program execution and the permissiveness of program execution?
Robustness

Are all SRS requirements related to fault tolerance and graceful degradation addressed in the design?
Structuredness

Does the design use a logical hierarchical control structure?
Traceability

Does the SDD show mapping and complete coverage of all requirements and design constraints in the SRS?
Are all functions in the SDD outside the scope of the SRS identified?
Are all functions identified so they can be uniquely reference by the code?
Does the SDD contain or reference a revision history which identifies all modifications to the design and the rationale for these changes?
Does the SDD reference the design notes which document design decisions relevant to the software design?
Have safety and computer security functions been flagged?
Understandability

Does the SDD avoid unnecessarily complex designs and design representations.
Is the SDD written to allow unambiguous interpretation?
Verifiability/Testability

Does the SDD describe each function using well-defined notation so that the SDD can be verified against the SRS and the code can be verified against the SDD?
Are conditions, constraints identified quantitatively so that tests may be designed?
B.2.3.Source Code Review

The following checklist contains the kinds of questions a reviewer during development may ask at the source code review based on [SOFTENG], [ANS104], and [EWICS2].

Completeness

Is the code a complete and precise implementation of the design as documented in the SDD?
Was the code integrated and debugged to satisfy the design specified in the SDD?
Does the code create the required databases, including the appropriate initial data?
Are there any unreferenced or undefined variables, constants, or data types?
Consistency

Is the code logically consistent with the SDD?
Are the same format, invocation convention, and structure used throughout?
Correctness

Does the code conform to specified standards?
Are all variables properly specified and used?
Are all comments accurate?
Are all programs invoked with the correct number of parameters?
Modifiability

Does the code refer to constants symbolically to facilitate change?
Are cross-references or data dictionaries included to show variable and constant access by the program?
Does code consist of programs with only one entry point and one exit point? (exception is with fatal error handling)
Does code reference labels or other symbolic constants rather than addresses?
Predictability

Is the code written in a language with well-defined syntax and semantics:
Was the use of self-modifying code avoided?
Does the code avoid relying on defaults provided by the programming language?
Is the code free of unintended infinite loops?
Does the code avoid recursion?
Robustness

Does the code protect against detectable runtime errors (e.g., range array index values, division by zero, out of range variable values, and stack overflow)?
Structuredness

Is each function of the program recognizable as a block of code?
Do loops only have one entrance?
Traceability

Does the code identify each program uniquely?
Is there a cross-reference framework through which the code can be easily and directly traced to the SDD?
Does the code contain or reference a revision history of all code modifications and the reason for them?
Have all safety and computer security functions been flagged?
Understandability

Do the comment statements adequately describe each routine, using clear English language?
Were ambiguous or unnecessarily complex coding used? If so, are they clearly commented?
Were consistent formatting techniques (e.g., indentation, use of white space) used to enhance clarity?
Was a mnemonic naming convention used? Does the naming reflect the type of variable?
Is the valid range of each variable defined?
Does the code use mathematical equations which correspond to the mathematical models described/derived in the SDD?
Verifiability

Are implementation practices and techniques that are difficult to test avoided?
B.2.4.Test Readiness Review

The test readiness review is usually conducted following completion of component testing or software integration testing. The purpose is to ensure readiness to begin formal integration testing or system testing without complications, and to ensure that test documentation is complete, that errors have been removed, and that use of the test facilities has been planned properly. The following are general questions that might be asked at these reviews:

Have recommended changes been made to the code as result of source code review or, as appropriate, component test or integration test?
Is the error rate sufficiently low to warrant beginning the next type of testing?
Are all test cases/procedures complete?
Is the test facility ready? Are schedules approved, personnel and physical requirements specified?
Have all test tools been checked?
Have all test procedures been checked?
B.2.5.Test Report Review

The purpose of this review is to assure the completeness of test activities. It is usually required at the end of development (i.e., following system testing). However, it may be conducted after completion of module, integration, or system testing.

The reviewer should ask questions about whether the test objectives were met (e.g., whether all test cases were executed, whether test results matched the expected outputs, and whether the results were traceable to the test cases and software requirements). The reviewer should also inquire about the nature of major anomalies (e.g., whether there were similarities, what corrective actions were taken). The reviewer should check that the anomalies existed only in the product, and were not caused by the test cases and procedures.

B.2.6.Development Documentation Review

[ASMENQA2] states that a development documentation review should be conducted following completion of the testing phase, to "assure completion and acceptability of the development documentation." Although [ASMENQA2] does not define development documentation, it is assumed that this consists of all documentation produced during the development phases of the lifecycle (e.g., requirements, design, coding). The purpose of the development documentation review seems to be checking for consistency among the development documents and to ensure that all changes have been made consistently.

Difference between Client server and webserver applications

2009-09-15T04:58:00.000-07:00

Taking a big step back, a Web server serves pages for viewing in a Web browser, while an application server provides methods that client applications can call. A little more precisely, you can say that:

A Web server exclusively handles HTTP requests, whereas an application server serves business logic to application programs through any number of protocols.

Let's examine each in more detail.

The Web server
A Web server handles the HTTP protocol. When the Web server receives an HTTP request, it responds with an HTTP response, such as sending back an HTML page. To process a request, a Web server may respond with a static HTML page or image, send a redirect, or delegate the dynamic response generation to some other program such as CGI scripts, JSPs (JavaServer Pages), servlets, ASPs (Active Server Pages), server-side JavaScripts, or some other server-side technology. Whatever their purpose, such server-side programs generate a response, most often in HTML, for viewing in a Web browser.

Understand that a Web server's delegation model is fairly simple. When a request comes into the Web server, the Web server simply passes the request to the program best able to handle it. The Web server doesn't provide any functionality beyond simply providing an environment in which the server-side program can execute and pass back the generated responses. The server-side program usually provides for itself such functions as transaction processing, database connectivity, and messaging.

While a Web server may not itself support transactions or database connection pooling, it may employ various strategies for fault tolerance and scalability such as load balancing, caching, and clustering—features oftentimes erroneously assigned as features reserved only for application servers.

The application server
As for the application server, according to our definition, an application server exposes business logic to client applications through various protocols, possibly including HTTP. While a Web server mainly deals with sending HTML for display in a Web browser, an application server provides access to business logic for use by client application programs. The application program can use this logic just as it would call a method on an object (or a function in the procedural world).

Such application server clients can include GUIs (graphical user interface) running on a PC, a Web server, or even other application servers. The information traveling back and forth between an application server and its client is not restricted to simple display markup. Instead, the information is program logic. Since the logic takes the form of data and method calls and not static HTML, the client can employ the exposed business logic however it wants.

In most cases, the server exposes this business logic through a component API, such as the EJB (Enterprise JavaBean) component model found on J2EE (Java 2 Platform, Enterprise Edition) application servers. Moreover, the application server manages its own resources. Such gate-keeping duties include security, transaction processing, resource pooling, and messaging. Like a Web server, an application server may also employ various scalability and fault-tolerance techniques.

QA Methodologies followed in any QA

2009-09-15T04:01:00.000-07:00

QA Methodologies and Test Management Systems
For the smooth progress of the software development process what is most important is the availability of advanced test management systems. If that is available then it can be said that half the job of the testers is done. But when selecting a test management system make sure that it fits well with the different QA methodologies. Now the most commonly used QA methodologies are Waterfall, Hybrid, V Model, and Agile. Now let’s look at the important aspects of the most common methodologies and how a good test management system should behave in the various methodologies.

Waterfall model, being structured, well documented and a disciplined process is mainly used for stable projects. Since it is non-iterative it allows multiple projects to run at a time. A good test management system will allow multiple releases within each project so that the QA team can handle major, minor and patch releases without changing processes.

V Model is a rigorous and measurable process and therefore is mostly used for complex projects. A suitable test management system helps in maintaining rigor in the testing process. It will also handle multiple testing cycles for System, Integration, and final Acceptance.

Agile, as the name suggests, is a flexible and fast process and its most important aspect is that it minimizes risk by developing software in a short period of time. A good test management system not only organizes and tracks testing by iterations but also handles multiple iterations within each release. It also facilitates information sharing and instant communication among the onsite and offshore testers.

Hybrid model is a combination of two or more QA methods. It makes transition from one model to another throughout the lifecycle of the project. A suitable test management system allows different models to co-exist peacefully by facilitating easy transition between various modes in the same system. It also allows reusability of test assets between different modes and lifecycles.

Assignment on Quality Metrics-1

2009-05-18T23:36:00.001-07:00

EXERCISE A– Testing Project

Post delivery defect leakage goal: This is the % of the testing defects identified in our organization(excludes client found defects)/total defects. It should be greater than 99%.

Effort and Schedule goals & formula:

Effort PCB Schedule PCB
Goal(Upper Limit) 15% Goal(Upper Limit) 5%
Goal(Lower Limit) 0% Goal(Lower Limit) -5%

Effort : (Actual Hours-Estimated Hours)* 100 / (Estimated Hours)

Schedule: ((Actual End Date-Actual Start Date)-(Estimated End Date- Estimated Start Date))*100 / (Estimated End Date-Estimated Start Date+1)

Phase Organization Goal Formula
RDD Completion of test preparation >=0.75 Review defects/Size
TDD Completion of test execution >= 1.25 Review defects/Size

Defects distribution across different testing cycles:

Usually 60% defects are found in the 1st test cycle, 90% of defects found by the 2nd test cycle, remaining in the last test cycle.

Note: Goals are usually tighter by 25% of the mean. While totaling defects use the weighted defects i.e. Severity 1 = 10, Severity 2 = 5, Severity 1 =1

Case Study 1:

Setting Goals

You are an independent testing team and the scope of work is system testing. There is another team Denver which has done similar projects within the same financial area. Their data is as follows;

Denver Data
Post delivery defect leakage 95%
Effort Variance UCL 50, LCL -40
Schedule Variance UCL 3, LCL 0
RDD .25
TDD .50
Size 600 Test Case Points

However this project is more complex and has many more interfaces to test. This would be data driven test where many SQL queries would have to be written to extract and validate the data. The size is 1000 Test Case points. You have planned 3 test cycles. Within each test cycle you have planned for 3 test rounds.

Q.1. Set the post delivery defect leakage, effort/schedule variance, RDD and TDD goals. How many defects would you expect at end of test preparation and test execution.
How many defects you would expect at end of each test cycle? Make your assumptions and identify the risks.

A.1.
Project Data Goals
Post delivery defect leakage 95%
Effort Variance UCL 50, LCL -40
Schedule Variance UCL 3, LCL 0
RDD .75, 750 defects
TDD 1.25,1250 defects, 1st Cycle – 750, 2nd cycle -375
Size 600 Test Case Points 1000

At end of test Planning

At end of test preparation the effort, schedule data looks like this;

Planned Actual
Effort 1200 man hrs 1500 man hrs
Schedule 10th June 2004 12th June 2004
Schedule Variance UCL 3, LCL 0
Review defects 450 defects

During your test planning phase some testers resigned and you had to quickly recruit new associates who joined towards the end of test planning phase.

The main business functionalities are adjustments, auto login, customer service transactions, BT related functionality, credit card payment and maintaining batch items. Many queries on the functionality were raised to the client. Mean time to resolve 70% of the queries was 2 days. 30% of the queries took more than 6 days to resolve.

The distribution of queries, test cases and reviews across functionality is;

Function/Module No of Queries Review defects Test Cases
Adjustments 18 65 245
Maintaining Batch Transactions & Create new batch 73
40 136
BT adjustments 27 9 234
Convenience Cheques 23 38 23
Customer Service Transactions 6 13 129
Letters & Rates 40 24 68
Auto Login 9 34 56
Credit card payments 23 55 143
Unpaid Tem Fee 45 19 111
Fee Tallies 12 34 98
Miscellaneous Adjustments 8 38 189
Refund Credit Balance 10 34 245
Other 47 136

Distribution of defects category wise across testware is as follows;

Function/Logic (incorrect testware, not as per functional requirements) ;187 defects
Suggestions ; 50
Test Coverage ; 213 defects

Q 2 What analysis would you do and what actions would you take? What goals would you re-visit? Make your assumptions and identify the risks.

End of Test Execution

At end of cycle 1 - The effort variance is 60%. It took a long time for the client to decide whether the defect raised was really a defect. The client had included some new functionality to test. During test execution the link to the client test environment was down.

Defect Trend Test Cases Defects – Severity
Stages Planned Executed Severity 1 Severity 2 Severity 3 Suggestion Total
Smoke Test 50 40 18 37 5 2
Cycle 2 1500 592 23 76 13 5
Cycle 3 1500 789 15 49 12 2
Cycle 4 1500 1979 25 38 11 4
Cycle 5 1979 1979 7 23 9 9
Cycle 6 1979 1979 2 7 0 0
Staging 2746 2768 39 99 31 5
Total 129 329 81 27
Rejected 34 22 35 5

Defect Reported Status Open Fixed Closed Deferred Reject Later Duplicate Total
Severity 1 2 9 34 32 34 2 16
Severity 2 20 20 170 52 22 11 34
Severity 3 3 6 13 4 35 8 12
Suggestion 5 5 10 5 2
Total

Q3. Plot the chart on test cases planned vs executed, % executed and defect trend for all severity levels? Plot the chart of the defect status, find out the % distribution of defect status? What analysis would you do? What actions you should take? What goals would you re-visit?

Defect Trend Test Cases Defects – Severity
Stages Planned Executed Severity 1 Severity 2 Severity 3 Suggestion Total
Smoke Test 50 40 18 37 5 2 370
Cycle 2 1500 592 23 76 13 5 623
Cycle 3 1500 789 15 49 12 2 407
Cycle 4 1500 1979 25 38 11 4 451
Cycle 5 1979 1979 7 23 9 9 194
Cycle 6 1979 1979 2 7 0 0 55
Staging 2746 2768 39 99 31 5 916
Total 129 329 81 27 3016
Rejected 34 22 35 5 485
Rejected % 26% 7% 43% 19% 16%

Defect Reported Status Open Fixed Closed Deferred Reject Later Duplicate Total
Severity 1 2 9 34 32 34 2 16 129
Severity 2 20 20 170 52 22 11 34 329
Severity 3 3 6 13 4 35 8 12 81
Suggestion 5 5 10 5 2 27
Total 25 40 222 98 96 23 62 665

A3 Find out why the defects were rejected, plot defects across test cases and functionality & re-estimate. Find out how many defects did not have corresponding test cases. Set up the defect resolution time with the client. In fact this should have been done during the planning phase. Do a causal analysis to find out the reasons for the rejected defects. For the client added functionality, re-estimate the testing effort and also find out if the test cases have been added for the newly added functionality

EXERCISE B–Testing Project

You are executing an independent system testing project. For system test you have planned 4 cycle of 2 weeks each:

Week Week 1 Week 3 Week 5 Week 7
Cycle Cycle 1 Cycle 2 Cycle 3 Cycle 4
Build Build 1 Build 2 Build 3 Build 4
Functionality being added 2 5 3 0

Build_1 was delivered on time i.e. first day of week 1. This was tested till end of second week. Defects found in Cycle 1 were 50. But Build 2 was delivered in week 4 instead of week 3 and that too was a broken build. After discussing this problem with the client, they have decided to give a working build which will have the functionality of Build 2and 3 together in week 5.

Q1. How will you plan testing of build which will have functionality of Build 2 and 3?
A1. In this scenario you will have 3 Cycles instead of 4 and you will add another cycle in Week 9.

Assignment on Quality Metrics-1

2009-05-18T23:36:00.000-07:00

important Basics of Performance Testing-3

2009-05-18T22:45:00.000-07:00

Basics: Underlying Operating System and Network Improvements
If you control the OS and hardware where the application will be deployed, there are a number of changes you can make to improve performance. Some changes are generic and affect most applications, while some are application-specific. This article applies to most server systems running Java application, including servlets, where you usually specify (or have specified to you) the underlying system, and where have some control over tuning the system. Client and standalone Java programs are likely to benefit from this chapter only if you have some degree of control over the target system, but some tips in the chapter apply to all Java programs.
It is usually best to target the OS and hardware as a last tuning choice. Tuning the application itself generally provides far more significant speedups than tuning the systems on which the application is running. Application tuning also tends to be easier (though buying more powerful hardware components is easier still and a valid choice for tuning). However, application and system tuning are actually complementary activities, so you can get speedups from tuning both the system and the application if you have the skills and resources. Here are some general tips for tuning systems:

• Constantly monitor the entire system with any monitoring tools available and keep records. This allows you to get a background usage pattern and also lets you compare the current situation with situations previously considered stable.
• You should run offline work during off-hours only. This ensure that there is no extra load on the system when the users are executing online tasks and enhance performance of both online and offline activities.
• If you need to run extra tasks during the day, try to slot them into times with low user activity. Office activity usually peaks at 9am and 2:30pm and has a load between noon and 1pm or at shift changeovers. You should be able to determine the user-activity cycles appropriate to your system by examining the results of normal monitoring. The reduced conflict for system resources during periods of low activity improves performance.
• You should specify timeouts for all process under the control of your application (and others on the system, if possible) and terminate processes that have passed their timeout value.
• Apply any partitioning available from the system to allocate determinate resources to your application. For example, you can specify disk partitions, memory segments, and even CPUs to be allocated to particular process.

As the entire chapter is lengthy, I've split them into various sections namely the links that followed. Click on the links to access the article.
CPU
Disk
Memory
Network

The above is taken from the publication, "Java Performance Tuning" written by Jack Shirazi. I would recommend to read this book as it provides not just tuning and bottleneck concepts bounded by Java. A simplified version (which is the summary of the chapter can be found here [Comming soon]).
Basics: CPU Bottlenecks
Java provides a virtual machine runtime system that is just that: an abstraction of a CPU that runs in software. (Note that this chapter is taken from the "Java Performance Tuning" written by Jack Shirazi and therefore alot of discussions circled around Java technologies.) These virtual machines run on a real CPU, and in this section the book discuss the performance characteristics of those real CPUs.

CPU Load

The CPU and many other parts of the system can be monitored using system-level utilities. On Windows, the task manager and performance monitor can be used for monitoring. On UNIX, a performance monitor (such as perfmeter) is usually available, as well as utilities such as vmstat. Two aspects of the CPU are worth watching as primary performance points. These are the CPU utilization (usually expressed in percentage terms) and the run-able queue of processes and threads (often called the load or the task queue). The first indictor is simply the percentage of the CPU (Or CPUs) being used by all the various threads. If this is up to 100% for significant periods of time, you may have a problem. On the other hand, if it isn’t, the CPU is under-utilized, but that is usually preferable. Low CPU usage can indicate that your application may be blocked for significant periods on disk or network I/O. High CPU usage can indicate thrashing (lack of RAM) or CPU contention (indicating that you need to tune the code and reduce the number of instructions being processed to reduce the impact on the CPU).

A reasonable target is 75% CPU utilization (which from what I read from different authors varies from 75% till 85%). This means that the system is being worked toward its optimum, but that you have left some slacks for spikes due to other system or application requirements. However, note that if more than 50% of the CPU is used by system processes (i.e. administrative and IS process), your CPU is probably under-powered. This can be identified by looking at the load of the system over some period when you are not running any applications.

The second performance indicator, the run-able queue, indicates the average number of processes or threads waiting to be scheduled for the CPU by the OS. They are run-able processes, but the CPY has no time to run them and is keeping them waiting for some significant amount of time. As soon as the run queue goes above zero, the system may display contention for resources, nut there usually some value above zero that still gives acceptable performance for any particular system. You need to determine what that value is in order to use this statistics as a useful warning indicator. A simplistic way to do this is to create a short program that repeatedly does some simple activity. You can then time each run of that activity. You can run copies of this process one after the other so that more and ore copies are simultaneously running. Keep increasing the number of copies being run until the run queue starts increasing. By watching the times recorded for the activity, you can graph that time against the run queue. This should give you some indication of when the run-able queue becomes too large for useful responses on your system, administrator if the threshold is exceeded. A guideline by Adrian Cockcroft is that performance starts to degrade if the run queue grows bigger than four times the number of CPUs.

If you can upgrade the CPU of the target environment, doubling the CPU speed is usually better than doubling the number of CPUs. And remember that parallelism in an application doesn’t necessarily need multiple CPUs. If I/O is significant, the CPU will have plenty of time for many threads.

Process Priorities

The OS also has the ability to prioritize the processes in terms of providing CPU time by allocating process priority levels. CPU priorities provide a way to throttle high-demand CPU processes, thus giving other processes a greater share of the CPU. If there are other processes that need to run on the same machine but it doesn’t matter if they were run slowly, you can give your application processes a (much) higher priority than those other processes, thus allowing your application the lion’s share of CPU time on a congested system. This is worth keeping in mind if your application consists of multiple processes, you should also consider the possibility of giving your various processes different levels of priority.

Being tempted to adjust the priority levels of processes, however, is often a sign that the CPU is underpowered for the tasks you have given it.

The above is taken from the publication, "Java Performance Tuning" written by Jack Shirazi. I would recommend to read this book as it provides not just tuning and bottleneck concepts bounded by Java.

Basics: Disks Bottlenecks
In most cases, applications can be tuned so that disk I/O does not cause any serous performance problems. But if, application tuning, you find that disk I/O s still causing a performance problem; your best bet may be to upgrade the system disks. Identifying whether the system has a problem with disk utilization is the first step. Each system provides its own tools to identify disk usage (Windows has a performance monitor, and UNIX has the sar, vmstat, iostat utilities.) At minimum, you need to identify whether the paging is an issue (look at disk-scan rates) and assess the overall utilization of your disks (e.g. performance monitor on Windows, output from iostat –D on UNIX). It may be that the system has a problem independent of your application (e.g. unbalanced disks), and correcting this problem may resolve the problem issue.

If the disk analysis does not identify an obvious system problem that is causing the I/O overhead, you could try making a disk upgrade or a reconfiguration. This type of tuning can consist of any of the following:
• Upgrading to faster disks
• Adding more swap space to handle larger buffers
• Changing the disk to be striped (where files are striped across several disks, thus providing parallel I/O. e.g. with a RAID system)
• Running the data on raw partitions when this is shown to be faster.
• Distributing simultaneously accessed files across multiple disks to gain parallel I/O
• Using memory-mapped disks or files

If you have applications that run on many systems and you do not know the specification of the target system, bear in mind that you can never be sure that ant particular disk is local to the user. There is a significant possibility that the disk being used by the application is a network-mounted disk. This doubles the variability in response times and throughput. The weakest link will probably not even be constant. A network disk is a shared resource, as is the network itself, so performance is hugely and unpredictably affected by other users and network load.

Disk I/O
Do not underestimate the impact of disk writes on the system as a whole. For example, all database vendors strongly recommend that the system swap files be placed on a separate disk from their databases. The impact of if not doing so can decrease database throughput (and system activity) but an order of magnitude. This performance decreases come from not splitting I/O of two disk-intensive applications (in this case, OS paging and database I/O).

Identifying that there is an I/O problem is usually fairly easy. The most basic symptom is that things take longer than expected, while at the same time the CPU is not at all heavily worked. The disk-monitoring utilities will also tell you that there is a lot of work being done to the disks. At the system level, you should determine the average peak requirements on the disks. Your disks will have some statistics that are supplied by the vendor, including:

The average and peak transfer rates, normally in megabytes (MB) per seconds, e.g. 5MB/sec. Form this, you can calculate how long an 8K page takes to be transferred from disk, and for example, 5MB/sec is about 5K/ms, so an 8K page takes just under 2ms to transfer.

Average seek time, normally in milliseconds (ms). This is the time required for the disk head to move radially to the correct location on the disk.

Rotational speed, normally in revolutions per minutes (rpm), e.g. 7200rpm. From this, you can calculate the average rotational delay in moving the disk under the disk-head reader, i.e., the time taken for half a revolution. For example, for 7200rpm, one revolution takes 60,000ms (60 seconds) divided by 7200rpm, which is about 8.3 ms. So half a revolution takes just over 4ms, which is consequently the average rotational delay.

This list allows you to calculate the actual time it takes to load a random 8K page from the disk, this being seek time + rotational delay + transfer time. Using the examples given in the list, you have 10 + 4 + 2 = 16 ms to load a random 8K page (almost an order of magnitude slower than the raw disk throughput). This calculation gives you a worst –case scenario for the disk-transfer rates for your application, allowing you to determine if the system is up to the required performance. Note that if you are reading data stored sequentially in disk (as when reading a large file), the seek time and rotational delay are incurred less than once per 8K page loaded. Basically, these two times are incurred only at the beginning of opening the file and whenever the file is fragmented. But this calculation is confounded by other processes also executing I/O to the disk at the same time. This overhead is part of the reason why swap and other intensive I/O files should not be put on the same disk.

One mechanism for speeding up disk I/O is to stripe disks. Disk striping allows data from a particular file to be spread over several disks. Striping allows reads and writes to be performed in parallel across the disks without requiring any application changes. This can speed up disk I/O quite effectively. However, be aware that the seek and rotational overhead previously listed still applies, and if you are making many small random reads, there may be no performance gain from striping disks.

Finally, note again that using remote disks adversely affects I/O performance. You should not be using remote disks mounted from the network with any I/O-intensive operations if you need good performance.

Clustering Files

Reading many files sequentially is faster if the files are clustered together on the disk, allowing the disk-head reader to flow from one file to the next. This clustering is best done in conjunction with defragmenting the disks. The overhead in finding the location of a file on the disk (detailed in the previous section) is also minimized for sequential reads if the files are clustered.

If you cannot specify clustering files at the disk level, you can still provide similar functionality by putting all the files together into one large file (as is done with the ZIP file systems). This is fine if all the files are read-only files or if there is just one file that is writable (you place that at the end). However, when there is more than one writable file, you need to manage the location of the internal files in your system as one or more grow. This becomes a problem and is not usually worth the effort. (If the files have a known bounded size, you can pad the files internally, thus regaining the single file efficiency.)
Cached File Systems (RAM Disks, tmpfs, cachefs)

Most OS provide the ability to map a file system into the system memory . This ability can speed up reads and writes to certain files in which you control your target environment. Typically, this technique has been used to speed up the reading and writing of temporary files. For example, some compilers (of languages in general, not specifically Java) generate many temporary files during compilation. If these files are created and written directly to the system memory, the speed of compilation is greatly increased. Similarly, if you have the a set of external files that are needed by your application, it is possible to map these directly into the system memory, thus allowing their reads and writes to be speeded up greatly.

But note that these types of file systems are not persistent. In the same way the system memory of the machine gets cleared when it is rebooted, so these file systems are removed on reboot. If the system crashes, anything in a memory-mapped file system is lost. For this reason, these types of file systems are usually suitable only for temporary files or read-only versions of disk-based files (such as mapping a CD-ROM into a memory-resident file system).

Remember that you do not have the same degree of fine control over these file systems that you have over your application. A memory-mapped file system does not use memory resources as efficiently as working directly from your application. If you have direct control over the files you are reading and writing, it is usually better to optimize this within your application rather than outside it. A memory-mapped file system takes space directly from system memory. You should consider whether it would be better to let your application grow in memory instead of letting the file system take up that system memory. For multi-user applications, it is usually more efficient for the system to map shared files directly into memory, as a particular fule then takes up just one memory location rather than duplicate in each process. Note that from SDK 1.4, memory-mapped files are directly supported from the java.nio package. Memory-mapped files are slightly different from memory-mapped file systems. A memory-mapped file uses system resources to read the file into system memory, and that data can then be accessed form Java through the appropriate java.nio buffer. A memory-mapped file system does not require the java.nio package and, as far as Java is concerned, files in that file system are simply files like any others. The OS transparently handles the memory mapping.

The creation of memory-mapped file systems is completely system-dependent, and there is no guarantee that it is available on any particular system (though most modern OS do support this feature). On UNIX system, the administrator needs to look at the documentation of the mount command and its subsections on cachefs and tmpfs. Under Windows, you should find details by looking at the documentation on how to setup a RAM disk, a portion of memory mapped to a logical disk drive.

In a similar way, there are products available that pre-cache shared libraries (DLL) and even executables in memory. This usually means only that an application starts quicker or loads the quicker, and so may not be much help in speeding up a running system.

But you can apply the technique of memory-mapping file systems directly and quite usefully for applications in which processes are frequently started. Copy the Java distribution and all class files (all JDK, application, and third-party class files) onto a memory-mapped file system and ensure that all executions and classloads take place from the file system. Since everything (executables, DLLs, class files, resources, etc.) is already in memory, the startup time is much faster. Because only the startup (and class loading) time is affected, this technique gives only a small boost to applications that are not frequently starting processes, but can be usefully applied if startup time is a problem.

Disk Fragmentation

When files are stored on disk, the bytes in the files are note necessarily stored contiguously: their storage depends on file size and contiguous space available on the disk. This non-contiguous storage is called fragmentation. Any particular file may have some chunks in one place, and a pointer to the next chunk that may be quite a distance away on the disk.

Hard disks tend to get fragmented over time. This fragmentation delays both reads from files (including loading applications into computer memory on startup) and writes to files. This delay occurs because the disk header must wind on to the next chunk with each fragmentation, and this takes time.

For optimum performance on any system, it is a good idea to periodically defragment the disk. This reunites files that have been split up so that disk heads do not spend so much time searching for data once the file-header locations have been identified, thus speeding up data access. Defragmenting may not be effective on all systems, however.
Disk Sweet Spots

Most disks have a location from which data is transferred faster than from other locations. Usually, the closer the data is to the outside edge of the disk, the faster it can be read from the disk. Most hard disks rotate at constant angular speed. This means that the linear speed of the disk under a point is faster the farther away the point is from the center of the disk. Thus, data at the edge of the disk can be read from (and written to) at the faster possible rate commensurate with the maximum density of data storable on disk.

This location with faster transfer rates usually termed the disk sweet spot. Some
(Commercial) utilities provide mapped access to the underlying disk and allow you to reorganize files to optimize access. On most server systems, the administrator has control over how logical partitions of the disk apply to the physical layout, and how to position files to the disk sweet spots. Experts for high-performance database system sometimes try to position the index tables of the database as close as possible to the disk sweet spot. These tables consist of relatively small amounts of data that affect the performance of the system in a disproportionately large way, so that any speed improvement in manipulating these tables is significant.

Note that some of the latest OS are beginning to include “awareness” of disk sweet spots, and attempt to move executables to sweet spots when defragmenting the disk. You may need to ensure that the defragmentation procedure does not disrupt your own use of the disk sweet spot.
The above is taken from the publication, "Java Performance Tuning" written by Jack Shirazi. I would recommend to read this book as it provides not just tuning and bottleneck concepts bounded by Java.
Basics: Disks Bottlenecks

In most cases, applications can be tuned so that disk I/O does not cause any serous performance problems. But if, application tuning, you find that disk I/O s still causing a performance problem; your best bet may be to upgrade the system disks. Identifying whether the system has a problem with disk utilization is the first step. Each system provides its own tools to identify disk usage (Windows has a performance monitor, and UNIX has the sar, vmstat, iostat utilities.) At minimum, you need to identify whether the paging is an issue (look at disk-scan rates) and assess the overall utilization of your disks (e.g. performance monitor on Windows, output from iostat –D on UNIX). It may be that the system has a problem independent of your application (e.g. unbalanced disks), and correcting this problem may resolve the problem issue.

If the disk analysis does not identify an obvious system problem that is causing the I/O overhead, you could try making a disk upgrade or a reconfiguration. This type of tuning can consist of any of the following:
• Upgrading to faster disks
• Adding more swap space to handle larger buffers
• Changing the disk to be striped (where files are striped across several disks, thus providing parallel I/O. e.g. with a RAID system)
• Running the data on raw partitions when this is shown to be faster.
• Distributing simultaneously accessed files across multiple disks to gain parallel I/O
• Using memory-mapped disks or files

If you have applications that run on many systems and you do not know the specification of the target system, bear in mind that you can never be sure that ant particular disk is local to the user. There is a significant possibility that the disk being used by the application is a network-mounted disk. This doubles the variability in response times and throughput. The weakest link will probably not even be constant. A network disk is a shared resource, as is the network itself, so performance is hugely and unpredictably affected by other users and network load.

Disk I/O
Do not underestimate the impact of disk writes on the system as a whole. For example, all database vendors strongly recommend that the system swap files be placed on a separate disk from their databases. The impact of if not doing so can decrease database throughput (and system activity) but an order of magnitude. This performance decreases come from not splitting I/O of two disk-intensive applications (in this case, OS paging and database I/O).

Identifying that there is an I/O problem is usually fairly easy. The most basic symptom is that things take longer than expected, while at the same time the CPU is not at all heavily worked. The disk-monitoring utilities will also tell you that there is a lot of work being done to the disks. At the system level, you should determine the average peak requirements on the disks. Your disks will have some statistics that are supplied by the vendor, including:

The average and peak transfer rates, normally in megabytes (MB) per seconds, e.g. 5MB/sec. Form this, you can calculate how long an 8K page takes to be transferred from disk, and for example, 5MB/sec is about 5K/ms, so an 8K page takes just under 2ms to transfer.

Average seek time, normally in milliseconds (ms). This is the time required for the disk head to move radially to the correct location on the disk.

Rotational speed, normally in revolutions per minutes (rpm), e.g. 7200rpm. From this, you can calculate the average rotational delay in moving the disk under the disk-head reader, i.e., the time taken for half a revolution. For example, for 7200rpm, one revolution takes 60,000ms (60 seconds) divided by 7200rpm, which is about 8.3 ms. So half a revolution takes just over 4ms, which is consequently the average rotational delay.

This list allows you to calculate the actual time it takes to load a random 8K page from the disk, this being seek time + rotational delay + transfer time. Using the examples given in the list, you have 10 + 4 + 2 = 16 ms to load a random 8K page (almost an order of magnitude slower than the raw disk throughput). This calculation gives you a worst –case scenario for the disk-transfer rates for your application, allowing you to determine if the system is up to the required performance. Note that if you are reading data stored sequentially in disk (as when reading a large file), the seek time and rotational delay are incurred less than once per 8K page loaded. Basically, these two times are incurred only at the beginning of opening the file and whenever the file is fragmented. But this calculation is confounded by other processes also executing I/O to the disk at the same time. This overhead is part of the reason why swap and other intensive I/O files should not be put on the same disk.

One mechanism for speeding up disk I/O is to stripe disks. Disk striping allows data from a particular file to be spread over several disks. Striping allows reads and writes to be performed in parallel across the disks without requiring any application changes. This can speed up disk I/O quite effectively. However, be aware that the seek and rotational overhead previously listed still applies, and if you are making many small random reads, there may be no performance gain from striping disks.

Finally, note again that using remote disks adversely affects I/O performance. You should not be using remote disks mounted from the network with any I/O-intensive operations if you need good performance.

Clustering Files

Reading many files sequentially is faster if the files are clustered together on the disk, allowing the disk-head reader to flow from one file to the next. This clustering is best done in conjunction with defragmenting the disks. The overhead in finding the location of a file on the disk (detailed in the previous section) is also minimized for sequential reads if the files are clustered.

If you cannot specify clustering files at the disk level, you can still provide similar functionality by putting all the files together into one large file (as is done with the ZIP file systems). This is fine if all the files are read-only files or if there is just one file that is writable (you place that at the end). However, when there is more than one writable file, you need to manage the location of the internal files in your system as one or more grow. This becomes a problem and is not usually worth the effort. (If the files have a known bounded size, you can pad the files internally, thus regaining the single file efficiency.)
Cached File Systems (RAM Disks, tmpfs, cachefs)

Most OS provide the ability to map a file system into the system memory . This ability can speed up reads and writes to certain files in which you control your target environment. Typically, this technique has been used to speed up the reading and writing of temporary files. For example, some compilers (of languages in general, not specifically Java) generate many temporary files during compilation. If these files are created and written directly to the system memory, the speed of compilation is greatly increased. Similarly, if you have the a set of external files that are needed by your application, it is possible to map these directly into the system memory, thus allowing their reads and writes to be speeded up greatly.

But note that these types of file systems are not persistent. In the same way the system memory of the machine gets cleared when it is rebooted, so these file systems are removed on reboot. If the system crashes, anything in a memory-mapped file system is lost. For this reason, these types of file systems are usually suitable only for temporary files or read-only versions of disk-based files (such as mapping a CD-ROM into a memory-resident file system).

Remember that you do not have the same degree of fine control over these file systems that you have over your application. A memory-mapped file system does not use memory resources as efficiently as working directly from your application. If you have direct control over the files you are reading and writing, it is usually better to optimize this within your application rather than outside it. A memory-mapped file system takes space directly from system memory. You should consider whether it would be better to let your application grow in memory instead of letting the file system take up that system memory. For multi-user applications, it is usually more efficient for the system to map shared files directly into memory, as a particular fule then takes up just one memory location rather than duplicate in each process. Note that from SDK 1.4, memory-mapped files are directly supported from the java.nio package. Memory-mapped files are slightly different from memory-mapped file systems. A memory-mapped file uses system resources to read the file into system memory, and that data can then be accessed form Java through the appropriate java.nio buffer. A memory-mapped file system does not require the java.nio package and, as far as Java is concerned, files in that file system are simply files like any others. The OS transparently handles the memory mapping.

The creation of memory-mapped file systems is completely system-dependent, and there is no guarantee that it is available on any particular system (though most modern OS do support this feature). On UNIX system, the administrator needs to look at the documentation of the mount command and its subsections on cachefs and tmpfs. Under Windows, you should find details by looking at the documentation on how to setup a RAM disk, a portion of memory mapped to a logical disk drive.

In a similar way, there are products available that pre-cache shared libraries (DLL) and even executables in memory. This usually means only that an application starts quicker or loads the quicker, and so may not be much help in speeding up a running system.

But you can apply the technique of memory-mapping file systems directly and quite usefully for applications in which processes are frequently started. Copy the Java distribution and all class files (all JDK, application, and third-party class files) onto a memory-mapped file system and ensure that all executions and classloads take place from the file system. Since everything (executables, DLLs, class files, resources, etc.) is already in memory, the startup time is much faster. Because only the startup (and class loading) time is affected, this technique gives only a small boost to applications that are not frequently starting processes, but can be usefully applied if startup time is a problem.

Disk Fragmentation

When files are stored on disk, the bytes in the files are note necessarily stored contiguously: their storage depends on file size and contiguous space available on the disk. This non-contiguous storage is called fragmentation. Any particular file may have some chunks in one place, and a pointer to the next chunk that may be quite a distance away on the disk.

Hard disks tend to get fragmented over time. This fragmentation delays both reads from files (including loading applications into computer memory on startup) and writes to files. This delay occurs because the disk header must wind on to the next chunk with each fragmentation, and this takes time.

For optimum performance on any system, it is a good idea to periodically defragment the disk. This reunites files that have been split up so that disk heads do not spend so much time searching for data once the file-header locations have been identified, thus speeding up data access. Defragmenting may not be effective on all systems, however.
Disk Sweet Spots

Most disks have a location from which data is transferred faster than from other locations. Usually, the closer the data is to the outside edge of the disk, the faster it can be read from the disk. Most hard disks rotate at constant angular speed. This means that the linear speed of the disk under a point is faster the farther away the point is from the center of the disk. Thus, data at the edge of the disk can be read from (and written to) at the faster possible rate commensurate with the maximum density of data storable on disk.

This location with faster transfer rates usually termed the disk sweet spot. Some
(Commercial) utilities provide mapped access to the underlying disk and allow you to reorganize files to optimize access. On most server systems, the administrator has control over how logical partitions of the disk apply to the physical layout, and how to position files to the disk sweet spots. Experts for high-performance database system sometimes try to position the index tables of the database as close as possible to the disk sweet spot. These tables consist of relatively small amounts of data that affect the performance of the system in a disproportionately large way, so that any speed improvement in manipulating these tables is significant.

Note that some of the latest OS are beginning to include “awareness” of disk sweet spots, and attempt to move executables to sweet spots when defragmenting the disk. You may need to ensure that the defragmentation procedure does not disrupt your own use of the disk sweet spot.
The above is taken from the publication, "Java Performance Tuning" written by Jack Shirazi. I would recommend to read this book as it provides not just tuning and bottleneck concepts bounded by Java.

I would also recommend visiting the site author by Jack himself and a couple of his mates. It is a very resourceful site for Java performance-related information. Click here to access it.
Basics: Memory Bottlenecks
Maintaining watch directly on the system memory (RAM) is not usually that helpful in identifying performance problems. A better indication that memory might be affecting performance can be gained by watching for paging of data from memory to the swap files. Most current OS have a virtual memory that is made up of the actual (real) system memory using RAM chips, and one or more swap files on the system disks. Processes that are currently running are operating in real memory. The OS can take pages from any of the processes currently in real memory and swap them out to disk. This is known as paging. Paging leaves free space in real memory to allocate to other processes that need to bring in a page from disk.

Obviously, if all the processes currently running can fit into real memory, there is no need for the system to swap out any pages. However, if there are too many processes to fit into real memory, paging allows the system to free up system memory to run more processes. Paging affects system performance in many ways. One obvious way is that if a process has had some pages moved to disk and the process becomes run-able, the OS has to pull back the pages from dusk before that process can be run. This leads to delays in performance. In addition, both CPU and the disk I/O spend time doing the paging, reducing available processing power and increasing the load on the disks. This cascading effect involving both the CPU and I/O can degrade the performance of the whole system in such a way that it maybe difficult to even recognize that paging is the problem. The extreme version of too much paging is thrashing, in which the system is spending so much time moving pages around that it fails to perform any other significant work. (The next step is likely to be a system crash).

As with run-able queues (see CPU section), a little paging of the system does not affect the performance enough to cause concern. In fact, some paging can be considered good. It indicated that the system’s memory resources are fully utilized. But at the point where paging becomes a significant overhead, the system is overloaded.

Monitoring paging is relatively easy. On UNIX, the utilities vmstat and iostat provide details as to the level of paging, disk activity and memory levels. On Windows, the performance monitor has categories to show these details, as well as being able to monitor the system swap files.

If there is more paging than is optimal, the system’s RAM is insufficient or processes are too big. To improve this situation, you need to reduce the memory being used by reducing the number of processes or the memory utilization of some processes. Alternatively, you can add RAM. Assuming that it is your application that is causing the paging (Otherwise, either the system needs an upgrade, or someone else’s processes may also have to be tuned), you need to reduce the memory resources you are using.

When the problem is caused by a combination of your application and others, you can partially address the situation by using process priorities (see the CPU “section”). The equivalent to priority levels for memory usage is an all-or-nothing option, where you can lock process in memory. This option is not available on all systems and is more often applied to shared memory than to processes, but nevertheless, it is useful to know. If this option is applied, the process is locked into real memory and is not paged out at all. You need to be aware that using this option reduces the amount of RAM available to all other processes, which can make overall system performance worse. Any deterioration in system performance is likely occurring at heavy system load, so make sure you extrapolate the effect of reducing the system memory in this way.

Basics: Network Bottlenecks
At the network level, many things can affect performance. The bandwidth (the amount of data that can be carried by the network) tends to be the first culprit checked. Assuming you have determined that bad performance is attributable to the network component of an application, there is more likely cause of bad network performance than network bandwidth. The most likely cause of bad network performance is the application itself and how it is handling distributed data and functionality.
The overall speed of a particular network connection is limited by the slowest link in the connection chain and the length of the chain. Identifying the slowest link is difficult and may not even be consistent: it can vary at different times of the day or for different communication paths. A network communication path lead from an application through a TCP/IP stack (which adds various layers of headers, possibly encrypting and compressing data as well), then through the hardware interface, through a modem, over a phone line, through another modem, over to a service provider’s router, through many heavily congested data lines of various carrying capacities and multiple routers with different maximum throughputs and configurations, to a machine at the other end with its own hard interface, TCP/IP stack and application. A typical web download route is just like this. In addition, there are dropped packets, acknowledgements, retries, bus contention, and so on.
Because so many possibilities causes of bad network performance are external to an application, one option you can consider including in an application is a network speed testing facility that reports to the user. This should test the speed of data transfer from the machine to various destinations: to itself, to another machine on the local network, to the Internet Service Provider, to the target server across the network, and to any other destinations appropriate. This type of diagnostics report can tell users that they are obtaining bad performance from something other than your application. If you feel that the performance of your application is limited by the actual network communication speed, and not by other (application) factors, this facility will report the maximum possible speeds to youruser.

Latency
Latency is different from the load-carrying capacity (bandwidth) of a network. Bandwidth refers to how much data can be sent down the communication channel for a given period of time and is limited by the link in the communication chain that has the lowest bandwidth. The latency is the amount of time a particular data packet takes to get from one end of the communication channel to the other. Bandwidth tells you the limits within which your application can operate before the performance become affected by the volume of data being transmitted. Latency often affects the user’s view of the performance even when bandwidth isn’t a problem.
In most cases, especially Internet traffic, latency is an important concern. You can determine the basic round-trip time for a data packets from any two machines using the ping utility. This utility provides a measure of the time it takes a packet of data to reach another machine and be returned. However, the time measure is for a basic underlying protocol (ICMP packet) to travel between the machines. If the communication channel is congested and the overlying protocol requires re-transmissions (often the case for Internet traffic), one transmission at the application level can actually be equivalent to many round trips.
It is important to be aware of these limitations. It is often possible to tune the application to minimize the number of transfers by packing data together, caching and redesigning the distributed application protocol to aim for a less conversational mode of operation. At the network level, you need to monitor the transmission statistics (using the ping and netstat utilities and packet sniffers) and consider tuning any network parameters that you have access to in order to reduce re-transmissions.
TCP/IP Stacks
The TCP/IP stack is the section of code that is responsible for translating each application-level network request (send, receive, connect, etc.) through the transport layers down to the wire and back up to the application at the other end of the connection. Because the stacks are usually delivered with the operation system and performance-tested before delivery (since a slow network connection on an otherwise fast machine and fast network is pretty obvious), it is unlikely that the TCP/IP stack itself is a performance problem.
In addition to the stack itself, stacks include several tunable parameters. Most of these parameters deal with transmission details beyond the scope of the book. One parameter worth mentioning is the maximum packet size. When your application sends data, the underlying protocol breaks the data into packets that are transmitted. There is an optimal size for packets transmitted over a particular communication channel, and the packet size actually used by the stack is compromise. Smaller packets are less likely to be dropped, but they introduced more overhead, as data probably has to be broken up into more packets with more header overhead.
If your communication takes place over a particular set of endpoints, you may want to alter the packet sizes. For a LAN segment with no router involved, the packets can be big (e.g. 8KB). For a LAN with routers, you probably want to set the maximum packet size to the size the routers allow to pass unbroken. (Routers can break up the packets into smaller ones; 1500 bytes is the typical maximum packet size and the standard for the Ethernet. The maximum packet size is configurable by the router’s network administrator.) If your application is likely to be sending data over the Internet and you cannot guarantee the route and quality of routers it will pass through, 500 bytes per packet is likely to be optimal.
Network Bottlenecks
Other causes of slow network I/O can be attributed directly to the load or configuration of the network. For example, a LAN may become congested when many machines are simultaneously trying to communicate over the network. The potential throughput of the network could handle the load, but the algorithms to provide communication channels slow the network, resulting in a lower maximum throughput. A congested Ethernet network has an average throughput approximately one third the potential maximum throughputs. Congested networks have other problems, such as dropped network packets. If you are using TCP, the communication rate on a congested network is much slower as the protocol automatically resends the dropped packets. If you are using UDP, your application must resend multiple copies for each transfer. Dropping packets in this way is common for the Internet. For LANs, you need to coordinate closely with network administrators to alert them to the problem. For single machines connected by a service provider, suggesting improvements. The phone line to the service provider may be noisier than expected: if so, you also need to speak to the phone line provider. It is also worth checking with the service provider, who should have optimal configurations they can demonstrate.
Dropped packets and re-transmissions are a good indication of network congestion problems, and you should be on constant lookup for them. Dropped packets often occur when routers are overloaded and find it necessary to drop some of the packets being transmitted as the router’s buffer overflow. This means that the overlying protocol will request the packets to be resent. The netstat utility lists re-transmission and other statistics that can identify these sorts of problems. Re-transmissions may indicate that the maximum packet size is too large.
DNS Lookup
Looking up network address is an often-overlooked cause of bad network performance. When your application tries to connect to a network address such as foo.bar.somthing.org (e.g. downloading a webpage from http://foo.bar.something.org), your application first translates foo.bar.somthing.org into a four-byte network IP address such as 10.33.6.45. This is the actual address that the network understands and uses for routing network packets. The is this translation works is that your system is configured with some seldom-used files that can specify this translation, and a more frequently used Domain Name System (DNS) server that can dynamically provide you with the address from the given string. DBS translation works as follows:
1. The machine running the application sends the text string of the hostname (e.g. foo.bar.something.org) to the DNS server.
2. The DNS server checks its cache to find an IP address corresponding to that hostname. If the server does not find an entry in the cache, it asks its own DNS server (usually further up the Internet domain-name hierarchy) until ultimately the name is resolved. (This may be by components of the name being resolved, e.g. first .org, then something.org, etc, each time asking another machine as the search request is successively resolved.) This resolved IP address is added to the DBS server’s cache.
3. The IP address s returned to the original machine running the application.
4. The application uses the IP address to connect to the desired destination.
The address lookup does not need to be repeated once a connection is established, but any other connections (within the same session of the application or in other session s at the same time and later) need to repeat the lookup procedure to start another connection.
You can improve this situation by running a DNS server locally on the machine, or on a local server if the application uses a LAN. A DNS server can be run as a “caching-only” server that resets its cache each time the machine is rebooted. There would be little point in doing this if the machine used only one or two connections per hostname between successive reboots. For more frequent connections, a local DNS server can provide a noticeable speedup to connections. Nslookup is useful for investigating how a particular system does translations.

Important Baiscs in performance Testing -2

2009-05-18T22:44:00.000-07:00

Basics: Max. Running Vusers in Analysis is different from Vuser Quantity in Controller?
There are couple of times when an individual starts to pick up LoadRunner ask this question. I've written the explanation here for the convenience of the newbies which I had previously responded in the Yahoo! group discussion.

Max. Running Vuser

It defines at any point of point of time, the maximum number of vusers running together concurrently (in Run state). This is the "state" or usually the requirement of a load test to reach "X" number of concurrent users. If a load test were to required to run 100 concurrent users, then the Max. Running Vuser must be 100. This is different from Vuser Quantity explained in the following.

Vuser Quantity

In the Controller, the Vuser Quantity is the total number of participating in the load test but it is different from Max. Running Vuser which is explained above. To view the total number of Vuser participating the load test (Vuser Quantity), open the Vuser Summary graph. Please take note that the Summary Report in the Analysis session displays the maximum number of vusers running in the scenario and not the total vuser particiapting.

Importanat Basics of Performance Testing-1

2009-05-18T22:39:00.000-07:00

Basics: Transaction Response TIme
What is Transaction Response Time?

Transaction Response Time represents the time taken for the application to complete a defined transaction or business process.

Why is important to measure Transaction Response Time?

The objective of a performance test is to ensure that the application is working perfectly under load. However, the definition of “perfectly” under load may vary with different systems.
By defining an initial acceptable response time, we can benchmark the application if it is performing as anticipated.

The importance of Transaction Response Time is that it gives the project team/ application team an idea of how the application is performing in the measurement of time. With this information, they can relate to the users/customers on the expected time when processing request or understanding how their application performed.

What does Transaction Response Time encompass?

The Transaction Response Time encompasses the time taken for the request made to the web server, there after being process by the Web Server and sent to the Application Server. Which in most instances will make a request to the Database Server. All this will then be repeated again backward from the Database Server, Application Server, Web Server and back to the user. Take note that the time taken for the request or data in the network transmission is also factored in.

To simplify, the Transaction Response Time comprises of the following:
1. Processing time on Web Server
2. Processing time on Application Server
3. Processing time on Database Server
4. Network latency between the servers, and the client
The following diagram illustrates Transaction Response Time.

Figure 1 Transaction Response Time = (t1 + t2 + t3 + t4 + t5 + t6 + t7 + t8 + t9) X 2

Note:
Factoring the time taken for the data to return to the client.

How do we measure?
Measuring of the Transaction Response Time begins when the defined transaction makes a request to the application. From here, till the transaction completes before proceeding with the next subsequent request (in terms of transaction), the time is been measured and will stop when the transaction completes.

Differences with Hits Per Seconds
Hits per Seconds measures the number of “hits” made to a web server. These “hits” could be a request made to the web server for data or graphics. However, this counter does not represent well to users on how well their application is performing as it measures the number of times the web server is being accessed.
How can we use Transaction Response Time to analyze performance issue?
Transaction Response Time allows us to identify abnormalities when performance issues surface. This will be represented as slow response of the transaction, which differs significantly (or slightly) from the average of the Transaction Response Time.
With this, we can further drill down by correlation using other measurements such as the number of virtual users that is accessing the application at the point of time and the system-related metrics (e.g. CPU Utilization) to identify the root cause.
Bringing all the data that have been collected during the load test, we can correlate the measurements to find trends and bottlenecks between the response time, the amount of load that was generated and the payload of all the components of the application.

How is it beneficial to the Project Team?
Using Transaction Response Time, Project Team can better relate to their users using transactions as a form of language protocol that their users can comprehend. Users will be able to know that transactions (or business processes) are performing at an acceptable level in terms of time.
Users may be unable to understand the meaning of CPU utilization or Memory usage and thus using a common language of time is ideal to convey performance-related issues.

My Favourite QTP Blog

2009-05-18T03:58:00.000-07:00

Yes, It's absolutely my favourite blog on QTP.

Visit here, You may be interested this one

Quality Center Interview Questions and Answers

2009-05-06T01:10:00.000-07:00

1. Why is TestDirector used for?

TestDirector is a test management tool. The completely web-enabled TestDirector supports high level of communication and association among various testing teams, driving a more effective and efficient global application-testing process. One can also create reports and graphs to help review the progress of planning tests, executing tests, and tracking defects before a software release.

2. Why are the requirements linked to the test cases?

TestDirector connects requirements directly to test cases, ensuring that all the requirements have been covered by the test cases.

3. What are the benefits and features of TestDirector?

TestDirector incorporates all aspects of the testing process i.e. requirement management, test planning, test case management, scheduling, executing tests and defect management into a single browser-based application. It maps requirements directly to the test cases ensuring that all the requirements have been covered by the test cases. It can import requirements and test plans from excel sheet accelerating the testing process. It executes both manual and automated tests.

4. What is the use of filters in TD?

Filters in TestDirector are mainly used to filter out for the required results. It helps to customize and categorize the results. For eg: to quickly view the passed and failed tests separately filters are used.

5. What is Test Lab?

In the Test Lab the test cases are executed. Test Lab will always be linked to the test plan. Usually both are given the same name for easy recognition.

6. How to customize the defect management cycle in Quality Center?

Firstly one should collect all the attributes that has to be part of the defect management like version, defect origin, defect details, etc. Later using the modify options in QC one can change the defect module accordingly.

7. What is the advantage of writing test cases in Quality Center than writing in excel sheet?

Although creating test cases in excel sheet will be faster than doing it in QC as excel is more user friendly when compared to QC one require to upload them to QC and this process may cause some delay due to various reasons. Also QC provides link to other tests which in turn is mapped to the requirements.

8. What is the difference between TestDirector and Quality Center?

The main difference is QC is more secured than TestDirector. In Quality Center the login page shows projects associated only to the logged in user unlike in Test Director where one can see all the available projects. Also test management is much more improved in QC than TD. Defect linkage functionality in QC is more flexible when compared to TD.

9. What is meant by Instance?

Test instance is an instance of test case in Test Lab. Basically it means the test case which you have imported in Test lab for execution.

10. What is the use of requirement option in TestDirector?

Requirement module in TD is used for writing the requirements and preparing the traceability matrix.

11. Is it possible to maintain test data in TestDirector?

Yes one can attach the test data to the corresponding test cases or create a separate folder in test plan to store them.

12. If one tries to upgrade from TestDirector 7.2 to QC 8.0 then is there risk of losing any data?

No there is no risk of losing the data during the migration process. One has to follow proper steps for successful migration.

13. How is a bug closed in TestDirector?

Once the test cases are executed in the Test Lab and bugs are detected, it is logged as a defect using the Defect Report Tab and sent to the developer. The bug will have 5 different status namely New, Open, Rejected, Deferred and Closed. Once the bug has been fixed and verified its status is changed to closed. This way the bug lifecycle ends.

14. In TD how are the test cases divided into different groups?

In the test plan of TestDirector one can create separate folder for various modules depending on the project. A main module in the test plan can be created and then sub modules be added to that.

15. What is the difference between TD and Bugzilla?

TestDirector is a test management tool. In TD one can write manual and automated test cases, add requirements, map requirements to the test cases and log defects. Bugzilla is used only for logging and tracking the defects.

16. Are TestDirector and QC one and the same?

Yes TestDirector and Quality Center are same. Version of TD 8.2 onwards was known as Quality Center. The latest version of Quality Center is 9.2. QC is much more advanced when compared to TD.

17. What is the instance of the test case inside the Test Set?

Test set is a place containing sets of test cases. We can store many test cases inside test set. Now instance of test case is the test case which you have imported in the test tab. If an another test case has the same steps as this test case till half way then you can create the instance of this test case.

18. What are the various types of reports in TestDirector?

In TD reports are available for requirements, test cases, test execution, defects, etc. The reports give various details like summary, progress, coverage, etc. Reports can be generated from each TestDirector module using the default settings or it can be customized. When customizing a report, filters and sort conditions can be applied and the required layout of the fields in the report can be specified. Sub-reports can also be added to the main report. The settings of the reports can be saved as favorite views and reloaded as required.

19. How can one map a single defect to more than one test script?

Using the 'associate defect' option in TestDirector one can map the same defect to a number of test cases.

20. Is it possible to create custom defect template in TestDirector?

It is not possible to create ones own template for defect reporting in TestDirector but one can customize the template that is already available in TestDirector as required.

21. Can a script in TD be created before recording script in Winrunner or QTP?

Any automation script can be created directly in TD. You need to open the tool (Winrunner or QTP) and then connect to TD by specifying the url, project, domain, userid and password. And then you just record the script like you always do. When you save the script, you can save it in TD instead of your local system.

22. How to ensure that there is no duplication of bugs in TestDirector?

In the defect tracking window of TD there is a “find similar defect” icon. When this icon is clicked after writing the defect, if anybody else has entered the same defect then it points it out.

23. How is the Defect ID generated in TestDirector?

The Defect ID is automatically generated once the defect is submitted in TD.

24. What does the test grid contain?

The test grid displays all the relevant tests related to a project in TD. It contains the some key elements like test grid toolbar with various buttons for commands which are commonly used when creating and modifying the tests, grid filter which displays the filter that is currently applied to a column, description tab which displays a description of the selected test in the test grid and history tab that displays the changes made to a test.

25. What are the 3 views in TD?

The three views in TD are Plan Test which is used to prepare a set of test cases as per the requirements, Run Test which is used for executing the prepared test scripts with respect to the test cases and finally Track Defects which is used by the test engineers for logging the defects.

26. How to upload data from an excel sheet to TestDirector?

In order to upload data from excel sheet to TD firstly excel addin has to be installed, then the rows in the excel sheet which has to be imported to TD should be selected, and then finally the Export to TD option in the tools menu of TestDirector should be selected.

27. How many types of tabs are available in TestDirector?

There are 4 types of tabs available in TestDirector. They are Requirement, Test Plan, Test Lab and Defect. It is possible to customize the names of these tabs as desired.

28. Is 'Not covered' and 'Not run' status the same?

Not Covered status means all those requirements for which the test cases are not written whereas Not Run status means all those requirements for which test cases are written but are not run.

29. How does TestDirector store data?

In TD data is stored on the server.

30. Why should we create an Instance?

Test Instance is used to run the test case in the test lab. It is the test instance that you can run since you can't run the test case in test set.

Set 2 :

1. What is meant by test lab in Quality Centre?
Test lab is a part of Quality Centre where we can execute our test on different cycles creating test tree for each one of them. We need to add test to these test trees from the tests, which are placed under test plan in the project. Internally Quality Centre will refer to this test while running then in the test lab.

2. Can you map the defects directly to the requirements(Not through the test cases) in the Quality Centre?
In the following methods is most likely to used in this case:
Create your Req.Structure
Create the test case structure and the test cases
Map the test cases to the App.Req
Run and report bugs from your test cases in the test lab module.

The database structure in Quality Centre is mapping test cases to defects, only if you have created the bug from Application. test case may be we can update the mapping by using some code in the bug script module(from the customize project function), as per as i know, it is not possible to map defects directly to an requirements.

3. how do you run reports from Quality Centre. Does any one have good white paper or articles?
This is how you do it
1. Open the Quality Centre project
2. Displays the requirements modules
3. Choose report
Analysis > reports > standard requirements report

4. Can we upload test cases from an excel sheet into Quality Centre?
Yes go to Add-In menu Quality Centre, find the excel add-In, and install it in your machine.
Now open excel, you can find the new menu option export to Quality Centre. Rest of the procedure is self explanatory.

5. Can we export the file from Quality Centre to excel sheet. If yes then how?
Requirement tab– Right click on main req/click on export/save as word, excel or other template. This would save all the child requirements

Test plan tab: Only individual test can be exported. no parent child export is possible. Select a test script, click on the design steps tab, right click anywhere on the open window. Click on export and save as.

Test lab tab: Select a child group. Click on execution grid if it is not selected. Right click anywhere. Default save option is excel. But can be saved in documents and other formats. Select all or selected option

Defects Tab: Right click anywhere on the window, export all or selected defects and save excel sheet or document.

6. How many types of tabs are there in Quality Centre. Explain?
There are four types of tabs are available

1. Requirement : To track the customer requirements
2. Testplan : To design the test cases and to store the test scripts
3. test lab : To execute the test cases and track the results.
4. Defect : To log a defect and to track the logged defects.

7. How to map the requirements with test cases in Quality Centre?
1. In requirements tab select coverage view
2. Select requirement by clicking on parent/child or grandchild
3. On right hand side(In coverage view window) another window will appear. It has two tabs
a) Tests coverage
b) Details
Test coverage tab will be selected by default or you click on it.
4. Click on select tests button a new window will appear on right hand side and you will see a list of all tests. You cans elect any test case you want to map with your requirements.

8. How to use Quality Centre in real time project?
Once completed the preparing of test cases
1. Export the test cases into Quality Centre( It will contained total 8 steps)
2. The test cases will be loaded in the test plan module
3. Once the execution is started. We move the test cases from test plan tab to the test lab module.
4. In test lab, we execute the test cases and put as pass or fail or incomplete. We generate the graph in the test lab for daily report and sent to the on site (where ever you want to deliver)
5. If we got any defects and raise the defects in the defect module. when raising the defects, attach the defects with the screen shot.

9. Difference between Web Inspect-QA Inspect?
QA Inspect finds and prioritizes security vulnerabilities in an entire web application or in specific usage scenarios during testing and presents detail information and remediation advise about each vulnerability.
Web Inspect ensures the security of your most critical information by identifying known and unknown vulnerabilities within the web application. With web Inspect, auditors, compliance officers and security experts can perform security assessments on a web enabled application. Web inspect enables users to perform security assessments for any web application or web service, including the industry leading application platforms.

10. How can w add requirements to test cases in Quality Centre?
Just you can use the option of add requirements.
Two kinds of requirements are available in TD.
1. Parent Requirement
2. Child requirements.

Parent Requirements nothing but title of the requirements, it covers high level functions of the requirements

Child requirement nothing but sub title of requirements, it covers low level functions of the requirements.

Advanced Quality Center 9.2 course

2009-05-05T21:11:00.000-07:00

Advanced Quality Center 9.2
INTENDED AUDIENCE
�� Quality Center Administrator
�� Quality Center Project managers
�� Quality Assurance Leads
�� Other Quality Center users responsible for managing projects, users, and workflow customizations
DURATION: 2 DAYS
OVERVIEW
This course covers the tasks that are performed by site administrators, project managers, and Quality Assurance (QA) leads to set up Quality Center (QC) projects before users can work with requirements, test cases, and defects. This hands-on course focuses on planning, creating, and customizing Quality Center projects. It is an important course for people involved in creating projects, modifying existing fields, creating user- defined fields, creating users, setting user permissions, and defining basic workflow.
COURSE OBJECTIVES
At the end of the course, you will be able to:
�� Identify the organization level specifications that need to be customized in Quality Center
�� Configure organization-wide Quality Center parameters
�� Customize project-specific Quality Center parameters •Maintain Quality Center projects
PREREQUISITES
Working knowledge of:
�� Windows
�� Testing concepts
INTRODUCTION TO QUALITY CENTER 9.2
• Identify the Quality Center modules.
PLANNING THE QUALITY CENTER IMPLEMENTATION
• Planning: considerations and key roles.
INTRODUCTION TO SITE ADMINISTRATION
• Logging on to Site Administration.
• Site Administrator components.
• Monitoring user connections and licenses.
CUSTOMIZING PROJECTS AT A GLOBAL LEVEL
• Project structure in Quality Center.
• Creating domains and projects.
MAINTAINING QUALITY CENTER PROJECTS
• Querying project tables.
• Copying, renaming, removing, backing up, restoring , and upgrading projects.
• Deleting Projects and Domains.
CUSTOMIZING USERS AT A GLOBAL LEVEL
• Creating and managing users.
• Configuring user authentication and LDAP import settings.
CUSTOMIZING OTHER QUALITY CENTER MODULES AT A GLOBAL LEVEL
• Configure mail parameters.
• Configure connection parameters.
CUSTOMIZING PROJECT-SPECIFIC ENTITIES
• An overview of Project Customization.
• An overview of customizing project-specific.
• Adding user-defined fields.
• Managing system or user-defined fields.
CUSTOMIZING LOOKUP LISTS
• Creating project-specific lists.
• Associating a list to a project entity.
• Managing project-specific lists.
MANAGING PROJECT-SPECIFIC USERS AND USER GROUPS
• Adding users to a project.
• Creating user groups.
• Customizing module access for user groups.
CUSTOMIZING MAILING OPTIONS AND ALERT RULES
• Designating mail fields.
• Defining mailing conditions.
• Configuring alert rules.
• Customizing Alert Rules.
• Configuring traceability notification rules.
CUSTOMIZING WORKFLOW
• Customizing lists.
• Customizing fields by user groups.
IMPLEMENTING WORKFLOW USING THE SCRIPT EDITOR
• Using the Script Editor.
• Adding a command button by using the Script Editor.

Quality Center 9.2 Course Syllabus

2009-05-05T21:03:00.000-07:00

QC9.2 : Quality Center 9.2 is a browser based Test Management tool which makes the tasks of testers,test leads and Managers very easy by providing the options to manage and organize activities like Requirements coverage, Test Case Management, test case execution report and Defect Management.

Going forward we will dwell into more details about QC 9.2.

Let us see the syllabus coverage of QC 9.2 as of now:

This syllabus is process based Quality Center Hands on course for Professionals :

Prerequisites
For the Using Quality Center 9.2 and Project Planning and
Customization course:
• Working knowledge of Windows and Testing concepts
For the Quality Center 9.2 Dashboard:
• Windows
• Testing concepts
• Quality Center
Course Outline
Using Quality Center 9.2
• Introduction
• Defining Test Requirements
• Analyze Requirement Risks
• Test Planning
• Test Execution
• Defect Tracking
• Reporting and Analysis
Quality Center 9.2 Project Planning and Customization:
• Introduction to Quality Center 9.2
• Planning the Quality Center Implementation
• Introduction to Site Administrator
• Customizing Projects at a Global Level
• Maintaining Quality Center Projects
• Customizing Users at a Global Level
• Customizing other Quality Center Modules at a Global Level
• Customizing Project-Specific Entities
• Customizing Lookup Lists
• Managing Project-Specific Users and User Groups
• Customizing Project-Specific Mailing Options
• Customizing Alert Rules
• Customizing Workflow

Using the Quality Center 9.2 Dashboard
• Introduction
• Dashboard Overview
• Configuring User Groups for the Dashboard
• Normalizing Projects
• Working with Portlets
• Creating and Publishing a Module
• Creating KPIs with SQL
• Dashboard Solutions

YOU are Just 10 Steps away to become QTP Expert

2009-04-25T06:30:00.000-07:00

10 steps to become QTP guru
Would you like to know how to become QTP guru?
The recipe is simple. You should learn the following:

1. VBScript
QTP uses VBScript language. That's why strong knowledge of VBScript is 'must have'.
Sources:
* w3schools' VBScript tutorial
* MS' VBScript Guide.

2. Software Testing Automation Frameworks
To write good automated tests, you should know how to write them.
There are different approaches and solution. There is not a silver bullet.
Sources:
* Test Automation Frameworks
* Choosing a Test Automation Framework
* Totally Data-Driven Automated Testing
* Seven Steps to Test Automation Success

3. HP QuickTest Professional Tutorial
This QTP tutorial cames with QTP.
It's quite clean and informative. Its main goal is to show and explain basic concepts of QuickTest Professional. It provides knowledge on:
* creating a test
* working with Object Repository (OR)
* running and analyzing tests
* verifying tests
* parameterizing, etc
Sources: '\help\QTTutorial.pdf' or '\help\Tutorial.chm' in QTP Install folder.

4. HP QuickTest Professional User's Guide
As previous tutorial, the present User's Guide come with QTP install too.
This guide is intended for QuickTest Professional users at all levels. It gives a deeper knowledge on:
* working with QTP Object Repositoies
* designing tests
* enhancing tests (checkpoints, parameterizing, etc)
* maintaining anf debugging tests
* advanced testing features, etc
Sources: '\help\QTUsersGuide.pdf' or '\help\MainUsersGuide.chm' in QTP Install folder.

5. COM/DCOM Technologies
When working with QTP, you will be closely linked with such Microsoft's applications, as: Excel, Word, Outlook. So, I recommend to get familiar with COM, COM Technologies, and COM objects of:
* MS Internet Explorer
* MS Excel
* MS Outlook

6. SQL
SQL is so important in programming, that I strongly recommend to spend time to learn SQL:
* concepts of RDBM
* selecting/updating/deleting data
* SQL queries optimizations
* databases administration
Sources:
* w3schools' SQL tutorial
* MS SQL Server - this and this
* Oracle DB

7. XML
XML is an extremely popular and useful format. I'm sure that нou will have to deal with data, stored in XML-files.
Sources:
* w3schools' XML tutorial

8. HTML, DOM
Since QuickTest Professional works perfectly with web applications, you should be expert in related fields - HTML, HTTP, DHTML, DOM, etc. They will simplify your future QTP script and make them more reliable and maintenable.
Sources:
* w3schools' HTML tutorial
* w3schools' DHTML tutorial
* A Beginner's Guide to the XML DOM
* DOM tutorials

9. HP QTP Knowledge Base
It contains a lot of practical articles about QuickTest Professional.
You can find there QTP webinars, QTP FAQs, documentations, solutions of your problems or different ways how to improve your QTP automated tests.
Sources:
* HP Self-solve Knowledge Search (requires an access to HP site)

10. Useful sites
Sure, it's impossible to know all information on QTP.
That's why I recommend to use these two sites to search answers to your QTP questions:
* Google
* Easy way to automate testing

LoadRunner Topics

2009-04-15T05:17:00.000-07:00

LoadRunner Certification Topics :

VuGen:

1)Recording Options
2)Run time settings
3)Correlation, correlation studio, correlation rules, auto-correlation.
4)Using the API to customize scripts
5)Parameterization (all types)
6)Command line arguements to the mdrv.exe process (type mdrv.exe into the start->run textfield) Also, in the loadrunn bin directory
7)web_set_sockets_option() there is much you can do with this one function
8)Earlier recording engine, new recording engine

Controller:

1)Scheduler
2)Monitor setup, best practices
3)IPSpoofer
4)Vuser Limits
5)MIListener

LoadGenerator:

1)How it communicates with the Controller
2)IPWizard
3)QTP, Winrunner setup
4)MOFW agent
5)mdrv.exe

< /span>Analysis:

1)Reading/Analyzing graphs
2)Merging graphs
3)Granularization
4)Web page diagnostics

LoadRunner:

1)How to setup the different componenets
2)How licensing works
3)Where do I create LoadGenerators/Controller/VuGen/Analysis?
4)How to determine the type of resources needed to perform a Load Test.

Skills required for performance Engineer

2009-04-09T05:20:00.000-07:00

In nutshell, here are the skills required besides very good SQL Knowledge :

* Network Basics: Knowledge of OSI, TCP/IP stack, DNS, DHCP, WINS, Routers.
* Protocols: Knowledge of Winsock, COM, HTTP, and Citrix.
* Programming/Scripting Languages: Knowledge of HTML, ASP, JSP, Java, C

Information on LoadRunner Certification

2009-04-09T05:09:00.000-07:00

Details, sample questions/ Exam preparation guide can be found:-

HP LoadRunner Software – HP0-M18
ftp://ftp.hp.com/pub/hpcp/epgs/HP0-M18_EPG.pdf

HP Virtual User Generator Software – HP0-M19
ftp://ftp.hp.com/pub/hpcp/epgs/HP0-M19_EPG.pdf

HP Certified Professional Scripting using HP Virtual User Generator Software – HP0-M83
ftp://ftp.hp.com/pub/hpcp/epgs/HP0-M83_EPG.pdf

HP Certified Professional Performance Testing using HP LoadRunner Software – HP0-M84
ftp://ftp.hp.com/pub/hpcp/epgs/HP0-M84_EPG.pdf

Sample Questions:
1.) What is the kind of testing used to check the behavior and performance of each tier?
Ans) Performance Testing
2.) What is the performance test objective which checks whether a new version of the software Adversely affect response time?
Ans) Benchmark Performance Testing, based (benchmarked) on previous version
3.) What is the performance test objective which checks whether the system is stable enough to go into production?
Ans) Detect all the bottlenecks
4.) The load generator should have connectivity to which components?select two? a.) web server b.) vugen c.) controller d.) analysis
Ans) (a),(c)
5.) Which component emulate the steps of real users using the application?
a.) vugen b.)Controller c.)load generator d.) analysis
Ans) a)
6.) What is the component used to monitor application infrastructure components which provide metrics for system tuning?
Ans) Controller
7.) What u will do after doing a debug run?
a.) re-run the scenario b.) tune the system c.) go to full load test and scalability test d.) inform
the project manager that system peformance is good and stop the load testing
Ans) c
8.) Which statement is an example of a conceptual goal?
A. The update transaction should function during heavy usage.
B. The Update transaction must attain 200 concurrent users during peak time.
C. The search transaction should respond within 5 seconds during normal usage.
D. The login transaction should respond within 4 seconds or less during heavy usage.
Ans) D)
9.) What is the LoadRunner term used to define a collection of Vusers within a scenario?
A. Set
B. Group
C. Profile
D. Scenario
Ans) B)
10.)A set of actions or user steps performed within an application to accomplish a business goal is called as
a.) business process
b.) transaction
c.) concurrency
d.) check point
Ans) b)
11.) An example of stress testing :
a.) Purchasing at an e-commerce site
b.)Searches at a search engine site
c.)Updated content display at a news site
d.)Downloading received orders or updating order status at a Client / Server system.
Ans) a)
12.) Master Data comes from :
a.) Application data
b.)user input
c.) System will generate
d.) from scripts
Ans) a)
13.) An example of user generated data
a.) e-mail address
b.) session id
c.) confirmation number
d.) userid and password
Ans) d)
14.) delay between two different iterations can be done by using which run time settings?
Ans) Pacing
15.) Iterations can be set under which option in run time settings dialog box?
Ans) RunLogic
16.) What is the type of concurrency happening when user was in the process of buying tickets?
Ans) ----
17.) What is the type of business process happening when multiple users were buying tickets?
Ans)-----
18.) How many transactions will need to be run per minute if a load test has to be run for four hours with 2000 users, assuming an average length of ten minutes?
Ans) 200 transactions
19.) what is the process to validate that there is enough test hardware available in the test environment,benchmark the business processes against the testing hardware?
Ans) Smoke Test
20.) vugen should have connectivity to which component to record vuser script?
a.) AUT(application under test)
b.) analysis
c.) controller
d.) load generator
Ans a)
21.) u r trying to change the server sampling rate and u couldn't do it what u should u do to make the process the successful?
a.) disconnect load generator and try again
b.) close loadrunner and open again
c.) shut down the corresponding server and try again
d.) u couldn't do that because u cannot change server sampling rate
22.) When u click start scenario option:
a.) load generator will automatically go to ready state and do the necessary operation
b.) U need to manually click the connect button in load generator window
c.) start wlrun.exe
d.) ping the controller
Ans) a)
23.) Before the vusers are executed, first they are
a.) initialized
b.) they will execute the scripts directly in AUT from load generator
c.) paused and then will do the intended operation
d.) vusers will go to rendezvous state
Ans) a)
24.) Option where vusers and groups will be automatically scheduled is
a.) Manual scenario
b.) Goal oriented scenario
c.) real life schedule
d.) Run until completion
Ans) d)
25.) What is the scenario outline for the condition update should successfully happen for 1000 concurrent users
a.) script should define update and 1000 concurrent users should run the script
b.) script should define update and 1000 users should run the script
c.) update transaction with manual scenario
d.) update transaction with goal oriented scenario
Ans) d)
26.) the type of file to be choosed in scenario groups is
a.) .lrr file
b.) .lra file
c.) .usr file
d.) .lrs file
Ans) d)
27.) What is the minimum value for transaction monitor frequency for small scenarios?
Ans) 1 minute
28.) What is the minimum value for transaction monitor frequency for large scenarios?
Ans) 1 minute
29.) if in initial stages of scenario execution during ramp up, if u feel that response time is
increasing, what u will do to run the scenario with current users?
a.) stop the scenario and decrease the no of vusers
b.) press the pause button in interactive schedule graph to continue with current no of users
c.) pause the users in run tab
d.) design a different scenario
Ans) a)
30.) what is the common run time settings for all protocols?
Ans) General
31.) What run-time settings we need to have for debug run?
Ans) Replay think time, Extended logs on,No pacing
31.) run time settings for the whole controller can be accessed from?(menu flow)
Ans)From controller's Run-time setings button
32.) What is the term u call to pass different input to a script?
Ans)Parameterisation
33.) Results of scenario execution are stored in
a.) controller machine
b.) load generator machine
c.) vugen machine
d.) analysis machine
Ans) Load generator,if you collate the results it will be copied in controller machine
34.) An intersection point in a business process is called as?
Ans) Rendezvous point
35.) extended log and data returned by server will be used for
a.) debug run
b.) isolate top time transaction
c.) full load
d.) scalability load
Ans) a
36.) What is the HTTP status code which indicates an individual business process is failing under load, or the Web application itself has crashed.
a.) 404
b.) 502
c.) 500
d.) 200
Ans) c)
37.) What are the two types of scheduling available in controller?
Ans) Schedule by Scenario, Schedule by Group
38.) what is the option in analysis used to isolate a specific measurement?
a.) drill down
b.) filter
c.) Autocorrelate
d.) graularity
Ans) b)
39.) What is the graph used to get transaction throughout the test?
a.) Average Transaction response time
b.) Response time per second
c.)Transaction response time distribution graph
d.)Transaction response time percentile graph
Ans) c)
40.) If number of hits become flattened, then the issue is with
a.) application server
b.) web server
c.) database server
d.) authentication server
Ans) b)
41.) If throughput graph becomes flattened out then the issue is with
a.) ramp up
b.) bandwidth
c.) firewall
d.) network
Ans) b)
42.) server time and network time can be identified using graph
a.) time to first buffer breakdown
b.) page download time breakdown
c.) Page Component Breakdown (Over Time) Graph
d.) Downloaded Component Size Graph
Ans) a)
43.) What is the type of graph displayed below
a.) Transaction Response Time (Distribution) Graph
b.) SLA graph
c.) Summary graph
d.) Transaction breakdown graph
Ans) Where is the Graph????
44.) what are the main formats in which we can produce analysis results(choose any two)
a.) ms excel
b.) crystal reports
c.) ms word
d.) HTML
e.) XML
Ans) c),d)
45.) Tell me two business process u will record out of which displayed below?
BusinessProcessNameTypical Day Peak Day DynamicContent MissionCritical Record
Sign in 70/hr 210/hr Light High ?
Create new account 10/hr 15/hr Moderate Low ?
Search for flights 130/hr 180/hr Moderate Moderate ?
View flight booking 20/hr 30/hr Moderate High ?
Ans) business process---Sign-in, Business process-Search for Flights
46) Which option in the Analysis tool allows you to compare a measurement in a graph with other measurements during a specific time range of a scenario and view similar trends?
A. Drill Down
B. Apply Filter
C. Merge Graphs
D. Auto Correlate
Answer: D
47) Which Analysis graph details transaction response times as a function of load?
A. Transactions per Second
B. Average Transaction Response Time
C. Transaction Response Time Under Load
D. Transaction Response Time (distribution)
Answer: C
48) During analysis of a scenario, you realize that the throughput becomes flat as Vusers continue to increase. What is the likely cause?
A. a bandwidth problem
B. a database server problem
C. a web server connection problem
D. an application server connection problem
Answer: A
49) Which option in the Analysis tool allows you to see the results of two graphs from the same load test scenario in a single graph?
A. Drill Down
B. Apply Filter
C. Merge Graphs
D. Auto Correlate
Answer: C
50) Which Analysis graph shows the number of transactions against transaction response times?
A. Transactions per Second
B. Average Transaction Response Time
C. Transaction Response Time Under Load
D. Transaction Response Time (distribution)
Answer: D
51) In the Analysis tool, what allows you to modify the scale of the x-axis?
A. think time
B. granularity
C. event type
D. transaction hierarchical path
Answer: B
52) Which Analysis graph details the delay for the complete path between the source and destination machines?
A. MS IIS
B. Windows Resource
C. Network Delay Time
D. Time to First Buffer Breakdown
Answer: C
53) Click the Exhibit button.Which type of graph does the image in the exhibit represent?

A. Merged graph
B. Drill Down graph
C. Cross Scenario graph
D. Auto Correlated graph
Answer: C
54) What does the elapsed time in the Scenario Status window refer to?
A. the time that has elapsed from when all Vusers have finished Initializing
B. the time that has elapsed from when all Vusers were in the running state
C. the time that has elapsed from when you press the start Scenario button
D. the time that has elapsed from when the first Vusers entered the running state
Answer: C
55) You configure a scenario to start with a delay of <05:00:00> (HH:MM:SS) just before leaving from work. You return the next morning to find that the scenario did not run. Why would this happen?
A. You failed to save the scenario.
B. You failed to save the schedule.
C. You failed to connect to the load generators.
D. You failed to press the START SCENARIO button.
Answer: D
56) What provides you with the level of activity associated with each business process during each hour of a typical work day?
A. Conceptual goals list
B. Business process profile
C. Task distribution diagram
D. System configuration diagram
57) When would you need to perform path translation for a scenario execution?
A. When you use Storage Area Network (SAN) to store the scripts and results
B. When you can use the Universal Naming Convention (UNC) for accessing the shared network drive being used to store scripts and results
C. When the scripts and results are configured to be stored on the current Vuser machine
D. When the shared network drive being used to store results and scripts is mapped to a different drive letter on the Controller versus the Load Generator
58) You executed a load test using scripts running on load generators placed at various points on the network. You need to determine the response times by load generator for the scenario run. Which features (used independently) within the Analysis tool, can help you accomplish this? (Select two.)
A. Auto correlate
B. Merge graphs
C. Set filter
D. Cross with result
E. Drill down

Check list before starting Performance testing

2009-04-09T05:07:00.000-07:00

Before starting loadtest we have to make sure the following check lists: :)

1.End users, customer, and project members have been notified in advance of the execution dates and hours for the capacity test
2.All service level agreements, response time requirements have been agreed upon by all stakeholders.
3.Contact list with names and phone numbers has been drafted for support personnel (onsite and remote)
4.Functional Testing of the application has been completed.
5.Restart the controller machine.
6.Ramp Up / Duration / Ramp Down is configured correctly.
7.All Load Generators are in a "Ready" status.
8.All Load Generators are assigned to appropriate scripts.
9.All scripts have the correct number of Vusers assigned to them.
10.All scripts have the correct number of Iterations assigned to them.
11.Correct pacing has been agreed upon and configured for all appropriate scripts.
12.Logging is set to Send messages only when an error occurs for all scripts.
13.Think Times have been enabled/disabled in the test scripts
14.Generate snapshot on error is enabled for all appropriate scripts.
15.Timeout values have been set to the appropriate values.
16.All content checks have been updated for the appropriate scripts.
17.Rendezvous points have been enabled/disabled for appropriate scripts
18.All necessary data has been prepared/staged and is updated in all scripts.
19.Any scripts with unique data requirements have been verified.
20.All scripts have been refreshed in the controller and reflect the most recent updates.
21.IP Spoofing has been enabled in the controller.
22.IP Spoofing has been configured on all appropriate Load Generators.
23.All LoadRunner Monitors have been identified, configured and tested.
24.Auto Collate Results should be enabled
25.Results directory and file name should be updated.

Yielding effectiveness from test cases : Metrics to be collected

2008-11-08T06:14:00.000-08:00

These are the best metrics for test leads and managers to get effectiveness.

Who wants to acquire the knowledge on metrics or who wants to implement the below metrics in the real projects.Recently I faced an incident in project, where I need to get effectiveness of test cases of the existing Application.I followed the some pointers in the list of 44 metrics. If you feel you have any more metrics other than the below list.Please feel free to post here.

“44” Metrics for Test Case Review Effectiveness
1. Major Defects Per Test Case Review
2. Minor Defects Per Test Case Review
3. Total Defects Per Test Case Review
4. Ratio of Major to Minor Defects Per Test Case Review
5. Total Defects Per Test Case Review Hour
6. Major Defects Per Test Case Review Hour
7. Ratio of Major to Minor Defects Per Test Case Review Hour
8. Number of Open Defects Per Test Review
9. Number of Closed Defects Per Test Case Review
10. Ratio of Closed to Open Defects Per Test Case Review
11. Number of Major Open Defects Per Test Case Review
12. Number of Major Closed Defects Per Test Case Review
13. Ratio of Major Closed to Open Defects Per Test Case Review
14. Number of Minor Open Defects Per Test Case Review
15. Number of Minor Closed Defects Per Test Case Review
16. Ratio of Minor Closed to Open Defects Per Test Case Review
17. Percent of Total Defects Captured Per Test Case Review
18. Percent of Major Defects Captured Per Test Case Review
19. Percent of Minor Defects Captured Per Test Case Review
20. Ratio of Percent Major to Minor Defects Captured Per Test Case Review
21. Percent of Total Defects Captured Per Test Case Review Hour
22. Percent of Major Defects Captured Per Test Case Review Hour
23. Percent of Minor Defects Captured Per Test Case Review Hour
24. Ratio of Percent Major to Minor Defects Captured Per Test Case Review Hour
25. Percent of Total Defect Residual Per Test Case Review
26. Percent of Major Defect Residual Per Test Case Review
27. Percent of Minor Defect Residual Per Test Case Review
28. Ratio of Percent Major to Minor Defect Residual Per Test Case Review
29. Percent of Total Defect Residual Per Test Case Review Hour
30. Percent of Major Defect Residual Per Test Case Review Hour
31. Percent of Minor Defect Residual Per Test Case Review Hour
32. Ratio of Percent Major to Minor Defect Residual Per Test Case Review Hour
33. Number of Planned Test Case Reviews
34. Number of Held Test Case Reviews
35. Ratio of Planned to Held Test Case Reviews
36. Number of Reviewed Test Cases
37. Number of Unreviewed Test Cases
38. Ratio of Reviewed to Unreviewed Test Cases
39. Number of Compliant Test Case Reviews
40. Number of Non-Compliant Test Case Reviews
41. Ratio of Compliant to Non-Compliant Test Case Reviews
42. Compliance of Test Case Reviews
43. Non-Compliance of Test Case Reviews
44. Ratio of Compliance to Non-Compliance of Test Case Reviews

CMMI Levels and 22 Key Process Areas

2008-09-25T20:38:00.000-07:00

There are five CMM levels, of which Level 5 is the highest.

CMM Level 1 is called "Initial". The software process is at CMM Level 1, if it is an ad hoc process. At CMM Level 1, few processes are defined, and success, in general, depends on individual effort and heroism.

CMM Level 2 is called "Repeatable". The software process is at CMM Level 2, if the subject company has some basic project management processes, in order to track cost, schedule, and functionality. Software processes are at CMM Level 2, if necessary processes are in place, in order to repeat earlier successes on projects with similar applications. Software processes are at CMM Level 2, if there are requirements management, project planning, project tracking, subcontract management, QA, and configuration management.

CMM Level 3 is called "Defined". The software process is at CMM Level 3, if the software process is documented, standardized, and integrated into a standard software process for the subject company. The software process is at CMM Level 3, if all projects use approved, tailored versions of the company's standard software process for developing and maintaining software. Software processes are at CMM Level 3, if there are process definition, training programs, process focus, integrated software management, software product engineering, intergroup coordination, and peer reviews.

CMM Level 4 is called "Managed". The software process is at CMM Level 4, if the subject company collects detailed data on the software process and product quality, and if both the software process and the software products are quantitatively understood and controlled. Software processes are at CMM Level 4, if there are software quality management (SQM), and quantitative process management.

CMM Level 5 is called "Optimized". The software process is at CMM Level 5, if there is continuous process improvement, if there is quantitative feedback from the process, and from piloting innovative ideas and technologies. Software processes are at CMM Level 5, if there are process change management, and defect prevention technology change management.

These Key Process Areas in CMMI Level are indicated in the below table.

Abbreviation Name Area Maturity Level
REQM Requirements Management Engineering 2

PMC Project Monitoring and Control Project Management 2

PP Project Planning Project Management 2

SAM Supplier Agreement Management Project Management 2

CM Configuration Management Support 2

MA Measurement and Analysis Support 2

PPQA Process and Product Quality Assurance Support 2

PI Product Integration Engineering 3

RD Requirements Development Engineering 3

TS Technical Solution Engineering 3

VAL Validation Engineering 3

VER Verification Engineering 3

OPD Organizational Process Definition Process Management 3

OPF Organizational Process Focus Process Management 3

OT Organizational Training Process Management 3

IPM Integrated Project Management Project Management 3

RSKM Risk Management Project Management 3

DAR Decision Analysis and Resolution Support 3

OPP Organizational Process Performance Process Management 4

QPM Quantitative Project Management Project Management 4

OID Organizational Innovation and Deployment Process Management 5

10 Truths of SLAs for IT Organization

2008-09-25T20:20:00.000-07:00

When I am Searching for a Freelance testing project in google, I came across a site where the project provider mentioned that I need to adhere to SLAs.I know about it but still the point he mentioned is : "As I am not IT savvy, you must adhere to standard SLAs,and if find ,your payments will be stopped.Now I digged out more into SLAs and lists out the key points here.
1.SLAs are a highly useful communication vehicle.
2. SLAs outline services provided, performance levels, and legal ramifications. According to the ASP Industry Consortium’s “Buyers Guide to Service Level Agreements,” Information that should be contained in an SLA includes the purpose of the SLA, description of service, duration of service, installation timetable, payment terms, termination conditions, and legal issues such as warranties, indemnities, and limitation of liability.
3. Business goals are always detailed in the SLA.
4. Performance levels drive pricing.
5. SLAs are always customized to the particular purpose.
6. Metrics are always outlined for service performance.
7. The supplier of the services is not the only (and should not be the only) party responsible for monitoring compliance.
8. The consumer and the supplier of a service are jointly responsible for the terms of the SLA.
9. An SLA is not a guarantee of service. However, nothing should be agreed upon unless it is determined to be a commercially reasonable and viable objective.
10. Remediation, revision, and renegotiation are always options.

Testing Metrics : A measure of Quality

2008-09-24T21:19:00.000-07:00

Yesterday one of my friend had given interview for an MNC in India with an experience of 4 years in testing.The interviewer asked him two Questions :

1. Where do you want to see after 4 years?
Well,as a go - getter, my friend answered it very enthusiastically that he want to be in QA Team. You can guess the next question what it would be.Here it is :

2. His second question is all about : What you know about metrics system in QA and tell me some sample metrics you collect and their formulas.

The guy knows little bit about metrics and it became a biased question for him.

After that we met and discussed about all and he asked me is there any website to learn metrics with formulas. I thought I can provide a Post in this blog and it will be better for all of us to learn .

That's why this post is here .Hope you will enjoy the post and content I took it from www.Exposys.com and you can see the same at www.exposys.com

or Here is the best book about metrics(Click on the link) and you can read it online for free or you can order a hard bounded copy of the same. Let us see now how and what are the metrics interested for Testers.

The Product Quality Measures:

1. Customer satisfaction index

This index is surveyed before product delivery and after product delivery
(and on-going on a periodic basis, using standard questionnaires).The following are analyzed:

* Number of system enhancement requests per year
* Number of maintenance fix requests per year
* User friendliness: call volume to customer service hotline
* User friendliness: training time per new user
* Number of product recalls or fix releases (software vendors)
* Number of production re-runs (in-house information systems groups)

2. Delivered defect quantities

They are normalized per function point (or per LOC) at product delivery (first 3 months or first year of operation) or Ongoing (per year of operation) by level of severity, by category or cause, e.g.: requirements defect, design defect, code defect, documentation/on-line help defect, defect introduced by fixes, etc.

3. Responsiveness (turnaround time) to users

* Turnaround time for defect fixes, by level of severity
* Time for minor vs. major enhancements; actual vs. planned elapsed time

4. Product volatility

* Ratio of maintenance fixes (to repair the system & bring it into compliance with specifications), vs. enhancement requests (requests by users to enhance or change functionality)

5. Defect ratios

* Defects found after product delivery per function point.
* Defects found after product delivery per LOC
* Pre-delivery defects: annual post-delivery defects
* Defects per function point of the system modifications

6. Defect removal efficiency

* Number of post-release defects (found by clients in field operation), categorized by level of severity
* Ratio of defects found internally prior to release (via inspections and testing), as a percentage of all defects
* All defects include defects found internally plus externally (by customers) in the first year after product delivery

7. Complexity of delivered product

* McCabe's cyclomatic complexity counts across the system
* Halstead’s measure
* Card's design complexity measures
* Predicted defects and maintenance costs, based on complexity measures

8. Test coverage

* Breadth of functional coverage
* Percentage of paths, branches or conditions that were actually tested
* Percentage by criticality level: perceived level of risk of paths
* The ratio of the number of detected faults to the number of predicted faults.

9. Cost of defects

* Business losses per defect that occurs during operation
* Business interruption costs; costs of work-arounds
* Lost sales and lost goodwill
* Litigation costs resulting from defects
* Annual maintenance cost (per function point)
* Annual operating cost (per function point)
* Measurable damage to your boss's career

10. Costs of quality activities

* Costs of reviews, inspections and preventive measures
* Costs of test planning and preparation
* Costs of test execution, defect tracking, version and change control
* Costs of diagnostics, debugging and fixing
* Costs of tools and tool support
* Costs of test case library maintenance
* Costs of testing & QA education associated with the product
* Costs of monitoring and oversight by the QA organization (if separate from the development and test organizations)

11. Re-work

* Re-work effort (hours, as a percentage of the original coding hours)
* Re-worked LOC (source lines of code, as a percentage of the total delivered LOC)
* Re-worked software components (as a percentage of the total delivered components)

12. Reliability

* Availability (percentage of time a system is available, versus the time the system is needed to be available)
* Mean time between failure (MTBF).
* Man time to repair (MTTR)
* Reliability ratio (MTBF / MTTR)
* Number of product recalls or fix releases
* Number of production re-runs as a ratio of production runs

Metric = Formula

Test Coverage = Number of units (KLOC/FP) tested / total size of the system. (LOC represents Lines of Code)

Number of tests per unit size = Number of test cases per KLOC/FP (LOC represents Lines of Code).

Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria

Defects per size = Defects detected / system size

Test cost (in %) = Cost of testing / total cost *100

Cost to locate defect = Cost of testing / the number of defects located

Achieving Budget = Actual cost of testing / Budgeted cost of testing

Defects detected in testing = Defects detected in testing / total system defects

Defects detected in production = Defects detected in production/system size

Quality of Testing = No of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100

Effectiveness of testing to business = Loss due to problems / total resources processed by the system.

System complaints = Number of third party complaints / number of transactions processed

Scale of Ten = Assessment of testing by giving rating in scale of 1 to 10

Source Code Analysis = Number of source code statements changed / total number of tests.

Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual Effort for Design and Documentation

Test Execution Productivity = No of Test cycles executed / Actual Effort for testing

WebSphere : Performance Testing and Analysis

2008-09-24T21:06:00.000-07:00

This article provides advice and methods for finding and resolving common performance problems with IBM® WebSphere® Portal. The WebSphere Portal SEAL team engages with high-profile customers experiencing significant problems with their WebSphere Portal deployments. In roughly 50 percent of these engagements, the major complaint is performance related. The task usually becomes one of finding and resolving these performance issues, often after the system has already been put into production.

Finding and fixing performance problems in a production environment is challenging on a number of levels. Optimally, most bottlenecks in the system should be found and fixed before the system is allowed into production. This article explains a tested process that can ensure that, with high probability, most of the significant performance issues are found and addressed before you promote a system to production. Performance testing and analysis have three main objectives:

1. Determining the load level at which a system under test fails
2. Finding bottlenecks in a system that throttle throughput and removing them as soon as possible and practical
3. Capacity planning, for example, predicting the amount of horsepower needed to sustain defined users loads within agreed-upon service level agreements (SLAs)

The system is defined as the complete end-to-end set of components required to deliver the requested Web page to the requesting user’s browser. The most visible and often the most troublesome components tend to be WebSpherePortal itself, the Websphere Portal database, the Lightweight Directory Access Protocol (LDAP), and the back-end systems (databases, application servers, and so forth) that supply content to the portlets.

The back-end systems in many systems tend to present the most risk in WebSphere Portal deployments because they are frequently maintained by separate organizations. This separation dilutes the communication channel between the WebSphere Portal deployment team and the back-end teams with respect to performance objectives.

The methodology presented in this article is a holistic approach that meets all three of the objectives when it is executed successfully.

The environment

To meet the performance test objectives outlined previously, the performance test environment needs to be either the production environment itself or a mirror of it that has, as practically possible, the same hardware, the same topology, and the same back-end systems. If any piece of this complex test topology is different from its production counterpart, you must extrapolate the results in the test environment to predict its effect in the production environment. These extrapolations generally require detailed implementation knowledge of the portal and the deployed applications, which generally are not available to the testing organization. By making the test environment equivalent to the production environment, your confidence in the test results as they relate to what actually happens in production becomes acceptable.

An important goal of the test environment is the repeatability of results. As slight changes are made in the system, repeatability ensures that you can accurately measure the effect of these changes. For that reason, it is optimal to have the system on an isolated network during the performance testing. Running the performance test on the production network introduces variability (for example, user traffic) that can skew such metrics as page render response time. There is also a more pragmatic reason to isolate the performance test network. Putting WebSphere Portal under stress likely puts the corporate network under stress. This stress is often problematic during normal business hours.

If placing the performance test on an isolated network is not feasible, you should at least try to ensure that the components of the test are all collocated on the same subnet of a network router. Normal WebSphere Portal best practice recommends using a gigabit Ethernet connection between the portal and its database. Optimally, this connection extends to the LDAP servers, the Web servers, and important back-end services. It is crucial that the load generator also be on a local segment to the Web server and/or the portal itself.

A common customer concern involves the load generators being on the same local LAN segment as the portal itself. In this case, “This test does not get a true picture of the performance of the system as it excludes the network from the data center to the users.” The answer to this concern is often difficult for customers to accept. The process described here is for tuning and resolving issues with the portal and its surrounding components. Trying to tune the network between the users (or the load generators) and the portals makes the analysis and problem resolution needlessly complex. We therefore remove it from the test. There are far better tools and processes for network tuning than the processes used here.

Portal infrastructure baseline

In contrast to the mirrored production environment, it is strongly advisable to also conduct an incremental set of baseline tests that exercise the infrastructure. At that point, subsequent tests should gradually augment the portal with customer-written code. The test plan should thus move from a simple topology to the final production topology to make it easier to isolate problematic components.

The first test is the complete WebSphere Portal infrastructure using an out-of-the-box portal. Transfer the database, and enable security. Make sure that all front-end Web servers, firewalls and load balancers are in place. Security managers (for example, Computer Associates SiteMinder or IBM Tivoli® Access Manager) should also be in place and correctly configured. Create a simple home page with a couple of portlets that do not access any back-end systems (for example, the World Clock portlet). Create a simple load testing script that accesses the unauthenticated home page and then logs in (authenticates) and idles without logging out. From this point, you want to add simulated users (Vusers) until the system is saturated. Using the bottleneck analysis techniques described below, find and fix any bottlenecks in the infrastructure. Note the performance baseline of this system.

Now, add to the system any customized themes and skins, and repeat the previous test. Find and fix any important bottlenecks in the revised system. Finally, as described below, add the actual portlets to be used on the home page and perform bottleneck analysis.

This baseline environment can be very effective in finding bottlenecks in the infrastructure that are independent of the application. Further, it can provide a reference when analyzing the extent to which the applications place additional load above and beyond the basic WebSphere Portal infrastructure.

Your strategy is to conduct the same tests listed below for bottleneck analysis in this baseline environment, optimize the environment, and then perform bottleneck analysis with the actual applications.

Application of the Portal Tuning Guide recommendations

Apply the recommendations outlined in the WebSphere Portal Tuning Guide to all systems before you embark on any performance testing. The guide provides a good starting point because it fixes known performance inhibitors in a default WebSphere Portal installation. Although bottleneck analysis would likely find the same problems, it is better to remove them from the beginning.

Load generation

A proper performance test also requires the use of a load generator that produces simulated user requests for Web pages. It is important that this tool produce such metrics as response time and page views per second. These metrics allow you to determine when the system fails its SLA contract or is saturated to the point that injecting more page requests per unit time does not result in higher page production. Saturation is discussed later in this article. The generator’s ability to aggregate data such as CPU utilization on the portal and HTTP servers as well as mod_status data from the HTTP server aids in problem determination.

A number of tools are commonly used to create Web traffic (also known as drive load) in the test system. Some of the more commonly used tools include Mercury LoadRunner, Borland SilkPerformer, and IBM Rational® Performance Tester.

It is important that the load generator have sufficient virtual users (vUsers) to drive the system to saturation. Note that virtual users do not map directly to actual users. A virtual user represents an active channel on the load generator. A virtual user may simulate multiple actual users; however, only one actual user can be active for each virtual user.

It is also important, especially in a WebSphere Portal context, that if the system requires authenticated access to the WebSphere Portal applications under test, that sufficient unique test user IDs exist in the LDAP directory and that scripts ensure that only a reasonable number of duplicated logins occur during the test. (A reasonable number in this context accounts for the fact that some users might have a couple of instances of the browser open, each with the same WebSphere Portal login ID.) WebSphere Portal has a large caching infrastructure for portal artifacts. These artifacts are generally cached on a per-user basis. If the load simulation uses the same user ID for all tests, performance appears artificially high because the artifacts do not need to be loaded from the LDAP directory and the database.
The general methodology

The sections below describe the iterative process that is used to do the actual analysis of the system. The sections that precede "The process" define concepts that are important to understand during the execution of that process.
User scenarios

To tune the WebSphere Portal system to handle large numbers of users and to accurately predict its ability to handle specific numbers of users correctly, it is important to determine the most probable scenarios for users of the system. The test must then accurately simulate those user scenarios using the load generator. One effective way to do this step is to list the most likely use cases. Write a script for each of these use cases or as many as are practical. Now, assign a probability of likelihood that a percentage of the whole user population will execute that scenario. As the test is run, assign use cases to Vusers in the same proportion as the expected general population. As the number of Vusers is ramped up (discussed later), try to maintain this proportion.

NOTE: "Vuser" is a LoadRunner term. It represents one active channel over which requests are made and returned.
Think time

Think time is the average amount of time that a normal user pauses during individual mouse clicks or key presses during the course of using WebSphere Portal. In the load generation tools, this time is usually programmable, yielding a random time within a predefined range.

As think time is reduced, the number of requests per second increases, which in turn increases the load on the system. Reducing think time generally increases the average response time for WebSphere Portal login and page-to-page navigation. Therefore, accurately estimating real user think time is important for producing an accurate model of the system in production, particularly for capacity planning.

In most use cases, a think time of 10 seconds plus or minus 50 percent is reasonable for a portal having experienced users. A figure closer to 30 seconds is more reasonable for a portal with inexperienced users.
Cookies and sessions

Generally, most real users log into the portal and execute the task that needs to be done; however, they rarely log out by using the logout button. Rather, they let the browser sit idle until their session times out. Typically, a lot of sessions in memory are waiting for cleanup pending the WebSphere Application Server session timeout. This behavior increases the Java™ Virtual Machine (JVM) heap working set, which increases the probability of heap exhaustion in the JVM. Heap exhaustion can be both a performance bottleneck and a cause for a JVM failure.

Effective simulations must model this behavior of users who do not explicitly log out. As each individual simulation executes a particular use case, it should end the use case by going idle as opposed to logging out. As the script cycles back around to log in a new user on this particular Vuser, the cookies for old session (typically JSESSIONID) and Lightweight Third-Party Authentication (LTPA) along with any application-specific cookies need to be cleaned up appropriately before logging in the next user using that script. This model also implies that sufficient test IDs need to exist so that a test ID can sit idle for the length of the WebSphere Application Server session timeout without risk of being reused until the previous session times out.
Metrics

It is important that the scripts be instrumented for metrics. The most important metrics are login response time along with page-to-page response times. Most of the load generators already provide aggregate Page View per second (PV/s) metrics. PV/s is the most important metric for determining the saturation point of the system.

At the conclusion of each test, a graph of Vusers ramp rate versus the three metrics is required for doing analysis.

In addition to the metrics gathered by the load generation tool, a system monitoring tool such as IBM Tivoli Composite Application Manager for WebSphere or the Computer Associates Wily IntroScope product should be employed. These tools run on the WebSphere Portal instance and instrument the JVM directly. They are useful in both detection and resolution of system bottlenecks, vUsers, or think time.

A common misconception is that to accurately simulate a large population that generates requests at a certain rate, a smaller number of users that generate requests with a smaller think time will suffice.

It's important to note that the effects of running with a small number of users and a low think time result in unrealistically high cache hit rates. It also means that too few sessions are created. Because session size is often a serious problem for many portlet applications, this approach gives an unrealistically good view of the system performance and leads to surprises in production.

Another poor practice is running a small set of vUsers with no think time.
Repeatability principle

In a large population, it is easy to assume that most user actions appear to be random as users navigate through the portal. Experienced users, though, typically use the same patterns over and over. Furthermore, from a test engineering perspective, the user scenarios need to be reasonably static so that system changes can be effectively measured from run to run.

Therefore, the definition of the repeatability principle is that for all runs of a particular scenario, the metrics (average response time, PV/s, saturation point, and so on) produced by the runs all converge to the same results if the runs are sufficiently long. Note that with more variation (that is, unique scenarios) in the test scripts, longer times are required to converge, on average.

The simulation scripts written for the performance tests should adhere to the repeatability principle.

Driving to saturation

Saturation is defined as the number of active Vusers at which point adding more Vusers does not result in an increase in the number of PV/s. Note that this saturation point is for a given simulation; each different simulation likely has a different saturation point. The saturation point varies depending on the usage pattern.

To effectively drive a system to saturation, add Vusers a few at a time, let the system stabilize , observe whether PV/s increase, and add more Vusers as possible. ("Stabilize," in this context, means that the response times are steady within a window of several minutes.) On LoadRunner, if you plot Vusers against throughput (PV/s), the PV/s initially rises linearly with the number of Vusers, then reaches a maximum and actually decreases slightly from that point. The saturation point is the number of Vusers at which the PV/s is at maximum.

Back to top

Bottleneck analysis

The goal of bottleneck analysis is to remove impediments to driving the system to a higher load. The metric defined for higher load is a higher number of PV/s at saturation. Therefore, bottleneck analysis removes impediments to improve the saturation point.

Bottlenecks in a WebSphere Portal environment under load are generally the result of contention for shared resources. This contention can be the result of synchronized Java classes, methods, or data structures, contention for serial resources (for example SystemOut.log) or excessive response times in back-end databases or Web servers. You must also be mindful of bottlenecks such as the network itself. Components such as routers and firewalls can impose congestion control or can be poorly tuned.

As load increases, contention for these resources increases, making contention locks easier to detect and correct. This detail is why effective load testing is a requirement for bottleneck analysis.

A common mistake is to focus only on page response times. Many performance testers prefer to optimize render response times because this delay is the most obvious user requirement. This type of performance analysis requires path length reduction in the customer portlet applications. Response time optimization is generally more appropriately done in a non-loaded system and with tooling specific to the task (for example, JProbe).

The process

The process of performing bottleneck analysis is straightforward. For a particular performance analysis (for example, LoadRunner) simulation, follow these steps:

1. Ramp a single WebSphere Portal JVM to saturation.
2. Determine the bottlenecks that exists at saturation.
3. Resolve the bottlenecks.
4. Unless satisfied with system capacity, go to step 1 and find the next bottleneck.

Note that this process is iterative. The key concept is that you fix one bottleneck to find the next bottleneck.

A single JVM is used in this case because detection of the bottleneck is much simpler. Finding and resolving cross-JVM contention can be quite complex. After a single JVM has been tuned as much as desired, you move on to the capacity planning analysis for multiple nodes as described later in this article.
Note on ramp rates

A common question in performance testing is the rate at which Vusers should be ramped into the system.

Do not ramp in several hundred users as quickly as possible until the system collapses. This approach is not representative of reality, and it does not provide repeatable results.

You should model reality. Predict or measure the actual highest ramp rate that you would expect your portal to endure. This rate might typically occur during the hours that your users most often log into your portal, such as first thing in the morning when they arrive at the office. We recommend that you ramp a small fixed number (for example, two Vusers per minute) for a set period of time (for example, five minutes), do not add any users for a time to let the system stabilize (for example, five minutes), then loop back and add another batch of Vusers in the same fashion.

This technique gives the portal time to fill the various caches in an orderly fashion and provides for the ability to more accurately detect saturation points.
Priming the portal

After a portal restart, a short script should be executed prior to the main test to preload certain caches (for example, WebSphere Portal access control and the anonymous page cache before the real test starts. Failure to do so can skew the initial response times inordinately.

Analysis techniques

After you have a portal at saturation, you can determine the cause of a bottleneck in the system by taking a Java thread dump (using a kill -3 command ) against the portal Java process under test. A thread dump shows the state of all threads in the JVM. The general procedure is to look for threads that are all blocked by the same condition or are all waiting in the same method of the same class.

In general, search for threads that are blocked or in a wait state. By ascertaining why certain classes statistically show up blocked, you can then proceed to remove that reason and thus remove the bottleneck. The next section discusses some common bottleneck problems. Going into all the problems is really the art of WebSphere Portal bottleneck analysis and takes time and experience to master.

If the bottleneck is not the WebSphere Portal JVM itself, detection and resolution techniques are varied and outside the scope of this article.

Common problems

This section lists common problems that many customers have seen during their performance testing.
Logging

Logging using direct writes to SystemOut.log or using a logging class such as log4j causes serialization between running threads and significantly degrades portal performance. In production portal systems, log only what is absolutely needed. When using log4j, log only errors; do not log warnings or informational messages. If logging is required for audit purposes, consider using a portal service or a different service running in a separate JVM.

Turn off all logging and remove all debug code that writes to files before doing performance testing.
Java class and variable synchronization

Use of method-level synchronization blocks where a method is in a monitor wait (MW) state with one method holding a lock can be problematic. In this case, you have Java code that is synchronized and is causing serialization in the system.

Use of synchronized class variables or synchronized HashMaps can also cause this problem.

In both cases (method or variable synchronization), the problem can be exacerbated by arbitrarily increasing the number of WebSphere Application Server transport threads in which the portal runs. By increasing the number of threads, you increase the probability of hitting portal code that is synchronized in this fashion, which ultimately serializes all the threads.
Database contention

If the thread dump indicates numerous threads waiting in Java Database Connectivity (JDBC) classes in Socket.read() methods, then there are likely response time issues in the database itself.

When threads are waiting on JDBC pool resources in WebSphere Application Server, you see the threads in a condition wait (CW) state in the WebSphere Application Server connection pool (J2C) classes. In this case, you might need to increase the pool size for this data source, or you might need to increase the number of connections that the database server can handle concurrently.
LDAP responsiveness

If several threads are in the Socket.read() method of the Java Naming and Directory Interface (JNDI) classes, they are likely waiting on results from the LDAP directory.
Excessive session sizes

If customer-written portlets are storing too much data in the session, that condition invariably leads to memory and performance issues.
Exceptions being thrown

Even though this problem might seem obvious, in many customer situations performance analysis and bottleneck reduction are attempted in systems that are repeatedly throwing exceptions in the logs. When the JVM is handling unchecked exceptions, it slows the JVM down and causes serial I/O (printing) to the SystemOut.log print stream, which serializes the WebSphere Application Server transport threads.

A more general issue involves trying to characterize and tune a system that is inherently flawed. All results that are generated in such an environment must be labeled as non-repeatable and subject to change (potentially in a significant way) as the flaws are eliminated.

Finally, it should be your policy that the WebSphere Portal system is not allowed to enter a high-load production environment with any errors in the logs.
Dynacache concerns DRS replication modes

WebSphere Portal requires that the WebSphere Application Server Dynamic Cache Service be enabled. The dynamic cache (or “dynacache” as it is commonly known) is a data structure that is commonly used to provide caching of data from back-end services (for example, database results) in WebSphere Portal. Dynacaches can ensure cache synchronization across a cluster of WebSphere Portal members. For proper operation in a cluster, WebSphere Portal requires that cache replication be enabled. The default mode of replication, PUSH, can cause performance problems, though, in the WebSphere Portal environment. This setting is contained on the deployment manager on a per portal cluster member (JVM) basis.

The use of NOT SHARED is strongly recommended for the vast majority of WebSphere Portal configurations. Three actions are needed to ensure that the node is fully enabled for WebSphere Portal. The first is to set the replication mode to NOT SHARED using the WebSphere Application Server console for each cluster member. The second is to install PK64925. The third is to install PK62457. Depending on which WebSphere Portal service level is installed, these two APARs might already be installed.

WebSphere Content Manager’s dynacaches also should be set to NOT SHARED. To complete this task, in the Deployment Manager console, navigate to Resources - Cache Instances - Object Cache Instances and change each of the individual cache instances. As of the time of this writing, there are 11 instances for WebSphere Content Manager.
Dynacache eviction concerns

Since WebSphere Portal version 5.1.0.2, the size of the WebSphere Portal dynacaches has been increased to a default that is appropriate for most customer’s WebSphere Portal applications. There are situations, though, in which these defaults are inadequate and can cause significant performance problems. For example, if a portal has a large number of derived pages with a common parent in WebSphere Portal V5.1.0.x, the portal access control (PAC) caches can be small enough to cause cache thrashing. Similarly, if the portal objectID cache is too small, thrashing occurs.

Customers need to install and use the advanced dynacache monitor and monitor all the caches. If one or more of the caches seem to have large amounts of least recently used (LRU) evictions, the size of that cache might need to be increased. The sizes of the WebSphere Portal caches are mostly located in the CacheManagerService.properties file.
Customer portlets

Some common problems noted from past customer engagements include the following:

1. Use of synchronized class variables.
2. Excessive database calls. Consider using DB caching layers or dynacache to reduce the load on application databases or back-end services.
3. Unsynchronized use of HashMaps. There are timing scenarios in which these classes get into infinite loops if separate threads hit the same HashMap without being synchronized.

Back to top

Capacity planning

The goal of capacity planning is to estimate the total number of WebSphere Portal JVMs required that satisfy a certain user population within predetermined SLA metrics prior to entering production.

Typical metrics include these:

1. Portal login response time (typically around four seconds)
2. Page-to-page response times after being already logged in (typically around two seconds)

The process

The process for running the load test looks very much like the one for running the test for bottleneck analysis except that there is now a second criterion for stopping the test. One criterion is saturation, as previously defined. The second criterion is failure of any of the SLA metrics.

If the test reaches saturation before any of the SLA metrics are exceeded and if it has already been determined that there are no bottlenecks that can or will be excised, then you can immediately calculate the number of nodes required.

If the SLA metrics are exceeded before reaching saturation, then you must analyze the failure to determine the next course of action. If you determine that you do not need to resolve the response time issues, then proceed directly to calculating the number of nodes, as discussed in the next section of this article.

Extrapolating results

In general, if a single WebSphere Portal node can sustain n users within given SLA metrics, then 2 nodes can sustain 1.95 * n users. The accepted horizontal scaling factor for a portal is .95. Thus, if a single WebSphere Portal node can sustain n users within given SLA metrics, then m nodes can sustain:

n (1 + .95 + .952 + .953 + … + .95 m )

Thus, the horizontal scaling factor is slightly less than linear.

This scaling factor assumes that the database capacity does not bottleneck the system. In fact, this scaling factor is primarily a metric of the degeneration of the WebSphere Portal database for logging in users.

Vertically cloning (scaling) is somewhat different. Vertical cloning is indicated when a single JVM saturates a node at a processor utilization around 80 percent or less. Note that in most cases, bottleneck analysis usually provides relief. In the absence of Java heap issues, a single JVM can usually be tuned to saturate a node at 85 to 90 percent processor utilization.

Vertical scaling is discussed more fully later in this article.

Testing with the full cluster

If sufficient load generation capacity exists (including test IDs), it is wise to do a final series of tests in which the whole user community is simulated against the full cluster to ensure viability of the entire system.

Failover testing

If there is a system requirement for full performance during a failover, this scenario should also be scripted and tested.

Before running this scenario, review the plugin-cfg.xml file at the HTTP server to ensure that the cluster definitions are correct. Consider adding the parameter ServerIOTimeOut to the cluster members. This parameter augments the ConnectIOTimeout parameter. ConnectIOTimeout is the amount of time before a cluster member is marked as down in the event that the remote server fails to open a socket connection upon request. The parameter is normally present in the plugin-cfg.xml file and defaults to 0, which means that it relies on the operating system to return timeout status to the plug-in instead of the plug-in explicitly timing the connection itself.

The parameter ServerIOTimeout is, by default, not included in plugin-cfg.xml. This parameter sets a time-out on the actual HTTP requests. If the portal does not answer in the allotted time, the server is marked down. This step is useful because there are certain classes of failures whereby the WebSphere Portal cluster member opens a socket upon request, but the JVM has hung and will not respond to HTTP requests. Without ServerIOTimeout, the plug-in does not mark the cluster member as down; however, it is not able to handle requests. This situation results in requests being routed to a hung server.

During this test, start with the cluster fully operational. Enable Vusers in your simulation to the maximum number that your SLA mandates. Then, stop one or more cluster members. You can do this step gracefully by stopping the cluster members from the deployment manager or by simulating a network failure by removing the Ethernet cable from a cluster node. Many other failure modes might be worth investigating (for example, database failures, Web service failures, and so on). After the simulated cluster member outage, ensure that the surviving cluster members handle the remaining load according to your system requirements. Then, restart the offline cluster members to ensure that the load returns to a balanced state over time.

Ongoing capacity planning

If a system is already in production and is meeting its current SLA goals, you also want to plan for future growth in the number of users of the system. Assuming that the applications on the WebSphere Portal do not significantly change, you can derive the necessary measurements and calculations from a running production system. You need proper tooling, though, to take the measurements.

In short, if n JVM can support x users, then each JVM can support (x/n)^(1/.95) users. Using the formula explained previously, you can easily plan for future growth.

Back to top

Vertical clustering considerations

A common technique for improving performance is to vertically clone the WebSphere Portal JVM on the same physical system. Engineers initially assume that if one JVM is good, two must be better.

The ultimate goal of vertical cloning is to increase the net aggregate throughput in transactions per second of the sum of the cluster members (clones) on a single node. This goal is usually possible only if, when running under the load, a single, well-tuned cluster member does not consume most of the CPU available in that node. In fact, in a well-tuned WebSphere Portal, vertical cloning always carries a cost. Vertical cloning is indicated when the benefits outweigh the costs.

WebSphere Application Server clustering comes in two flavors. The first is the horizontal type. In this arrangement, a functionally equivalent duplicate of an application server is created on another node. This duplication is done with a WebSphere component known as the deployment manager. The resulting set of equivalent nodes is known as a cluster. The result is that a front-end HTTP server can forward a request from a client to either of the cluster members (clones), and the result is identical.

Similarly, you can also create cluster members vertically, which means that multiple JVMs are created on the same node. Each cluster member can serve the same content just as in the horizontal cluster member case.

In the WebSphere Portal case, each cluster member shares the one (and only one) WebSphere Portal database. This statement changes slightly in WebSphere Portal V6, but it is true for V5.x. Therefore, as the number of cluster members increases, the WebSphere Portal database has a higher likelihood of becoming a bottleneck due to the dilution of its capacity.

Costs of vertical clustering

When additional cluster members are active on the same physical node, costs are associated with it. First, there is process context switching. The operating system must now manage additional processes (JVMs).

Second, there is more contention for processor resources. Generally, vertically clustering is always a bad choice if the number of active cluster members exceeds the number of processors in the node less one. You should never have three cluster members on a three-processor node, for example. Two cluster members on a three-processor node might be acceptable under certain conditions.

Indications for vertical clustering

This section describes some of the situations in which vertical cluster members provide value.
Reliability

Apart from performance concerns, having additional cluster members might make sense strictly for reliability reasons. If a WebSphere Portal installation is a single node, then in the event of a software failure that crashes one JVM (without crashing the operating system), you can mitigate the effect of the crash by adding vertical cluster members. The assumption is that most software failures are localized to a single JVM and do not affect the others on the same node. Therefore, the cluster continues serving requests while the failing JVM is restarted.
Memory utilization

In a 32-bit operating system, process address spaces are limited to 4 gigabytes of memory. Most operating systems split this space as 2 gigabytes of user space and 2 gigabytes of kernel space. There are exceptions whereby the user space can be increased to 3 gigabytes and the kernel reduced to 1 gigabyte (Solaris, AIX®, and Microsoft® Windows® 2003 Enterprise, for example).

If the address space available to the JVM is 2 gigabytes, then the JVM can allocate approximately a 1.5-gigabyte heap space.

There are cases when the combination of the WebSphere Portal base memory working set, along with the total memory required for all the portlets running during stress, could approach and exhaust the 1.5-gigabyte heap. When this happens, and if there is still a significant amount of processor resource available (20 to 30 percent or more), then vertical cloning could increase the total throughput of the box by effectively creating 3 gigabytes of JVM heap and dividing the workload evenly between the two 1.5-gigabyte heap JVMs.
Java synchronized methods and class variables

If your WebSphere Portal application (and the portal itself) uses enough synchronized methods or class variables, you can, under load, end up with a high and frequent number of blocked threads in the application server. You can identify this situation by taking thread dumps under load and noticing that there are lots of Web container threads sitting in MW state waiting for these synchronized artifacts.

In this case, reducing the maximum number of Web container threads on a per-cluster-member basis reduces these stalls. If, after that change, the processor is not consumed as described previously, then vertical cloning can increase the aggregate throughput for the whole node.

Back to top

Conclusion

With proper testing before putting WebSphere Portal into production, you can remove many common performance problems, thereby providing for a much smoother user experience. This article provided a framework for building the test plan and execution processes needed to ensure that performance is acceptable and predictable as the system is deployed to production.

Resources

* Participate in the discussion forum.

* Learn more about IBM WebSphere Extended Cache Monitor.

* Refer to the IBM WebSphere Portal Information Center tips on performance tuning.

* Refer to the IBM WebSphere Portal Performance and Tuning Guide.

About the author

Alex Lang joined IBM in 1982. Since that time he has had various technical and management assignments in networking, digital signal processing, Java advocacy, and IBM WebSphere. He is currently the technical team lead for the WebSphere Portal SEAL team. His primary focus is resolving critical customer situations with the architecture, deployment, and operation of WebSphere Portal.

Webshpere monitoring on remote server

2008-09-23T21:37:00.000-07:00

Dear Reader,

This post I have taken from IBM forum to relate it to the monitoring with Loadrunner.

TO monitor Web application Servers,First we need to configure the servers to allow monitoring by Loadrunner.

Let us see an example from the IBM forum to set up the servers to monitor websphere application server from remote server :

In this topic, a Web application, cviewer, is taken as an example. Before you start monitoring its status, you need to ensure the following:

1. Ensure WASCE is successfully installed on two different machines, for example, serverA and serverB.
2. Start the servers on the two machines. Under a privileged user or root, enter the following command to start up the server: WASCE_HOME/bin/startup(.sh|.bat). Ensure the server is started successfully.

To monitor the status of cviewer on a remote server, take the following steps:

1. Install the cviewer application on serverA. Ensure the cviewer application is started correctly by checking the URL http://\[server ip:port]/cviewer/.
2. Define a monitor for serverA on the serverB:
1. Log in to serverB administrative console with the default account/password "system/manager".
2. Add a server first. Click Server -> Monitoring, and then click Add server, fill in the following fields and click Add.
* Name=serverB
* IP/Hostname=[IP or hostname of serverB]
* Protocol=[default=EJB]
* Port=[default=4201]
* Username=system
* Password=manager
* Password(verify)=manager
3. Start server status query. Click "Enable Query" and ensure Stat.Query column is running.
4. Add the web module "cviewer" to "Statistics Collected". Under WebModule, click + on the line of cviewer. The cviewer will be listed under Live Statistics. The following data is collected:
* Active Session Count - shows the number of active sessions in use currently
* Expired Session Count - shows the number of expired sessions
* Rejected Session Count - shows the number of sessions that were not created because the maximum number of active sessions was reached.
* Processing Time - shows the cumulative processing times of requests by all servlets
* Session Alive Time Count - the count of sessions which are alive in the same time duration
* Session Alive Time MaxTime - the maximum time an expired session ever lived
* Session Alive Time MinTime - always equals -1
* Session Alive Time TotalTime - the total live time of sessions which are alive in the same time duration
* Session Count - the total number of sessions created by this manager
* Startup Time - the time in milliseconds it took to start this context
* TLD Scan Time - the time used to scan the TLD contents of all tag library descriptor files (including those inside the jars that are "accesible" to this Web application) and register any application event listeners found there.
3. To delete the server monitoring, go back to the Monitoring portlet, click Edit in the Actions column, and then click Delete this server.
4. To view serverA's live statistics, click serverA to view the live statistics.
5. To stop the server on serverA, enter the command WASCE_HOME/bin/shutdown.sh
6. Check serverA's status in serverB's console.

In the next Post, Let us relate these concept to configure the server to monitor it from LoadRunner.

Dear Readers,
Hope you enjoy this post and let us know if you have any comments or suggestions on this.

What are the skills required for LoadTester

2008-09-23T21:08:00.000-07:00

To become a DECENT run of the mill Loadrunner engineer,

I would first recommend understanding the various processes and
theory's of performance testing, tuning. After all LoadRunner is
simply a tool to facilitate the process. If you don't know the
process you're throwing darts drunk and blindfolded.

HardSkills which serve well...
C programing.
Basic understanding of the Network Stack and a little bit of
everything else. You don't have to know Data Modeling but you should
know SQL, You don't need to know Diferential Algebra but you should
understand Statistics. You don't need to know queuing theory but you
should learn how to determine a bottle neck.

SoftSkills which serve well...
project management, charisma (you'll need to beg, bribe, charm or
threaten to get your requirements)
* Components such as web servers, application servers, database servers, operating systems, networks and network elements such as load balancers.

Here you can the summary of skills of LoadTesters in a nutshell

* You need not have "guru" level knowledge of each of the components but should have operational knowledge and an understanding of the performance issues associated with the components.
* For example, a load tester should know what multi-way joins, indexes and spin counts are and what affect they have on a database server.

* Protocol(s) used between the client and server such as HTTP/HTML, ODBC, SQL*NET, and DCOM.

* The LoadRunner script language is ANSI C. It helps to know the C language, but the scripts are generated and manipulated by LoadRunner, so there is usually not need to directly edit the code. There is also a icon script view which completely hides the C code.

* Load testing is not a heads down coding exercise. You will work with many parts of an organization to coordinate activities, schedules and resources.

* Daily interaction with a variety of people requires good oral and written communication skills as well as good people skills. If you prefer to sit in a cube by yourself, you should stay in functional testing or development.

Relate Rstatd With LoadRunner

2008-09-23T21:02:00.000-07:00

LoadRunner description	LoadRunner ref [1]	Rstatd value ref [2]	Rstatd type ref [2]	sun.mib variable (all counter data types) ref[3]	UCD-SNMP-MIB variable ref [4]	UCD-SNMP-MIB data type ref [4]
Average number of processes in ‘Ready’ state during the last minute	Average load	avenrun_1 avenrun_5 avenrun_15	float	MISSING	ucdavis.laTable.laEntry.laLoad.1 (1m ave) ucdavis.laTable.laEntry.laLoad.2 (5m ave) ucdavis.laTable.laEntry.laLoad.3 (15m ave)	String
Collisions per second detected on the Ethernet	Collision rate	colls	counter	rsIfCollisions	MISSING	N/A
Number of switches between processes or threads, per second	Context switch rate	cswitch	counter	MISSING	ucdavis.systemStats.ssRawContexts.0	Counter32 (number of switches)
Percent of time that the CPU is utilized	CPU utilization	cpubusy	counter	rsSystemProcessTime + rsUserProcessTime	ucdavis.systemStats.ssCpuRawSystem.0 + ucdavis.systemStats.ssCpuRawUser.0	Counter32 (100ths of seconds)
Rate of disk transfers	Disk rate	disk	counter	rsDiskXfer1 + rsDskXfer2 + rsDiskXfer3 + rsDiskXfer4	ucdavis.systemStats.ssIORawSent.0 + ucdavis.systemStats.ssIORawReceived.0	Counter32 (number of blocks)
Errors per second while receiving Ethernet packets	Incoming packets error rate	ipkts	counter	rsIfInErrors	if.ifInErrors	Counter32
Incoming Ethernet packets per second	Incoming packets rate	ierrs	counter	rsIfInPackets	if.ifInOctets.	Counter32
Number of device interrupts per second	Interrupt rate	intr	counter	rsVIntr	ucdavis.systemStats.ssRawInterrupts.0	Counter32
Errors per second while sending Ethernet packets	Outgoing packets error rate	oerrs	counter	rsIfOutErrors	if.ifOutErrors.1	Counter32
Outgoing Ethernet packets per second	Outgoing packets rate	opkts	counter	rsIfOutPackets	if.ifOutOctets.1	Counter32
Number of pages read to physical memory, per second	Page-in rate	pgin	counter	rsVPagesIn	ucdavis.systemStats.ssRawSwapIn.0	Counter32 (number of blocks)
Number of pages written to pagefile(s) and removed from physical memory, per second	Page-out rate	pgout	counter	rsVPagesOut	ucdavis.systemStats.ssRawSwapOut.0	Counter32 (number of blocks)
Number of pages read to physical memory or written to pagefile(s), per second	Paging rate	pgin + pgout	counter	rsVPagesIn + rsVPagesOut	ucdavis.systemStats.ssRawSwapIn.0 + ucdavis.systemStats.ssRawSwapOut.0	Counter32 (number of blocks)
Number of processes being swapped(in)[1]	Swap-in rate	pswpin	counter	rsVSwapIn	MISSING	N/A
Number of processes being swapped(out)	Swap-out rate	pswpout	counter	rsVSwapOut	MISSING	N/A
Percent of time that the CPU is utilized in system mode	System mode CPU utilization	Not explicitly stated	Not explicitly stated	rsSystemProcessTime	ucdavis.systemStats.ssCpuRawSystem.0	Counter32 (100ths of seconds)
Percent of time that the CPU is utilized in user mode	User mode CPU utilization	Not explicitly stated	Not explicitly stated	rsUserProcessTime	ucdavis.systemStats.ssCpuRawUser.0	Counter32 (100ths of seconds)

[1] Though described as “number of processes” by Mercury, the underlying counters appear to represent number of pages swapped in/out in ref[2] and [3].