Software is a set of computer instructions contained within a file that can be accessed by using a filename. Software is also called a program or an application. In the mid-1960s, mathematicians proved that any program could be constructed using three flowchart shapes or structures.
A structure is either a
Sequence - a series of instructions that do not contain a branch to another instruction caused by an if or loop control structure.
Selection - is a control structure governed by an if keyword.
Loop - is a control structure that can cause a series of instructions to repeat.
Software Testing is the controlled verification of a set of data being input to a control structure and verifying that the output from that control structure meets expected results. The testing should include stress testing (input data that will stress the boundary conditions) and tests that emphasize or mimic how the end-user will use the product. On one project that I worked on, we developed a partnership with some of our customers that enabled us to use
some of their applications to test our code. This step enabled us to see how the customer was using our product and it helped us to develop more stressful testing.
"Media reports in January of 2005 detailed severe problems with a $170 million high-profile U.S. government IT systems project. Software testing was one of the five major problem areas according to a report of the commission reviewing the project. Studies were under way to determine which, if any, portions of the project could be salvaged."
I think that the above quote refers to VCF. I do not have any first hand knowledge of why the above project is in trouble, but many projects with this amount of dollar failure fail because
Ill defined specifications and no early prototype
Poor Architecture - high-level design and prototype
Poor Design - low-level design
If the specifications are ill defined or not understood, then the end product is usually not going to be well received.
I strongly believe that if the high level guidelines are followed, the success of the project is improved.
Testing does not inject defects into software. Defects are injected during one or more of the following phases.
Requirements Definition - if they are ambiguous and can lead to ill defined specifications
Requirements Analysis - sometimes in the abstract, customers cannot see the end product and sign off on requirements that can lead to a product they don't want or cannot use.
This is where a prototype is very useful.
High Level Design
Low Level Design
Coding - Implementation
Because of inadequate test/functional coverage, areas of the code may not be fully tested, and this may lead to defects not being detected.
On a project, the Test Manager reported defects uncovered during system test for three consecutive weeks. On the fourth week, the IS manager blew up and asked the Test Manager why he kept finding problems. The IS manager should have asked the development manager why his group was injecting so many defects.
Many IS managers do not understand the development cycle and how defects are injected. Therefore, they often do not make the correct decisions on how to manage a development cycle.
What does it take to get management to take quality assurance seriously? First, you must understand that most managers are measured on cost. Defects/bugs add to the cost of developing a product. Usually, it is more cost effective to remove defects early in the development cycle. For example, a company I used to work for estimated
that a defect reported by the customer cost $23,000.00 to fix and have that fix shipped to the customer. The cost of each defect was charged back to the effected development organization. Therefore, the cost for developing a product had a line item that was accumulated from the cost of fixing defects throughout the lifecycle of that product.
In short, quality can and does affect the profitability of a product.
If you can show the cost of removing defects at each level of the development cycle, you can usually get the attention of management. For instance, a problem injected during high-level design could propagate into 25 problems by System Test and leak nine problems to the customer. Depending on the severity of the design problems, it is possible that a
development effort could be scrapped (see VCF). Consider that a problem is injected during high-level design; the following chart shows the cost of detecting/removing the problem.
Low-level design review
Function Test Case
System Test Case
The numbers shown in the table above will vary depending on the size and efficiency of an organization.
It should be noted that many high-level design problems are not detected until
After product ship - this may be the first time the customer sees the product.
System Test - product does not work properly because of poorly defined interfaces.
The lesson to be learned is that the end-customer should be involved throughout the development cycle and development should make extensive use of prototypes. In addition, it is more cost effective to gear the development cycle toward defect prevention than in defect detection.
This question should be asked in different two ways. For example,
Why are there bugs in software?
Why are bugs shipped with the product?
These questions will be addressed in the following sections.
Why are there bugs in software?
Software development is not as precise or as mature a discipline as mathematics or engineering. Some people want to treat software development as an art form instead of a science.
Suppose you receive the requirement to sort an array in ascending order, how would you proceed? As the developer, you may think that the requirement calls for in-line code. As a user of the requirement, you may be expecting a subroutine. As a second user of the code,
you may be expecting a method in a class. Without further clarification, as in an interface document, there may be several implementations based on erroneous interpretations of the original sort requirement. This is a rather simple case, but you can see how
more complicated requirements that are poorly worded can lead to misunderstandings. Java provides at least sixteen sort methods for arrays Luckily; the requirement specified the sort was for arrays.
Suppose the writer of the requirement was intending that the sort implementation was for tables and records and then you can imagine the problems that would occur if the developers thought the requirements were for arrays.
Let's take this example further. Assume the writer of the sort requirement was intending the sort routine was for schedulers and processes were to be dispatched based on priority. The implementations could be disastrous in a radar/missile application if the developers thought the requirement was for arrays.
There could be errors in the implemention. Suppose the requirement was for arrays and the developer implemented a solution that could only handle unique values. What would happen, as in the case of a telephone directory, if the requirement called for handling unique and non-unique values?
Anyone who has ever gone on a project to add to an existing application, where there is no documentation for the existing code, knows horror stories. I once saw code where the maintainer of the code would write subroutines when in-line code was the better way to implement the fix. Can you imagine the confusion that occurred when there were subroutine fixes calling subroutine fixes four and five levels deep to get to the latest fix. I saw this type of code in Y2K legacy code.
Adding to the confusion was the lack of architectural or design documentation.
Why are bugs shipped with the product?
Sometimes developers get creative and add code that is not called for in the specifications. Since the testers do not know about this code, they may never test for it. Therefore, code that has defects are shipped without being tested.
The test coverage may keep up with new requirements, in the case of changing requirements occ
urring late in the development cycle. This is another example of code not being tested.
I have seen examples of code that has been rigorously tested and it passed System Test. However, one or more customers use the code in ways that were not anticipated, which caused the shipped code to fail. This is a case where the developer needs to know how the customers will use the shipped code.
In some organizations, the emphasis is on shipping code and making ship dates. The focus is not on testing and code quality. I have seen cases where the Development group missed date after date. To make the ship date, the Development Manager cut back on testing the code. The result was that defects were shipped to the customer. Some organizations have the philosophy of using the customer to test the code.
I once heard a manager say; "We'll fix the code once it has shipped." I think this philosophy has cost us dominance in the automobile industry and may be a cause for outsourcing.
A test organization does not have a test matrix, automated test cases or coded test cases and they wonder why they are shipping defects to the customer. The Test organization needs to verify that they have test coverage for the code that is being shipped and the test cases can be run in a repeatable fashion.