7.    Test Plan

 

As much thourough testing as possible must be done on the IFESS system before it’s delivery to ensure the highest quality product possible. Our testing will be completed in 5 stages:

·        Database Testing

·        Module Testing

·        Integration Testing

·        Product Testing

·        Acceptance Testing

 

In order to streamline the development process and locate bugs as early as possible, testing will be taking place concurrently with programming.

 

7.1.      Test Stages

 

7.1.1             Stage 1: Database Testing

 

 The first stage involves testing the datatabase. The database will be the first component constructed by the development team and database code inspection will happen concurrently with the programming. As components of the database are completed they will be delivered to the inspection team for inspection and review.

 

After the database has been completed some black box testing of the database will be performed using query testing to ensure the integrity of the database structures and to ensure the database meets the specified requirements.

 

7.1.2      Stage 2: Module Testing

 

As modules are delivered by the development team, inspection of module code will be performed by the inspection team to ensure that it meets requirements and to find as many bugs as possible. After inspection of each module, modules will be black box tested individually in preparation for integration.

 

7.1.3      Stage 3: Integration Testing

 

After each module is tested individually it will be integrated into the IFESS framework. More black box tests will be performed to ensure compatibility between dependent modules. In the case where a module is added to the framework that does not share any depencies with any other modules, only simpler smoke tests will be performed.

 

7.1.4             Stage 4: Product Testing

 

After all the modules have been integrated into the frame work, GUI and full product testing will be performed.  As IFESS GUI itself contains very little functionality, a brief inspection of the GUI code will be performed to ensure that the correct module code is executed in the correct places.

 

Full black box product testing will also commence at this time. During this stage, as much testing as possible of all product functionality will be executed given the time available. Special steps must be taken to ensure the sample tests delivered by Fly-By-Wire will be executed and verified to ensure success during the system demo.

 

7.1.5             Stage 5: Acceptance Testing

 

The acceptance testing stage of this project will be relatively simple. A customer demonstration is planned for November 27. Tests executed during this demonstation are the acceptance tests for our project. All acceptance tests will have been executed in the product testing phase to guarantee their flawlessness during the demo.

7.2.      Schedule

Due to the event driven nature of the IFESS system and its utlilization of third party applications code inspection and black box product testing will be the focus. All testing will be completed along the following time schedule.

 

7.3.      White box testing

 

The software development team will be tackling the most complicated subsystems first and then work their way to the more trivial systems, while the testing team will follow right on their heels. As soon as a module is complete, the testing team will commence inspecting every line of the code, communicating any problems, and then moving onto the next module when it becomes available.

 

Every aspect of the source code for the database, functional modules and GUI will be inspected with respect to the following criteria:

 

Completeness            Checks that every aspect of the requirements agreed upon with Fly-by-Wire are implemented

Conformity How closely the code/functionality conforms to the specifications agreed upon with Fly-by-Wire

Reliability Verifies that the code will not crash or fail upon any input.  If the code produces an appropriate error at a given point, this must be handled in a dignified manner

Integrity When extenuating circumstances are encountered, such as heavy use or corrupt input from external sources, the code must still function in an appropriate way

Statement coverage Every statement in the code must be inspected with no exceptions.

Branch coverage Every branch in the code must be inspected with no exceptions. The execution guards must be checked and verified that they are correct.

 

When inconsistencies such as a logical error or bug is found in the code, a bug report will be generated describing exactly where the problem is and, if possible, what the solution should be. This may require consultation with the development team in cases when the problem is not very clear. When a comparison of the code functionality is made with the Fly-by-Wire specifications and it is found to be lacking, a report will be generated as well describing the omition and the adjustments required.

 

As noted in the introduction white box testing will occur during stages 1, 2, and 4 of testing.  It will be used to test the database, functional modules and the GUI.  In other words, all operational modules (modules with detailed functionality) will be white box tested while logical modules will not.  A schedule of when each module will be ready can be found in the management plan..

7.4.      Black box testing

 

Black box testing of the IFESS framework will provide extensive and comprehensive tests to take every module to its limit. After each module has been completely verified the module under test (MUT) is ready for integration. Every MUT undergoes a number of very strict tests: equivalence tests, boundary value tests, functional tests and integration tests. These tests must be designed to simulate as many input scenarios and possible.

Bug reports will be generated and delivered to the development team when bugs are found. Black box testing will

 

7.4.1.              Equivalence Tests

 

Every input into the MUT will be broken into separate equivalence classes: classes in, classes below, and classes above the desired input range.  Every element in these classes is expected to perform the same way so only a few inputs from each class will need to be tested.

 

7.4.2              Boundary Value Tests

 

The boundry where the equivalence classes are divided is a key point for errors. Every equivalence class will have its boundary values tested; two for each class.

 

7.4.3              Functional Tests

 

Since our modules are very independent we have the ability to do functional tests. We will test all the major functions in the MUT to ensure its proper use. The inputs used for these functional tests will be equivalence classes with boundary values that correspond to the input the module will be receiving.

 

7.4.4              Integration Tests

 

Once a module has been thoroughly tested, it is then integrated with the remainder of the system. The major functions of a module will be tested with the same input as with the module test. The results should be the same.

 

7.4.5              Sample Black Box Test Plan

 

Every black box test performed will utilize these tactics. The following is a sample test case for the Movies module:

 

 

Function: listMovie(genre)

 

 

Equivalence class 1

 

Genre exists in the system

 

 

Equivalence class 2

 

Genre does not exist in the system

 

 

Border cases:

 

First genre entered in the system

Last genre entered in the system

 

 

 

Test case 1

Input

equivalence class 1

Expected output

·         List of movies contained in the genre, if no movies in the genre a blank list will be returned

Preconditions

·         At least three different genres are entered into the system

·         “genre” is the input from the user indicating which genre is desired

·         Database queries have all been tested and work

Postconditions

·         List of movieID’s is returned; no changes are made to the system

 

 

Test case 2

Input  

equivalence class 2

Expected output:

·         Error indicating the genre does not exist

Preconditions

·         Input “genre” must not have a matching genre in the system

·         Database can handle invalid queries

Postconditions

·         Error is returned; no changes are made to the system

 

 

Test case 3

Input  

Border element of equivalence class 1 (First genre entered into the system)

Expected outpu

·         List of movies contained in the genre, if no movies in the genre a blank list will be returned; should not perform any differently than test case 1

Preconditions

·         At least three different genres are entered into the system

·         “genre” is the input from the user indicating which genre is desired

·         Database queries have all been tested and work

Postconditions

·         :List of movieID’s is returned; no changes are made to the system

 

 

Test case 4

Input  

Border element of equivalence class 1 (Last genre entered into the system)

 

Expected output:

·         :Expected output: List of movies contained in the genre, if no movies in the genre a blank list will be returned; should not perform any differently than test case 1

 

Preconditions

·         At least three different genres are entered into the system

·         “genre” is the input from the user indicating which genre is desired

·         Database queries have all been tested and work

Postconditions

·         List of movieID’s is returned; no changes are made to the system

 

 

 

Function: playMovie(movieID)

 

 

Equivalence class 1

 

Set of all movies in the system

 

Equivalence class 2

Set of all movies restricted from use in the system

 

Equivalence class 3

 

 

Movies not in the system

 

 

Border cases

 

Lowest movieID in class 1

Highest movieID in class 1

Lowest movieID in class 2

Highest movieID in class 2

MovieID in action genre

MovieID in comedy genre

MovieID in romance genre

MovieID in others genre

 

 

 

 

Test case 1

Input  

movieID from equivalence class 1

 

 

Expected output:

·         Expected Output: Movie will start to be played outputting video data

 

Preconditions

·         Movies exist in equivalence class 1.

·         Passenger has permission to play movie.

·         The IFESS system is enabled

 

Postconditions:

·         : movie will be played from the system

·         Passenger’s billing account will show the purchase if applicable.

 

 

 

Test case 2

Input  

movieID from equivalence class 2

Expected output:

·         error indicating that the movie does not exist

Preconditions

·         Movie exists in the system but is not able to be played

·         The IFESS system is enabled

Postconditions:

·         Error is returned; no change is made to the system

 

 

Test case 3

Input  

movieID from equivalence class 3

Expected output:

·         error indicating that the movie does not exist

Preconditions

·         Movie does not exist.

·         The IFESS system is enabled

Postconditions:

·         Error is returned; no change is made to the system

 

 

Test case 4

Input  

Lowest movieID in class 1

Expected output:

·         movie will start to be played outputting video data

Preconditions

·         : movies exist in equivalence class 1.

·         Passenger has permission to play movie.

·         The IFESS system is enabled

Postconditions:

·         movie will be played from the system

·         Passenger’s billing account will show the purchase if applicable

 

Test case 5

Input  

Highest movieID in class 1

Expected output:

·         movie will start to be played outputting video data

Preconditions

·         movies exist in equivalence class 1.

·         Passenger has permission to play movie.

·         The IFESS system is enabled

Postconditions:

·         movie will be played from the system

·         Passenger’s billing account will show the purchase if applicable

 

 

 

Test case 6

Input  

Lowest movieID from equivalence class 2

 

 

Expected output:

·         error indicating that the movie does not exist

 

Preconditions

·         Movie exists in the system but is not able to be played

·         The IFESS system is enabled

 

Postconditions:

·         Error is returned; no change is made to the system

 

 

 

Test case 7

Input  

Highest movieID from equivalence class 2

 

Expected output:

·         error indicating that the movie does not exist

 

Preconditions

·         Movie exists in the system but is not able to be played

·         The IFESS system is enabled

Postconditions:

·         Error is returned; no change is made to the system

 

 

Test case 8

Input  

movieID from equivalence class 1 with genre action

Expected output:

·         movie will start to be played outputting video data

Preconditions

·         movies exist in equivalence class 1.

·         Movies has genre action

·         Passenger has permission to play movie

·         The IFESS system is enabled

Postconditions:

·         movie will be played from the system

·         Passenger’s billing account will show the purchase if applicable

 

 

 

Test case 9

Input  

movieID from equivalence class 1 with genre comedy

Expected output:

·         movie will start to be played outputting video data

Preconditions

·         movies exist in equivalence class 1

·         Movie has genre comedy

·         Passenger has permission to play movie.

·         The IFESS system is enabled

Postconditions:

·         movie will be played from the system

·         Passenger’s billing account will show the purchase if applicable.

 

 

Test case 10

Input  

movieID from equivalence class 1 with genre romance

Expected output:

·         movie will start to be played outputting video data

Preconditions

·         movies exist in equivalence class 1.

·         Movie has genre romance

·         Passenger has permission to play movie.

·         The IFESS system is enabled

Postconditions:

·         movie will be played from the system

·         Passenger’s billing account will show the purchase if applicable

 

 

 

Test case 11

 

Input  

movieID from equivalence class 1 with genre action

Expected output:

·         movie will start to be played outputting video data

 

Preconditions

·         movies exist in equivalence class 1.

·         Movie has genre others

·         Passenger has permission to play movie.

·         The IFESS system is enabled

 

Postconditions:

·         movie will be played from the system

·         Passenger’s billing account will show the purchase if applicable.

 

 

 

Integration Test for Movie module:

 

The same tests done on the module will be performed but more focus will be on the post conditions to ensure the state of the system is exactly how it should be.  Also instead of having stubs for the billing system the actual billing system will be used and it must have been tested first.

 

Error reporting: As bugs are found bug reports will be generated in Bugzilla and sent to the developers

7.5.      Integration Testing

 

During the integration testing phase, a bottom up approach will be taken with the primary goal of fault isolation. This approach was chosen due to the fact that the dependencies between modules is relatively minor in the IFESS system. It was felt that the possibility of major design faults being left undetected until late in testing is very unlikely (this possibility represents the major flaw to bottom-up testing). Integration testing will be performed in parallel to the implementation and module testing in order to efficiently make use of time and catch problems as they are created.  In other words, as modules pass the implementation testing, they will begin integration testing.

 

The basic approach is to add the core modules to the IFESS framework one by one, and perform appropriate testing after each addition.  This includes “smoke testing” (i.e. reality checks), as well as black box integration testing.  The next stages of integration testing can only begin once the current stage has passed its tests.  Shown below is the dependency model for the major system modules.

 

 

Modules will be added to the IFESS framework in the following order:

 

1.      Information Update

2.      Billing

3.      Food

4.      Movies

5.      Music

6.      In-Flight Informtation System

7.      Internet

8.      Games

9.      Help

10.  Display System (GUI)

 

A more detailed explanation of the testing procedures for each stage of integration is outlined below. The details of the smoke testing will not be explicitly described for each module (the general procedure is to start the module or collection of modules and feed it trivial inputs in order to verify that it runs).

 

7.5.1.    Information Update

 

The information update system interfaces directly with the back end data store, and is therefore an appropriate choice of first module.  Doing this gives us a convenient method of adding and editing test data.  This module also interfaces with the display system that is to be added last.  After adding this module to the framework, the following tests will be performed upon integration:

 

1.      Smoke test of the information update module

2.      Black box testing of the module’s integration with the IFESS framework will be tested

 

7.5.2.    Billing

 

The billing module interfaces with the back end data store, as well as the food and movie modules that will be added later on. The following tests will be performed upon integration:

 

1.      Smoke test of billing module

2.      Smoke test of the current framework.

3.      Black box testing of the module’s integration with the IFESS framework

 

7.5.3.    Food

 

The food module interfaces with the back end data store, the display system (GUI), and the billing module that was previously added.  The following tests will be performed upon integration:

 

1.      Smoke test of food module

2.      Smoke test of the current framework

3.      Black box testing of:

·        The food module’s integration with the billing module.

·        The food module’s integration with the IFESS framework.

 

7.5.4.    Movies

 

The movie module interfaces with the back end data store, the display system (GUI) module, as well as the billing module that was previously added.  Upon integration, the following tests will be performed:

 

1.      Smoke test of movie module

2.      Smoke test of the current framework

3.      Black box testing of:

·        The movie module’s integration with the billing module.

·        The movie module’s integration with the billing, food modules

·        The movie module’s integration with the IFESS framework.

 

7.5.5.    Music

 

The music module interfaces with the back end data store and the display system (GUI) module. Upon integration, the following tests will be performed:

 

1.      Smoke-test of music module

2.      Smoke test of current framework

3.      Black box testing of the music module’s integration with the IFESS framework.

 

7.5.6.    In-Flight Information

 

The in-flight information module interfaces with the back end data store, and the display system (GUI) module.  Upon integration, the following tests will be performed:

 

1.      Smoke test of in-flight information module

2.      Smoke test of the current framework

3.      Black box testing of the in-flight information module’s integration with the IFESS framework.

 

NOTE: The In-Flight Information system is unique within IFESS, as it also integrates with an external system (the Fly-By-Wire in-flight information data feed).  Separate integration testing with this system will need to be performed.  However, as this system will not be available to Dangerous Minds until late in the development process, the integration testing will be performed outside of the main testing phase.

 

7.5.7.    Internet

 

The internet module interfaces with the back end data store, and the display system (GUI) module.  As it is relatively independent of other modules, the integration testing will be rather trivial.  Upon integration, the following tests will be performed:

 

1.      Smoke test of the internet module

2.      Smoke test of the current framework

3.      Black box testing of the internet module’s integration with the IFESS framework.

 

7.5.8.    Games

 

The game module interfaces with the back end data store, and the display system (GUI) module. As it is relatively independent of other systems, integration testing should be trivial.  Upon integration, the following tests will be performed:

 

1.      Smoke test of games module

2.      Smoke test of current framework

3.      Black box testing of the game module’s integration with the IFESS framework.

 

7.5.9.    Help

 

The help module has next to no internal logic itself, and does not interface with any modules besides the display system.  Integration testing for this module is expected to be trivial.  Upon integration, the following tests will be performed:

 

1.      Smoke test of help module

2.      Smoke test of current framework

3.      Black box testing of the help module’s integration with the IFESS framework.

 

7.5.10.     Display System (GUI)

 

The display system module is at the top level of the system.  Therefore, it is the last to be integrated into the IFESS framework.  It essentially interfaces with all major components of the system.  However, if major integration problems are found here, it should still not be difficult to isolate the problem, as the integration of all other modules have been tested.  Upon integration, the following tests will be performed:

 

1.      Smoke test of display system

2.      Smoke test of current framework

3.      Black box testing of the display system’s integration with the IFESS framework.

 

7.5.11.     Closing

 

Once all of these tests have been passed, the IFESS system will have been fully integration tested, and the next stage of testing may proceed.

7.6.      Product Testing

 

Product testing will begin after all of the modules have been integrated into the IFESS framework. The Test Execution team will perform most of the product testing with some help provided by the Inspection team. Product Testing will take place between $$ and $$.

This phase of testing will focus on three areas:

 

 

 

7.6.1    Functionality

 

Test cases need to be written for product testing that cover as much user functionality as possible. This is our last line of defense before the system goes out the door and we want to make sure that nothing is going to fail during the customer demo. The acceptance tests provided by the customer after reviewing our Conceptual Design document must be run as an essential component of this phase. Here is the structure and format to use when writing test cases.

 

 

Test Case Name

 

Pilot Successfully Reserving a Meal

 

Preconditions

 

The Pilot is logged into the IFESS system

 

 

Postconditions

- The meal order has been successfully placed in the IFESS system

- Confirmation has been given to the pilot indicating his order has been

   successful

- The meal order totals are updated in the database

 

Steps:

1.      Click on the Food Button

2.      Click on the Meal Tab

3.      Push the order button for one of the meals

4.      Confirm the order

 

7.6.2              Stress Testing

 

Load limits of the system need to be tested as much as possible. Simulations must be run of 500 users streaming movies and music from the server. The results of these tests need to be thoroughly documented.

 

7.6.3              Performance Testing

 

If time permits some benchmarking of the system will be run to gauge approximate run times of the system. The Test manager will inform you if any performance testing will be done.

7.7.      Acceptance Testing

The Acceptance testing phase will be a simple part of testing. The customer has provided us with some sample test cases and these will be demonstrated during the presentation of the system. The tests will have already been run during the product testing phase to ensure that the demonstrated functionality works as expected.

7.8.      Monitoring

 

We will be using the bug tracking software Bugzilla to log and track bugs. Both the inspection and execution teams will use Bugzilla to report bugs to the development team. Administration of Bugzilla will be handled by the test execution team.

7.9.      Regression Testing

 

Any bugs that are found must be reported to the development team. Once new builds have been received that include code fixes, bugs must be verified and any other functionality that was affected by the code must be retested.

 

The most important thing at the conclusion of our product testing is that the system is ready for the customer demonstration.

7.10.  Testing Teams

 

Inspection Team

Justin Paquin                             Inspection Lead

Tyler Tarabula                          Associate Code Inspector

 

Test Execution Team

Jon Oliver                                 Manager of Executions

James Anderson                       Associate Executioner

Pantanowitz Motshegwa           Associate Executioner