The Effect of Program and Model Structure on the Effectiveness of MC/DC Test Adequacy Coverage (original) (raw)

The effect of program and model structure on mc/dc test adequacy coverage

Proceedings of the 13th international conference on Software engineering - ICSE '08, 2008

In avionics and other critical systems domains, adequacy of test suites is currently measured using the MC/DC metric on source code (or on a model in model-based development). We believe that the rigor of the MC/DC metric is highly sensitive to the structure of the implementation and can therefore be misleading as a test adequacy criterion. We investigate this hypothesis by empirically studying the effect of program structure on MC/DC coverage.

Model-Based Test-Suite Minimization using Modified Condition/Decision Coverage (MC/DC)

International Journal of Software Engineering and Its Applications, 2015

Testing is very expensive for high-assurance software, like commercial aircraft systems, weapon research, weather forecast, earthquake forecast, and software used for safety critical system. A small and simple flaw in the end product can be enough for destroying the entire effort of the developer with a huge unrecoverable damage to the society. For this reason, Federal Aviation Administration's requirement is that, the testsuites should be comprises of Modified Condition/Decision Coverage (MC/DC) adequate. By using logic coverage criteria lots of flaws can be removed for safety critical software. MC/DC was proposed by NASA, and had been widely accepted in the field of testing. MC/DC is an effective verification technique, and helps to uncover safety faults. It is a challenge to minimize the number of test-suites when there is a partial change in the software. This can be achieved by using models. Unified Modeling Language (UML) not only helps to design software but also plays a vital role in detecting the faults early phase of design and in minimizing the test-suite. Existing test-suite minimization techniques investigated by different researchers may not be effective in minimizing MC/DC-adequate test-suites because they do not consider the complexity of the present software. A new approach for test-suite minimization is presented in this work, using dissimilarity matrix, which can be well fitted with MC/DC. We also present the results generated out of a case study of the test-suite minimization.

MC/DC Implications for Software Testing from (Com-binational) Logic Design

Structural testing is often the most common sought criteria for exercising aspects of control flow (i.e. such as statement, branch and path coverage). In many cases, criteria based on statement, decision and path coverage appears sufficiently effective for testing (in terms of selecting the appropriate test cases for testing consideration) the various parts of the software implementation. In other cases involving complex predicates, criteria based on statement, branch, and path coverage appear problematic owing to the problem of masking (where one variable is "masking" the effects of other variables). Addressing this issue, this paper discusses the strategy for structural testing based on Multiple Condition/Decision Coverage (MC/DC). In doing so, this paper also highlights the implication of MC/DC for (combinational) logic design.

Reinforced Condition/Decision Coverage (RC/DC): A New Criterion for Software Testing

ZB2002: Formal Specification and Development in Z and B, 2nd International Conference of B and Z Users, 2002

A new Reinforced Condition/Decision Coverage (RC/DC) criterion for software testing is proposed. This criterion provides further development of the well-known Modified Condition/Decision Coverage (MC/DC) criterion and is more suitable for testing of safety-critical software. Formal definitions in the Z notation for RC/DC, as well as MC/DC, are presented. Specific examples of using of these criteria are considered and some features are formally proved.

Guided test generation for coverage criteria

2010 IEEE International Conference on Software Maintenance, 2010

Test coverage criteria including boundary-value and logical coverage such as Modified Condition/Decision Coverage (MC/DC) have been increasingly used in safety-critical or mission-critical domains, complementing those more popularly used structural coverage criteria such as block or branch coverage. However, existing automated test-generation approaches often target at block or branch coverage for test generation and selection, and therefore do not support testing against boundary-value coverage or logical coverage. To address this issue, we propose a general approach that uses instrumentation to guide existing test-generation approaches to generate test inputs that achieve boundary-value and logical coverage for the program under test. Our preliminary evaluation shows that our approach effectively helps an approach based on Dynamic Symbolic Execution (DSE) to improve boundary-value and logical coverage of generated test inputs. The evaluation results show 30.5% maximum (23% average) increase in boundary-value coverage and 26% maximum (21.5% average) increase in logical coverage of the subject programs under test using our approach over without using our approach. In addition, our approach improves the fault-detection capability of generated test inputs by 12.5% maximum (11% average) compared to the test inputs generated without using our approach.

A formal analysis of MCDC and RCDC test criteria

Software Testing, Verification and Reliability, 2005

a mandatory requirement for the testing of avionics software as per the DO-178B standard. This paper presents a formal analysis for the three different forms of MCDC. In addition, a recently proposed test criterion, Reinforced Condition Decision Coverage (RCDC), has also been investigated in comparison with MCDC. In contrast with the earlier analysis approaches that have been based on empirical and probabilistic models, the principles of Boolean logic are used here to study the fault detection effectiveness of MCDC and RCDC criteria. Based on the properties of Boolean specifications, the analysis identifies the detection conditions for six kinds of faults. The results allow measurement of effort required in testing and the effectiveness of generated test sets satisfying the MCDC and RCDC criteria.

On the Danger of Coverage Directed Test Case Generation

Lecture Notes in Computer Science, 2012

In the avionics domain, the use of structural coverage criteria is legally required in determining test suite adequacy. With the success of automated test generation tools, it is tempting to use these criteria as the basis for test generation. To more firmly establish the effectiveness of such approaches, we have generated and evaluated test suites to satisfy two coverage criteria using counterexample-based test generation and a random generation approach, contrasted against purely random test suites of equal size. Our results yield two key conclusions. First, coverage criteria satisfaction alone is a poor indication of test suite effectiveness. Second, the use of structural coverage as a supplement-not a target-for test generation can have a positive impact. These observations points to the dangers inherent in the increase in test automation in critical systems and the need for more research in how coverage criteria, generation approach, and system structure jointly influence test effectiveness.

Systematic Model based Testing with Coverage Analysis

Aviation safety has come a long way in over one hundred years of implementation. In aeronautics, commonly, requirements are Simulink Models. Considering this, many conventional low level testing methods are adapted by various test engineers. This paper is to propose a method to undertake Low Level testing/ debugging in comparatively easier and faster way. As a first step, an attempt is made to simulate developed safety critical control blocks within a specified simulation time. For this, the blocks developed will be utilized to test in Simulink environment. What we propose here is Processor in loop test method using RTDX. The idea is to simulate model (requirement) in parallel with handwritten code (not a generated one) running on a specified target, subjected to same inputs (test cases). Comparing the results of model and target, fidelity can be assured. This paper suggests a development workflow starting with a model created in Simulink and proceeding through generating verified and profiled code for the processor.

Generating minimal fault detecting test suites for general Boolean specifications

Information and Software Technology, 2011

Context: Boolean expressions are a central aspect of specications and programs, but they also oer dangerously many ways to introduce faults. To counter this eect, various criteria to generate and evaluate tests have been proposed. These are traditionally based on the structure of the expressions, but are not directly related to the possible faults. Often, they also require expressions to be in particular formats such as disjunctive normal form (DNF), where a strict hierarchy of faults is available to prove fault detection capability. Objective: This paper describes a method that generates test cases directly from an expression's possible faults, guaranteeing that faults of any chosen class will be detected. In contrast to many previous criteria, this approach does not require the Boolean expressions to be in DNF, but allows expressions in any format, using any deliberate fault classes. Method: The presented approach is based on creating test objectives for individual faults, such that ecient, modern satisability solvers can be used to derive test cases that directly address the faults. Although the number of such test objectives can be high depending on the considered fault classes, a number of optimizations can be applied to reduce the test generation eort. Results: Evaluation on a set of commonly used benchmarks shows that despite guaranteeing fault coverage, the number of test cases can be reduced even further than that produced by other state of the art strategies. At the same time, the fault detection capability is not aected negatively, and clearly improves over state of the art criteria for general form Boolean expressions. Conclusion: The approach presented in this paper is shown to improve over the state of the art with respect to the types of expressions that can be handled, the fault classes that are guaranteed to be covered, and the sizes of test suites generated automatically. This has implications for several elds of software testing: A main application is specication based testing, but Boolean expressions also exist in normal source code and need to be tested there as well.

A method for analyzing system state-space coverage within a t-wise testing framework

2010 IEEE International Systems Conference, 2010

Inadequate state-space coverage of complex configurable systems during test phases is an area of concern for systems engineers. Determining the state-space coverage of a proposed or executed test suite traditionally involves qualitative assessment, rendering meaningful comparative analysis between tests for a given system or across multiple systems difficult. We propose a method for assessing state-space coverage of a test suite utilizing t-wise testing, a combinatorial technique borrowed from the software testing community which generalizes pair-wise testing. We refine traditional notions of a t-wise test suite to analyze the configuration coverage of a test plan. This provides a methodology and a set of metrics to assess both the level and the distribution of state-space coverage. We detail a proof-of-concept experiment using this partial t-wise coverage framework to analyze Integration and Test (I&T) data from three separate NASA spacecraft.