Testing using architectural designs is intended to determine if the realised system meets its specification, and works as a whole. The growing complexity of embedded systems requires new techniques that are able to support testing of extra-functional requirements, like energy usage. Since testers might have limited resources to manually create test for all performance scenarios, researchers from Mälardalen University in Sweden proposed a method for model testing (1) (2) that automatically generates tests by selecting the potentially troublesome simulations of an embedded system modeled in EAST-ADL (3), an architectural language used in automotive domain.
Specifically, the method select tests based on random simulations using UPPAAL SMC (4) (the statistical extension of the UPPAAL model checker) and shows how to seed faults in the EAST-ADL model and select tests cases based on the energy-related fault detection capability using the MATS tool (5).
Architectural design models are created during software development by choosing components and connectors representing the whole software system and its high-level structure. EAST-ADL (3) is an architectural description language dedicated to the development of automotive embedded systems. The functionality of the system is defined at four levels of abstraction, as follows: (i) the Vehicle Level, the highest level of abstraction, describes the electronic features as they are perceived externally, (ii) the Analysis Level provides an abstract functional representation of the architecture, (iii) the Design Level provides a detailed functional representation of the architecture, together with the allocation of these elements onto the hardware platform, and (iv) the Implementation Level provides the implementation of the system using AUTOSAR elements.
UPPAAL SMC (4) is a statistical model checker used for the analysis of different performance properties of networks of priced timed automata (PTA) with stochastic semantics. Statistical model-checking generates stochastic simulations to estimate probabilities and probability distributions over time with given confidence levels, so the technique scales better than exact symbolic model-checking especially when used on industrial sized architectural design models. PTA are extensions of timed automata with cost variables that can evolve at integer rates and are used in this work to capture the resource usage. The resource usage is modeled via a function which assigns costs to both locations and edges.
The proposed framework (2) describes a mutation testing framework that uses energy consumption goals to automatically select test suites based on random system simulations. The framework is enabled by transforming the EAST-ADL model into a network of priced timed automata (4) and it is composed of several steps:
An Energy-aware Mutation Testing Framework for EAST-ADL Architectural Models
- Energy-Aware Mutant Generation. Ideally, such mutants should represent commonly-occurring faults, but since there is no previous work on identification of resource-related faults at the architectural level, researchers at Mälardalen University propose in this work a set of mutation operators applied on: (i) the EAST-ADL resource annotation (Energy Consumption Replacement Operator (ERO)), (ii) the timing behavior of an EAST-ADL component (Period Replacement Operator (PRO), Execution Time Replacement Operator (ERO)), and (iii) the structure of the functional architecture (Component Removal Operator (CRO), Component Insertion Operator (CIO), and Triggering pattern Replacement Operator (TRO)). These mutation operators are systematically applied to the entire EAST-ADL model, resulting in a set of energy-aware mutants, each simulating one syntactic model change.
- EAST-ADL to Priced Timed Automata Transformation. In order to use UPPAAL SMC for test case generation, we transform the EAST-ADL model (with energy consumption annotations) into a PTA model. Each EAST-ADL component is automatically transformed into a network of two PTA: an interface automaton and a behavior automaton. The triggering of each component, timing information, as well as the resource annotations, are included in the interface model. The energy consumption starts at the moment data is read from the input ports until the component writes the data to the output ports. This means that the energy consumed by each component increases with the execution time, modeled as a cost variable.
- Detection of Equivalent Mutants. Mutation testing introduces artificial faults or changes into the model (mutant generation) and checks whether a test suite is able to detect them. However, there are mutated models which keep the semantics unchanged and thus cannot be detected by any test. The problem of detecting equivalence between two models is an undecidable problem and can affect the mutation result. Given the fact that the energy consumption is of continuous nature, in this method they have to resort to a specialised type of SMT-solving suitable for hybrid systems.
- Test Suite Generation. Executable test cases are created using the MATS tool (5), by extracting the input parameters and the energy values at predefined time points from the simulation traces produced by UPPAAL SMC. Each test input is a vector of signals where the time-dependent behavior of the model is executed using an ordered sequence of signals. Each test case is executed on both the original model and its mutated counterpart. In order to minimize the final set of test cases we remove the test cases not contributing to the mutation score.
- Mutant Detection Criteria. A mutant is detected by a test suite if the energy signal diverges drastically at certain time points from the expected values (e.g, substantial energy deviations). We use a threshold to check if the distance between each value at each time point is larger than this threshold. If there is at least one energy value in the energy in the mutated model for which the distance is larger than the expected threshold then we consider the mutant detected.
The results of their studies (1,2) show the efficiency of the method by carrying out a pilot evaluation using a Brake-by-Wire industrial system. This system is modeled in EAST-ADL architectural language annotated with energy resources. The results show that the method is efficient in terms of generation time and shows evidence that statistical model checking is useful for testing the energy consumption of a realistic industrial system. Regarding the generation of simulations for energy analysis, UPPAAL SMC is able to find a solution in 7.2 seconds within 50 simulations of the model (as shown in the simulation below). In addition, the method is able to calculate the mean value provided by UPPAAL SMC for the maximum consumption is 447.2 energy units over 100 runs. Using this estimation, one can use feasibility analysis to analyze the probability for the energy consumption to stay within the available energy threshold provided by the model and select suitable test cases.
Generation of test cases for energy consumption showing nominal and worst-case tests.
The expectation is that this method can help a tester to select test suites showing divergent energy consumption and can be useful to increase fault detection while reducing the cost of testing. In addition, the results could allow an industrial designer to gain deeper understanding into the system’s resource-usage behavior, and consequently adjust and optimize both software and hardware designs accordingly.
References:
(1) Marinescu, Raluca, et al. “Automatic Test Generation for Energy Consumption of Embedded Systems Modeled in EAST-ADL.”International Conference onSoftware Testing, Verification and Validation Workshops (ICSTW). IEEE, 2017. Link
(2) Marinescu, Raluca, et al. “An Energy-aware Mutation Testing Framework for EAST-ADL Architectural Models.” 29th Nordic Workshop on Programming Theory NWPT’17, 01 Nov 2017, Turku, Finland, Finland. TUCS Lecture Notes, 2017. Link
(3) Debruyne, Vincent, Françoise Simonot-Lion, and Yvon Trinquet. “EAST-ADL—An architecture description language.” Architecture Description Languages. Springer, Boston, MA, 2005. 181-195. Link
(4) UPPAAL tool. http://www.uppaal.org/
(5) MATS tool. https://github.com/JLN93/MATS-Tool
Eduard is a postdoctoral researcher at Mälardalen University in Västerås, Sweden. He earned an Engineer’s degree from Polytechnic University of Bucharest in 2009 and a PhD from Mälardalen University in 2016. Currently he is doing research on model-based testing and test automation. Eduard ives in Västerås with his wife and two jolly dandie dinmont terrier dogs.