Testing Elasticity with Declarative Data Mocking

This page contains the links to implementation code, automation and installation scripts, test reports and other material that complement the submission of the paper by Alessio Gambi and Schahram Dustdar titled:

Testing Elasticity with Declarative Data Mocking

Abstract
Elasticity is the ability of applications to adapt to time-varying workloads by dynamically acquiring and releasing resources. Elastic systems leverage elasticity to provide consistent quality of service while minimizing operating costs. Currently practitioners test elasticity by providing sequences of inputs to the system under test and checking that the system adapts as they expect. Generating sequences of data that mock the evolution of time-varying workloads is complex, and developers resort to hardcoded values and predefined execution orders to implement them. This results in inflexible test cases that are difficult to understand and have limited effectiveness.
We propose an approach that enables developers to improve their test cases for testing elasticity by automatically generate these sequences of data from high-level, declarative specifications. We analyzed the source code of a real-life implementation of elastic systems to extract the relevant properties of the data used in its test cases. This led us to define a conceptual model for formalizing such properties into executable specifications and implement a set of declarative data mocks that testers can invoked from their code to obtain test data.
We show the benefits of our approach on a real-life case study by recreating the original test cases with our declarative data mocks and by improving the quality of the available test suites.

The paper was submitted to the The 37th International Conference on Software Engineering ICSE’15.

Additional Material

Additional material includes the code that implements the various declarative data mocks presented in the paper, code that implement the test cases used to conduct the evaluation, various reports and instruction to setup the project (using Eclipse) and repeat the experiments.
For convenience we include the link to the Brooklyn project which is the real-life project that we used as case study for the study presented in the paper.

Declarative data mocks for testing elasticity

The code of the declarative data mocks, the setup instruction, and the scripts to repeat the experiments can be downloaded from Git Hub. The code requires Java7 and Maven, and it is ready to run.

Examples of test reports resulting from conducting the evaluation can be accessed via the links below:

The Brooklyn Project – Home

In our evaluation we considered the version 0.7.0-M1 (Milestone M1) of the Brooklyn project, as that was the most up-to-date version available at time of writing.

The code of the 0.7.0-M1 version can be downloaded from this page.

We ran the JavaNCSS tool to gather common complexity metrics for Java project. The summary report produced by JavaNCSS can be found here.
A JavaNCSS report focused on the AutoScalerPolicy is available here instead.

The full code of the Brooklyn can be downloaded from https://github.com/apache/incubator-brooklyn, but remember to switch to the 0.7.0-M1 branch for the repeating the evaluation !