SLE WS 2013 Assignment 3
Reading about grammar-based testing.
(Grammars are meant here in the broad sense to include schemas, metamodels, etc.)
Description
Read the paper Controllable combinatorial coverage in grammar-based testing (the "CCC" paper from hereon).
You should also look through some of the papers cited in the CCC paper (assuming that you may find some of them interesting).
Deliverables
You are not required to prepare any presentation, but please be prepared for a discussion of the CCC paper and grammar-based testing and how it could be applied to FSML and how a corresponding component could be added to your development for assignment 2. (A grammar-based testing extension is indeed the subject of assignment 4.) More specifically, let us discuss the following questions:
- What sort of test-data should be generated? (Concrete syntax, abstract syntax, input for the state machine, …)
- What sort of testing objective may we assume?
- How could we deal with the well-formedness properties of FSML during test-data generation?
- Should we be generating negative test cases? What kind?
- What sort of test-data algorithms may be applicable?
- Are we aware of an applicable test-data generation frameworks for some of the platforms used for assignment 2?
- Would it be feasible to implement a test-data generation for some of the platforms used for assignment 2?
Further reading
- Some of the students in the course may be interested in metamodel-based testing, specifically testing of model transformation. There exist several methods and tools out there. For instance, you may look at the work by Antonio Vallecillo.
- Those interested in an existing grammar-based testing tool may consider Peter Maurer's DGL, even though it may be difficult ("I really don't know.") to get the tool running. In any case, there is accessible documentation on the DGL paper, which perhaps inspires a tool effort for assignment 4.
- Property-based testing is relatively popular these days; see e.g., Haskell's QuickCheck. This sort of framework can also be used, to a limited degree, for grammar-based testing, as one can design, for example, data generators for the types of an abstract syntax. QuickCheck-like tools are available for a number of languages.
- Test-data generation frameworks are available for a number of programming languages and platforms; see, for example, Randoop for Java. You are well advised to analyze the situation for your platform of choice (assignment 2). Quite likely, different kinds of tools are available, in which case you should understand and explain their distinguishing characteristics. It may happen that you cannot identify an existing framework for your platform, in which case you may be planning for a basic implementation of a framework.
General logistics
See the course page for deadlines and conditions on successfully completing the assignments.