Charette: Key Challenges in Automated Software Testing

The program this year featured a multi-day charette on the key near-term challenges in software test automation. The charette had two purposes: stimulating near-term research and fostering group discussion and cooperation among participants.

Four key challenges and research pathways to achieve them were identified and discussed with the workshop participants:

• Test script generation, maintenance, and evolution (reconstructed from notes; slide presentation not available)
• Test model generation from requirements
• Automated analysis of test results (no slides presented, link is a photo of notes pages)
• “Impact measurement,” i.e., measuring the economic impact of test automation for internal marketing purposes

To provide guidance, the workshop organizers showed guide slides before each session. These guide slides can be found here.

Charette Process

The charette was divided into four sessions of one half-hour to one hour. During the first session, participants were called upon to identify what they thought the key challenges in software testing would be. We asked that the challenges meet the following criteria: (a) the solution domain should be software test automation; (b) the challenge hadn’t been solved before, or hadn’t been solved well enough; (c) the challenge should involve real research; (d) the challenge should be worthwhile to do, i.e., a good business case could be made for it; and (e) the challenge should be reasonably possible to do, i.e., a research prototype should be possible within one year and a business prototype should be possible within two years. Since this was a brainstorming activity, the key challenges were simply recorded on a flip chart without criticism at that time. A total of 12 key near-term challenges were identified by the participants.

Participants were then given three colored stickers each to use to vote on the challenges. Participants who identified themselves as being from academia were given one color, and self-identified industrial participants were given a different color. Only those challenges which had votes of both colors and for which a group leader volunteered went forward. The challenges that met these criteria were:

• Integration testing: 8 academic, 4 industrial
• Test script generation, maintenance, and evolution: 9 academic, 3 industrial
• Test model generation from requirements: 9 academic, 2 industrial
• Automated analysis of test results: 9 academic, 3 industrial
• “Impact measurement,” i.e., measuring the economic impact of test automation for internal marketing purposes: 7 academic, 2 industrial

• The topic of test case reduction and prioritization attracted 8 votes, but no industrial ones, so it was dropped. Other topics had fewer votes, and were dropped. We wanted to identify roughly four challenges, but we allowed five in the actual charette.

Participants then assigned themselves to one of these teams. (The integration testing challenge had insufficient volunteers, so it was dropped.)

In the second session, the teams each attempted to identify research pathways to address their key challenge, or sub-challenges within it. Following this, each team prepared a presentation on their research pathways in the third session, and presented their ideas to the workshop as a whole in the fourth session.