Experimentation Project

Experimentation projects in automated testing

Contact: Wishnu Prasetya

Automating Blackbox Debugging of Flash-based Applications

Imagine a Flash-based application that have been instrumented, so that user interactions are logged. The resulting log is thus essentially a sequence of entries, and each entry represents a user interaction/event (such as clicking on a button). When an error occurs, we can in theory replay the sequence of interactions to reproduce the error, which then allows a programmer to further inspect the execution. When an error is observed after an event e, it is not necessarily so that e itself is the cause of this error. It could be another event b that precedes a. Debugging here is defined as trying to identify the event that causes the error. A simple strategy to debug is to remove one event x from the log, e.g. we choose the last event, and then execute it. If the error does not disappear then x is the the cause. Then we repeat the procedure, selecting a different event y to try. If after removing x the error does disappear, then x could be the cause. But it is also possible, that removing x causes the application to take a different execution path; one that avoids the actual cause of the error. Since we don't know this upfront, we will have to manually inspect x.

Since the log can be very long, the above approach can be very inefficient. To improve this, you will get set of equations, e.g. ab = b, or ab=ba, saying that from a certain abstraction the sequence ab will behave as b, or will behave as ba. For example, imagine the sequence:

cabababd-->x

The event d is called the witness event. It is the event that exposes the error, denoted by x, so that you notice it. Note that d is not necessarily the cause of the error. If you were given the equations ab=b and bb=b, the above sequence can be reduced to

cb

which would then be easier to debug. Another example:

bccccccd---> x

If we know cd = dc, it can be rewritten to:

bdccccc

But since d was the witness, it is likely that when we execute the latter sequence, the error will again surface after d (and not after a c). This implies that it is then only necessarily to debug

bd

Notice that what we essentially do is that we remove events of which we know that they are more likely not to contribute towards the error. Events that we cannot remove have to debugged in the traditional way as described before. The new scheme may seem to be more efficient, but there is also a challenge there. The equations were determined based on some abstraction. So, applying them to reduce a log may produce a new log that is actually not executable. So, only reductions that produce executable logs should be applied. But to check whether a reduced log is executable, we have to execute it and see if it does not break in the middle. This costs time. So, some strategy would be needed here.

Your task is to implement this debugging scheme in JavaScript? , which we will use to control a Flash-based application from a web browser. You need to produce a modular implementation that can be easily extended later. You need to implement at least one strategy as meant above, and then conduct some benchmarking on the strategy.

Inferring, Injecting, and Testing Specifications

Currently we have tools to infer pre and post-conditions of methods from the logs they generate during the testing phase. The strength is limited (this holds for any approach of dynamic inference), which means that the generated specifications are only partial. Nevertheless, we find a way to strengthen it by parameterizing the specifications with scenarios. Unfortunately we do not yet have the facility to measure how strong the resulting specifications are.

This can be solved by building a tool to inject the inferred specifications back into the methods, and providing a mutation test capability to measure their strength. Special attentions may have to be given on how to encode scenarios parameters, and how to inject mutations that do not break them. Additionally, mutation test is notoriously expensive. Creative ideas to cut down the computation cost will be greatly appreciated.

Logging Exceptional Behavior

Currently we have a quite sophisticated infrastructure to log ActionScript? programs. However, we are still unable to log exceptional behavior. We really would like to improve this because in practice it is such behavior that usually leads to failure, and thus worth logging. A target method may show exceptional behavior if it throws an exception. It then enters a handler, if one is provided, or the method exits. This is currently not logged. Worse, in the second case the forced exit will case an important logging invariant to be broken, namely if the method entrance is logged, then there should be a corresponding exit log entry. We need you to extend the existing framework.