[Dept. of Computer Science]

Experimentation Project ACS


Title Statistical comparisons of algorithms over multiple data sets
Student Durk Kingma
Supervisor Peter de Waal, Dirk Thierens
ECTS 7.5
Related Course(s) Advanced Data Mining, Probabilistic Reasoning, Learning from Data.
Description Analysis of Variance (ANOVA) is a well-established statistical method for testing whether data sampled from several groups differ in their mean values.
The conditions under which ANOVA is valid are well understood, but unfortunately ANOVA is often applied in situations where these conditions are not met.
The data may not be normally distributed, for instance, or the variances in the data of the different groups are not equal.
A typical example of incorrect use of ANOVA is seen in the comparison of the performances of algorithms over multiple data sets.
For these situations alternative testing methods are available, e.g. the Wilcoxon signed rank test or the Friedman test.

The objective of this experimentation project is to investigate how much the conditions for ANOVA can be relaxed before the Wilcoxon or Friedman tests become better.

For this experimentation project some tests need to be implemented. The choice of programming language is free.

Reference:

J. Demsar. Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research 7 (2006): 1-30.

Special Note