New Paper Accepted to Automated Software Engineering Journal

I am very glad to learn today that our paper is accepted to Automated Software Engineering Journal. This paper proposes a nice ordering scheme for various machine learning algorithms that are used in software effort estimation. Furthermore, it also suggests a good sanity check section, in which we propose an evaluation scheme that makes use of two statistical tests to cluster algorithms, whose performances are not statistically significantly different. I am very lucky to work with my supervisor Dr. Tim Menzies and our brilliant collaborator Dr. Jacky Keung in this project. Below is the abstract of this paper and here is the link to the latest version (but not yet the camera ready version).
Abstract
Background: Conclusion instability in software effort estimation (SEE) refers to the inconsistent results produced by a diversity of predictors using different datasets. This is largely due to the “ranking instability” problem, which is highly related to the evaluation criteria and the subset of the data being used.
Aim: To determine stable rankings of different predictors. 
Method: 90 predictors are used with 20 datasets and evaluated using 7 performance measures, whose results are subject to Wilcoxon rank test (95%). These results are called the “aggregate results”. The aggregate results are challenged by a sanity check, which focuses on a single error measure (MRE) and uses a newly developed evaluation algorithm called CLUSTER. These results are called the “specific results.” 

Results: Aggregate results show that: (1) It is now possible to draw stable conclusions about the relative performance of SEE predictors; (2) Regression trees or analogy-based methods are the best performers. The aggregate results are also confirmed by the specific results of the sanity check. 

Conclusion: This study offers means to addresses the conclusion instability issue in SEE, which is an important finding for empirical software engineering.