Informatica 32 (2008) 97-98 97
Estimation of Individual Prediction Reliability Using Sensitivity Analysis of Regression Models
Zoran Bosnic
University of Ljubljana, Faculty of Computer and Information Science, Tržaška 25, Ljubljana, Slovenia E-mail: zoran.bosnic@fri.uni-lj.si http://lkm.fri.uni-lj.si/zoranb/dissertation.htm
Thesis Summary
Keywords: regression, predictions, correction of predictions, sensitivity analysis, prediction error, prediction accuracy Received: February 10, 2008
The paper is the extended abstract of dissertati on which is concerned wi th the estimati on of reliability for the individual predictions ofregression models (in contrast to estimating the accuracy of the whole model) and wi th the use of sensi ti vity analysis in that area. The dissertati on studies the ways of optimal reliabili ty estimate selection among 9 studied reliability estimates and evaluates the methodology on large number of standard benchmark domain as well as on real domains.
Povzetek: Članek povzema doktorsko disertacijo, ki se ukvarja z ocenjevanjem zanesljivosti posameznih regresijskih napovedi v strojnem učenju.
1 Introduction
The dissertation [1, 2, 3] discusses the reliability estimation of individual regression predictions in the field of supervised learning. In contrast with average measures for the evaluation of model accuracy (e.g. mean squared error), the reliability estimates for individual predictions can provide additional information which could be beneficial for evaluating the usefulness of the prediction and possible consequential actions.
Measuring the expected prediction error is very important in risk-sensitive areas where acting upon predictions may have financial or medical consequences (e.g. medical diagnosis, stock market, navigation, control applications). In such areas, appropriate local accuracy measures may provide additional necessary information about the prediction confidence. The described challenge is illustrated in Figure 1.
prediction \ prediction ) MSE prediction
prediction + reliability prediction + reliability l	prediction + reliability
examples
Figure 1: Reliability estimate for the whole regression model (above) in contrast to reliability estimates for individual predictions (below).
The methods for reliability estimation of individual prediction can be either bound to a particular model formalism [4] or be model-independent and therefore more general. The dissertation focuses on the objective to develop a new approach from the second group.
2 Reliability Estimates
The dissertation proposes 9 new individual prediction reliability estimates which are independent of the regression model. Three of newly proposed estimates are developed by adapting the sensitivity analysis [5] approach for the use in supervised learning. To apply the principles of the sensitivity analysis, we propose a framework for controlled modification of the input (learning set) and outputs (regression predictions) in supervised learning setting. By making minor modifications in the learning set we exploit the instabilities in predicted values and use them to compose reliability estimates. Six remaining estimates are either adapted from related work (three estimates are generalized for use with all 8 regression models), newly proposed using local error modeling approach (two estimates) or based on linearly combining two individual estimates among former (one estimate). The linear combination of estimates is performed for all combinations of eight individual estimates by equally weighing (averaging) estimates in the combination. The best combined estimate was selected for further evaluation.
82 Informatica 32 (2008) 79-84
M. Grobelnik et al.
3 Automatic Selection of the Best Estimate
We study the problem of the reliability estimate selection based on the given problem domain and the regression model. We discuss and define two possible solutions of this problem, based on meta-learning and internal cross-validation approach. In the context of meta-learning we propose a possible attribute description of the meta-learning problem and define it as a classification problem, where each class represents one of the 9 proposed reliability estimates. We also use the meta-classifier to explain which estimate is optimal for the given model and domain properties.
In the approach with internal cross-validation, we select such estimate for the use with the test examples which achieved the best results on the separate subset of learning examples. This approach was tested in standard cross-validation manner.
4 Results and Conclusion
The testing of reliability estimates was performed by correlating the estimates with the prediction error and by statistically evaluating the obtained correlations. For testing, 28 standard benchmark domains from publicly accessible repositories and 8 regression models were used (regression trees, linear regression, neural networks, bagging, support vector regression, locally weighted regression, random forests, generalized additive model). The testing results showed usefulness of the proposed reliability estimates especially for the use with regression trees, where one of the proposed estimates correlated with the prediction error in 86% of the testing domains. Both methods for automatic selection of reliability estimates outperformed individual estimates.
The individual estimates and both approaches for automatic selection of the optimal estimate were tested in a real domain from the area of medical prognostics. The results exhibited a significant number of correlations between the reliability estimates and the prediction error in the majority of tests. The statistical comparison of reliability estimates to prediction evaluations of the medical experts showed that our reliability estimates correlate to prediction error with statistically equal correlation as the manual evaluations of the experts do. This results therefore showed the potential of the proposed methodology in practice.
tion Science, http://lkm.fri.uni-lj.si/ zoranb/dissertation.htm.
[2]	Z. Bosnic and I. Kononenko (2007) Estimation of individual prediction reliability using the local sensitivity analysis, Applied intelligence (online edition), http://www.springerlink. com/content/e27p2584 387532g8/?p= d31c3f4 3a4 4 54 6ee9135 0 62 5fe68e4e1&pi= 2, pp. 1-17.
[3]	Z. Bosnic and I. Kononenko (2008) Estimation of re-gressor reliability, Journal of intelligent systems, volume 17(1/3), pp. 297-311.
[4]	C. Saunders, A. Gammerman and V. Vovk (1999) Transduction with Confidence and Credibility, Proceedings ofIJCAI2, pp. 722-726.
[5]	O. Bousquet and A. Elisseeff (2002) Stability and generalization, Journal of Machine Learning Research 2, pp. 499-526.
References
[1] Z. Bosnic (2007) Estimation of individual prediction reliability using sensitivity analysis of regression models (in Slovene), PhD Thesis, University of Ljubljana, Faculty of Computer and Informa-