Applying Rule Extraction & Rule Refinement techniques to (Blackbox) Classifiers

Black-box classifiers are able to classify unseen instances, once they have been trained on an appropriate (domain) dataset. Such classifiers have the advantage of being generally very efficient but the disadvantage of not being able to explain their processes to a user. For these reasons, over the last decade or so, a number of rule extraction algorithms have been developed which are able to extract a rule-set from classifiers. The focus of this project has been to re-implement a state-of-the-art rule extraction system, OSRE [1], and then to show that when the extracted rules are refined by the Knowledge Refinement system, FIXIT, that the refinement process, in virtually all cases, improves the fidelity of the refined rule-set when compared with the rule-set extracted by OSRE. A statistically significant difference between these two approaches has been demonstrated. We investigated 4 classifiers (2 blackbox (Neural Networks & SVM), 1 Bayesian classifier & 1 (Decision-Tree-based) whitebox) and 4 domains, so a total of 16 Classifier-Dataset combinations were considered. In only 1 case (6.25%) was the result slightly worse; 5 cases (31.25%) were the same (these could not be improved), and the remaining 10 cases (62.5%) show significant improvements. In the future, we intend using similar approaches to improve the accuracy of the classification; this study focuses on fidelity.