Clinical decision support systems (CDSS) often base on rules that are inferred from collected patients' histories, together with expert judgements and consented medical guidelines. This type of advisor system is known as rule-based reasoning system or expert system which classifies a given test instance into a particular outcome from the learned rules. The test instance carries multiple attributes which are usually the values of diagnostic tests. In this paper, we propose a classifier ensemble-based method for supporting disease diagnosis. The ensemble data mining learning methods are applied for rule generation, and a multi-criteria evaluation approach is used for selecting reliable rules over the results of the ensemble methods. The efficacy of the proposed methodology is illustrated via an example of a thyroid disease classification.