Evaluating university research: same performance indicator, different rankings

Abstract: Assessing the research performance of multi-disciplinary institutions, where scientists belong to many fields, requires that the evaluators plan how to aggregate the performance measures of the various fields. Two methods of aggregation are possible. These are based on: a) the performance of the individual scientists or b) the performance of the scientific fields present in the institution. The appropriate choice depends on the evaluation context and the objectives for the particular measure. The two methods bring about differences in both the performance scores and rankings. We quantify these differences through observation of the 2008-2012 scientific production of the entire research staff employed in the hard sciences in Italian universities (over 35,000 professors). Evaluators preparing an exercise must comprehend the differences illustrated, in order to correctly select the methodologies that will achieve the evaluation objectives.