The VQR, Italy's second national research assessment: Methodological failures and ranking distortions

Abstract: The 2004-2010 VQR, completed in July 2013, was Italy's second national research assessment exercise. The VQR performance evaluation followed a pattern also seen in other nations, in being based on a selected subset of products. In this work we identify the exercise's methodological weaknesses and measure the distortions that result from them in the university performance rankings. First we create a scenario in which we assume the efficient selection of the products to be submitted by the universities and from this simulate a set of rankings applying the precise VQR rating criteria. Next we compare these "VQR rankings" with those that would derive from application of more appropriate bibliometrics. Finally we extend the comparison to university rankings based on the entire scientific production for the period, as indexed in the Web of Science.