This post could only conceivably be of interest to a professional scientist of one sort or another...
Few subjects have probably been blogged on more by UK scientists than the Research Excellence Framework... otherwise known as REF ...otherwise known as "Son of RAE".
Anyway, this post is about the issue of "metrics" vs "impact". Making some attempt to quantify research excellence. Now currently, metrics are out. Journal Impact Factors are lambasted and Citations (i.e., H-indexes) are out the window. The accepted feeling is that someone can just read through your paper and make an objective judgement on it. This person will almost certainly not be an expert in your field and if they are they will either be your buddy or your rival!! Mmmm.
Now if it were me I would just stick to the metrics. It is flawed of course, but if you are going to be judging people, at least a metric system allows people to see where they are at openly.
My feeling is that the reason metrics are being abandoned is because it is too difficult to control, i.e., it doesn't necessarily give the right answer. So what is the "right" answer. Well someone once Tweeted: "The secret of any assessment exercise is to find criterion which put ones self at the top"... oh I think that was me... Anyway. I think it is true. You are a god of research, massive reputation, influence, sit on all the best University Committees and the BBSRC... BUT, once you have split your "output" between a couple of junior academics... and excluded your own funding... your CV may not look any better than a junior academic any more. Perhaps someone has gotten to their 4 reasonable papers before you and you have 50 papers, but none are really any more than reasonable. So bingo: "IMPACT"... now you can look at yours and theirs and say, theirs was in the same journal and has got the same citations, but mine has "impact" and there's doesn't. Not only that, but "I" will be at the meeting where that totally arbritary judgement is made and the junior academic won't be. QED. "Impact" it is. I can sit around with my new friends and we can all decide why "our" papers are better than "their" papers.
Now lets give a numerical example. I know nothing about football. But there are two teams I want to compare which is best. Aston Villa and Accrington Stanley. OK here is the metric approach. Aston Villa are in the premiership (publish in high impact factor journals and have a have a lot of citations)... Accrington Stanley are in league division 2 (as or writing, which is a kind of lower impact factor journal and a fewer cites type of division). I conclude Aston Villa are a better "team". Now without metrics. I have watch AS play and they seem to play with high intensity. When they kick the ball they make a lot of noise and they all seem to pass well and run about a lot. Aston Villa, well their kits seem to stay cleaner and they have a pretty neat organisation at the back. They don't seem to run about so fast or kick the ball so hard. However, since I live down the road from AS's home ground and all my friends follow them I amd going to give them the nod.....
