Monday, April 25, 2005

Metrics - just because something is fashionable doesn't mean it's right

There seems to be a lot of talk about metrics at the moment, mainly about how poor the current indicators are (cost per hire, time to hire) and how we should all replace them with such things as Quality of Hire and even a sort of Return on Investment of recruitment.

I suspect that I've been looking at these sort of things since my first time as a recruitment manager - some 8 or so years ago. I've debated them with others and spent far too much time thinking about them. For some years I decided that I was probably just being a bit thick but as I aged I have decided that maybe I could have been right after all. (Though of course stubbornness probably increases with age) The reason I couldn't work out how to calculate such stats is that they probably can't be calculated.

OK, some caveats. First I am of the 'no stats is better than bad stats' school of thought, especially when credibility is at stake. Second, spending my university life studying things such as econometrics gave me a love of models, but also an understanding of what makes a good one - that is, I'm a bit critical of these things but also love them.

Quality of Hire - why it is only appropriate in a few, limited situations

The first problem here is how to define quality. My Swiss friends tell me that one way of indicating quality is "does it have a 'made in Switzerland' label on it"? (The Swiss being keen of totally over-engineering things - just see what happens when they make a camera) Of course performance of people is far more complex.

You could use things such as performance appraisals, targets etc. but the problem here is that they usually don't accurately describe the contribution the person makes to the firm. The 'closer' you are to the client the easier (maybe) it is to measure contribution but add management into the mix and it all gets very hard.

The other area where you might be able to accurately measure contribution is if the person is completing a heavily process-defined role. If they are making widgets you can count the number of widgets. However, anything to do with knowledge working is very hard to identify and measure the contribution because much of the value is derived not from the individual, but their relationships with others.

So if you can't define and measure quality how do you measure quality of hire? Of course you can use the performance measures that you have but be aware that they're inaccurate and that by setting them as goals for selection you might accurately select the 'wrong' thing.

Let's presume you can define performance, then the function will probably come out as something like this:

P=fn(Tt, Ti, A, E) where

Tt= trainable talent (knowledge, skills) - you can test this
Ti=inherent talent - you can get a good view of this
A=Attitude - you can get a good view of this, though it changes over time
E=Environment - this is out of the control of the recruiter

Let's presume we can develop a great selection model, with measurement around a bunch of relevant competencies. You use it with a group of people that is large enough to be statistically relevant, you measure the performance after a period of time and then you use your data to constantly validate the contribution made by each competency which you accurately measured. What would be the problem here?

Well (a) few people recruit so many of the same type that you can build a good model and (b) it all presumes that the environment remains constant, and this is rarely the case. Things that could change this are management changes, process changes, technology changes, external economic changes... the list is long. If any of this changes, was the performance a factor of the environmental change or your accurate selection?

To measure quality of hire there is an inherent time lag - that is, performance is shown over time as the person learns about their new environment. The individual may deliver the most performance in later roles. They may be average in their first role but great in the second. This would be the real performance of the person, but how do you measure that?

How accurate can selection be? Well, I reckon that you can get about a 70% accuracy on selection for anything other than the most 'process' role. You will get a normal distribution from those people you do select and I guess you want this distribution to be higher than if you had no selection (which of course you are unlikely to be able to test) and that the average is higher than your existing population (which will change when the new people join)

Finally, was that performance driven by factors you selected on or the value added by development, management etc. Try convincing your managers that an individual's performance is purely a result of your selection.

Return on Investment - innacurate and worthless

Again let's consider the average knowledge worker or manager. To measure ROI you need to understand the return - the economic contribution the person makes to the firm, and the investment. The investment is possible to measure, but not as easily as you think. The economic contribution is extremely difficult.

First, value is often based on the relationships between individuals and not the individuals themselves. You would need to understand the networks accurately and measure the additional value of all those relationships. A great manager may not actually 'make' things but by coaching might release value in others. To measure the economic value you would need to identify this value.

Secondly, these models are dynamic, not static. If a department loses someone they don't not do that persons work, they reallocate it. This effects intangibles such as departmental morale which in turn impacts performance. You need to model and understand all of this.

You could try saying 'without this person we would be $x worse off' but remember facilities could probably say 'without desks we would be $y worse off'. Both would be silly.

Can you estimate all of this to a degree to provide worth? Yes you probably can, especially in a non-complicated environment. Go and hire yourself a team of economists who can do time-series analysis on dynamic models. And prepare that the model will change as the environment changes. Understand it's a 'best guess'

My take on one of the reasons that HR has a low credibility in most organisations is that it tries to 'copy' other parts of the organisation only to make a mess of it. Most senior management with any comprehension of numbers will soon see the weakness of either ROI of recruitment or Quality of Hire. Go in with a dollar value of the economic contribution that an individual makes and most will probably smile sweetly and then smirk behind your back. Either that, or want to understand how you modeled accurately a whole organisation. Remember if you had an accurate model they could then manage it perfectly.

Final word - stop worrying about things you can't measure, stop feeling you have to 'prove' your contribution and start concentrating on making your managers appreciate your contribution. What ultimately matters is their perception of what value you add.