Assessing Assessment : with a diversity spin

An ERE blog entry that PolyPlacements tweeted about reminded me of a bunch of discussions I have taken part in recently about assessment tools in the job board world.

I think you can make a strong case that fully-automated assessment tools are the holy grail of technology when it comes to recruiting and hiring. And yet nobody really seems to have the answer. Why is that?

One point of view, as is found in the aforementioned blog post, is that the tests themselves are to blame. Well. Okay. But I think the problem is more fundamental than that. The biggest hurdle I see is that in order to have effective tests you need a system that can objectively measure subjective answers but that's a difficult proposition from both a language and computer science perspective.

So to step back and explain what I mean. In order to have an automated (at least partially) assessment tool you need to be able to "score" the answers that the test user gives. The problem comes in designing questions (and a response mechanism) that be objectively measured. In the end most of these questions end being no better than "Do you have 5 years experience with X? Yes/No" which fails in multiple ways. The most obvious way it fails is a dishonest user who replies yes to every question. But the problem is worse than that as well because it doesn't properly capture the honest person who has 4 years and 6 months experience. Or the person who didn't really understand the question but has the relevent experience. Etc.

I am of the belief that at a certain point you have to accept that you won't be able to detect users with malicious intent purely through software. But that's okay because there are other aspects of the hiring process, like background and reference checking, that are intended to deal with these sorts of problems and frankly do it well. So I think one needs to focus on improving the process so that you aren't filtering out good candidates too early in the process.

These kinds of issues are especially relevant for TalentOyster because for some of our candidates the "traditional" sorts of screening tests act as obstacles to their getting hired. For example an assessment test that isn't PWD "friendly" and causes a good, qualified candidate to be filtered out. Or the assessment test that features such culturally specific terms and language that it can't be understood by anyone who didn't grow up in Canada.

And while we are working on assessments tools from a diversity angle I tend to think that the answers we find will be helpful in general as well. That is, to say the somewhat obvious, an assessment tool that works for a diverse audience will, by defintion, work for everyone.