Damn Lies and Statistics

In the light of the recent NORAS report and predictable “We’re Number One!” press releases, I thought it would interesting to put down a few thoughts about research and stats in our market.

First of all, although I’ve never been a fan of NORAS, Alistair and Tim deserve a lot of credit for their tenacity in keeping it running regularly for this long. It’s also always interesting to see some new top line stats on online recruitment. However it is absolute not the utopian credible measurement and planning tool the sites who take part seem to be determined to persuade me it is

The longest running debate in online recruitment has always been about consistent independent measurement of job sites and how any resulting stats might be used to select the “correct” job board. Conferences have been run, meetings have been held, lunches have been expensed, and drinks have been consumed. What should we measure? How do we compare apples with apples? Who should pay for it?

The more I think about and the more I talk to my clients, the more I realise that the whole thing is a waste of time and brain power………in short a big red herring. Let me explain why

Once upon a time there was thing called “Old Media” and it consisted of things made of paper and analogue signals flying through the air into other things called radios and TVs. There were also people called advertisers who were interested in whether they were getting any kind of return on investment. No one really knew because it couldn’t be measured and that really wasn’t very good. So in order to keep everyone happy lots of lovely statistics started to appear and things like ABC and NRS and lots of other collections of letters were born

One day though something called Digital Media arrived. It could be measured, it could tell its advertisers exactly what they were getting for their money and go into such minute detail they could change the way they advertised on an hour by hour basis and thus really drive their return on investment. Unfortunately though a lot of people missed the point and argued for years about how generalisations and umbrella statistics should be applied to this new arrival.

Hopefully you can see the point I’m making.

This is usually the moment that someone says to me something like “Well that’s all very well Matt but lots of clients still aren’t using online recruitment. We need things like NORAS to get them started and show all the beginners how to choose which job boards to use”

At this point let’s look to the world of digital advertising in the product arena. A parallel debate is raging there about introducing some kind of planning currency that is similar to TV advertising. Again a waste of time but the only reason it is being pursued is to convince massive brand advertisers like P & G to move money from TV to online. This kind of reasoning does not hold up in the online recruitment space though.

The biggest recruitment advertising spenders in the market all already mostly all over the job boards. Granted they might not be doing things in the right way or spending as much as they should but that is to do with strategic understanding and isn’t something umbrella stats can address. The big spending laggards in our market are elements of the public sector and believe me when I say that this is not an issue that statistics and surveys are going to solve! If it was we would have reached the tipping point already.

I also don’t think auditing job board applications is the answer either. Firstly with so many different technology platforms in place I would be amazed if everyone could ever agree on a standard definition of an application. Secondly the only metric worth knowing is applications driven by job boards through corporate web sites and none of the clients savvy enough to measure this are going to, or indeed should, give away their competitive advantage by publishing it!!

Finally it would be a bit one sided to rant on about things that are, after all, attempts to grow our market, without offering an alternative.

Rather than focusing on numbers of things and generalisations why don’t we focus on users, experiences and behaviour? I don’t want to know so much about what jobseekers do and how many of them do it, I want to know about their thinking, motivation, influences and reasons why. Its only by doing this that I believe we can make our market fulfil its significant future potential

Comments

[Russell]

Interesting article Matt. Having come from a press background I
remember only too well the times that we would get overlooked for
ads/campaigns despite overwhelming 'statistical' evidence that we were
the most suitable medium? Why? A potent combination of tradition and the
fear of trying anything new; that the advertiser should perservere with
it's usual media choice because it had worked for them in the past.
Better to hedge your bets with something you know might work, rather
than something that might-not.

Online, we have more stats and audience data than we could have ever
hoped possible in the press, but I fear that as online recruitment
becomes more established the same behavioural patterns will be evidenced
by advertisers. A small site could have the best set of audited stats
out there, but could still get overlooked in favour of one of the...
"behemoth" sites because of advertiser laziness.

So a rhetorical question I guess, but how can small sites make a big
noise? If the stats start to become superflous will we not just see a
consolidation of the bigger sites?

[Richard Purvis]

Russell is right, as I'm trying to launch sites right now I know the pain of apathy in this market. As HR have not, and rightly so, taken on the mantle of recruitment experts because of OLR exists. I'd pass the buck of responsibility to the ad agencies to independently gather response data and publish this info, hold sites like monster and totaljobs up to account and push specialist verticals to really prove they're specialist
There should be no more debate. let me use an dodgy Purvis analogy. Ready?
Think of us sites as taking an exam to prove our worth, you wouldn't ask us to mark our own papers and if you did why are you surprised when a "little" fudging takes place, if you are marking our papers then...man this is going nowhere. Point is there is no-one else who can do it clients won't because they pay, rec cons won't give away info to competition and sites lie.

All very well but it still leave me with the problem how do I launch new verticals without stats or clients willing to move out of their comfort zone.

Rich

That last bit is more of a cry for help than anything!

[John Whitehurst]

I have found over the last year that the wider we push a media buy - the better the results.

It is very easy and lazy to buy across the main digital media propositions, but not always the most effective.

It is through the use of stats; looking at audience size and then linking it to response data that we have been able to move budget to these smaller propositions.

The big four sites are all good - but putting all your eggs in a few baskets will not work in the long term.

Give the long tail as good read - we have seen it happening and improving effectiveness already.

But it is through good effective planning we have seen the results, independent stats are great for this as most media owners still don't understand their log files (this is a good point about ABCe and NORAS as they help to give a level of understanding to a poorly educated market).

But i have a feeling that a number of people are running down the easy route already. The only way to stop this is "Keep on pushing the market".

[Matt]

It must be a nightmare being a start up in this market you have my sympathies! In an ideal world a start up would have a cunning creative sales / PR / B2B marketing strategy to get noticed and or a sales team with a lot of contacts and existing relationships. So far so much the same with any new / small business. Once noticed they would find their way onto some "long tail" schedules and thus prove their effectiveness (don't forget there are still loads of crap sites out there) and live happily ever after. That's how things should work and that's how the progressive clients / agencies do work. Unfortunately that's not the norm. Instead there seems to be a barrier of habit, ignorance and misunderstanding of the medium. I still get the impression sometimes that a lot of people are just trying to work out which one web site is the equivalent of the newspaper they always use. Education needs to happen but I'm not sure dumbing things down into surveys and generalisations is the best way to do it

[Andrew Gordon]

We take part in NORAS and have done since it launched. We find that it helps us position ourselves in the minds of the customer.

However, surely the only metric that really matters is which was the media - on or offline - you sourced the appointed person(s) from?

For a while we've been asking our clients to share this information with us.

You can see the latest results here:

www.jobs.ac.uk/recruiters/marketresearch/

We would be very happy to do this with ALL of our clients. But not all of them monitor their recruitment in this way.

No surprises there.

But what WOULD be really surprising is if all jobs boards conducted this sort of exercise too.

[Alex]

OOOOO – a proper debate and everything. Get in!

You make a fair case Mr Alder. But your appreciation surely comes from having: 1. an excellent base knowledge of the medium & 2. an assumption of a certain level of application management technological “sophistication”. Most agencies and a truckload of recruiters / companies, as you admit many in the public sector, have a long long way to go before they’d have the underlying systems and appreciation to realise the kind of intelligent media planning of which you speak.

I do believe you (and your colleagues – previous and current) have seen the future Matt – and the worry for other agencies is now how to make up the ground that is opening up before them in an environment where many still have no idea about how they might secure margin in this brave new world (other than through a few web build projects).

I don’t think there’s any real secret amongst most of us reading (certainly not those contributing to) this blog that the level of sophistication in digital media evaluation and planning that a small band have pioneered has significantly outstripped the rest of the market as it stands. Now such a competitive advantage doesn’t last for ever (economics dictates), and, if I’m right, a significant percentage of companies are not really that close to being in a position to implement and take advantage of what is still “relatively” early adoption methodologies. For many it’s still quite scary – for others it’s nothing more than confusing buzzwords. The market, absolutely 100%, will move towards the kind of tracking and analysis of which you speak – but there are many many many who are still quite a way off and may continue to drag their feet.

So in the mean time – how does NORAS help? Well I’m still not sure that it does anymore than anything else you can access? For me it’s a bit of a disparate collection of job boards that doesn’t provide a true reflection of any one market sector. ABCe verification of user traffic - that’s not NORAS, that’s ABCe. User analysis through surveys – surely just a snap shot in time, where as the CV or jobs-by-email database info gives me just the same if not a more expansive and robust analysis of who’s actually using the site and who I can gain access to?

I’ve said it before and I’ll say it again – offer a facility that analyses across all job board issued data, irrespective of them paying for inclusion. Payment for inclusion is no way to run an independent piece of research (certainly not if it’s worth its salt and in its sixth year of production) – but if it’s integrity and usefulness is raised then I believe the market (until everyone catches up with Matt & Co. of course) might just be willing to pay to review and use the results. This approach, and main stream use of the facility, I believe would then help “encourage” the participating sites to submit their databases to ABCe scrutiny to give real credibility to their user statements (preventing the “fudging” Richard refers to and we’re all aware happens).

Russ & Rich - I feel your pain. When you hold up the cost of offline activity against that of online targeting then I still see no good reason why companies aren’t unlocking more budget for trialling the niche channels (the much vaunted long tail recruitment) – but if they’ve not got the facilities in place to evaluate real success, or indeed the ability / understanding to interpret that data, then it’s still a bit of a frustrating hard sales slog. God knows we’re still banging out considered schedules / proposals where the end result is just a single job posting on a single site.

NORAS, in its current format, does nothing as far as I can see to really further online recruitment usage, and probably not even that much for those sites who decide to invest a significant chunk of cash in what is effectively little more than a targeted PR exercise. What I’m sure you don’t necessarily want to hear as a salesman though is that you need to keep the faith. The world is moving online, it’s just taking a little longer than anyone really anticipated. In the mean time you have the challenge of maintaining your candidate base – I know, difficult without the advertisers – but maintain that as best you can, keep the faith and then let the real tracked delivery stats speak for themselves (hopefully they’ll do so in the right way for you).

[Ben]

Surely, the responsibility of addressing the confusion/misunderstanding and lack of appreciation of value doesn’t just lie with the Ad Agencies. For a long time now, several of the more learned representatives of the industry have been working to explain to our clients that there is more to a ‘good job board’ than simply large numbers.

However, no matter how hard we try, we come up against six-monthly claims of ‘we’re the biggest’, ‘we’ve got the most job postings’, ‘we spend the most money advertising on TV so we must be the best’.

I hear what you’re saying Richard, and to be honest, just as with traditional media, I really don’t think there is a ‘quick fix’ answer to developing a good vertical market. The successful verticals are generally those that offer value and interest to their key target audience. They build up a loyalty over time. It’s tough. It really is. You know that. And they work closely with third parties to continue to press their message home – using these third parties to help get a foot in the door.

It’s tough for the agency too – as we can see a good quality product and do promote it – you know that I, and certainly a lot of my colleagues here, have worked with you for a number of years now. You present a well thought out, well delivered justification for your products. You always have and I guess always will. But, you have to understand that there are so many people out there who don’t – so many it’s scary (including still some of the larger publishers whose representatives struggle to fully understand what it is that matters in the online market) – and yet still expect us to provide them with business too as they ‘are really good’. Unfortunately for products that are well conceived, this makes your lives much more difficult.

For a long time now we hear of businesses launching online that really have put little thought into their product, don’t truly understand the market, and have almost launched something ‘because they can’. Understandably, the market is therefore quite cautious – and is not necessarily keen to spend its money on new products until they are proven. And that’s where the problem comes – how do we prove success?

It’s certainly not just a numbers game – and it very much can be influenced by ad agencies helping their clients to monitor their activity more closely.

As Alex mentions, part of the issue is the size of campaign being run. Contrary to popular perception, a lot of clients are still only really ‘dipping their toes’ into the wonderful world of web. Tracking response properly often requires engaging with third parties who can at times be obstructive, difficult and generally make the process a lot more difficult than if it were not being actioned. As such, I guess a lot of organisations just want the easy life.

This is hugely frustrating, but unfortunately is a fact of life, and also still a reflection of the media itself – still seen in a lot of place as cheap and cheerful (frustratingly). However much we understand and appreciate the medium, this is unfortunately still the case in a lot of businesses. The majority of recruiters are still not fully appreciating the benefits of proper tracking. And then, when they are repeatedly contacted by web sites claiming they are the biggest and the best because they can quote ‘large numbers’, the client is going to become confused.

When NORAS was first proposed, the ad agencies were very interested. We attended a pre-survey meeting to discuss what sort of information we would like to see (including a number of relatively senior non-online focussed professionals) and spent a good couple of hours providing our input and requests, along with well-thought out reason behind those requests. And what happened? Well, the type of information we wanted was ‘over-ruled’ by the job boards who were ‘paying’ – because it perhaps would not have positioned them as well in the results.

As with Matt, I would like to take my hat off (I don’t have a hat, but I thought a metaphor would help here) to Alastair and Tim at Enhance for trying. They persist in pushing NORAS through, but the difficulty is that they need to run a business, and to do that, they need to generate revenue.

When Tim used to carry out the research for The Guardian – Workthing – I remember that the findings were much more interesting and informative – albeit at a relatively base-level – because they weren’t influenced directly by trying to position one job board against another. Instead, they were set up to provide an overview of market trends, user requirements and perhaps user habit (I think – it was a long time ago, but I do vaguely remember this sort of information).

Statistics on number of page impressions, unique users etc. are really basic ‘numbers’ but tell us very little about the job boards themselves. They merely supply a metric with which the poorly informed layperson can make an assumed comparison.

I’m with the guys above in saying that the focus of the industry needs to shift away from trying to compare ‘the number of this’ or ‘the number of that’ and instead move across to researching user-habits, candidate and client needs and trying to find reasons why a candidate decides to use X or Y for their next job. Depressingly though, I think the results will be pretty inconclusive…

Phew – I need a coffee.

[Sinead]

'Job-hunters favour Internet recruitment, survey reveals'
'Recruitment Advertising is Moving Online'

I've read these two headlines in email newsletters/websites lately and have honestly felt that a similar more insightful headline could equally have been the following:

'24 Hours in a Day'
'7 Days in a Week'

I guess herein is the problem that a lot of us may feel when they see the latest NORAS results. What insight is the research actually offering employers and agencies........I mean insight, genuine insight into the online recruitment process?

Conventional wisdom tells us it makes sense to measure job boards in a like for like manner. The fact that most of the major online recruitment players are now on board (i.e. Monster has finally succumbed) does provide some level of credibility to the research. (I can only imagine the level of delight; satisfaction and hope on the day that Monster did finally decide to come on board after years of resistance/inability.)

The fact that NORAS exists will go some way to giving that 'level of comfort' to certain clients to convince them to migrate spend and creativity online. These may be clients who still resist the obvious benefits that the Internet and its actual mainstream every day audience present to an advertiser who wants to engage in dialogue with prospective candidates. (do these clients still exist?)
The fact that NORAS performs this function is to be welcomed. However it is absolutely critical that it is recognized that it's research outlining applications across job boards and industry section/locations etc is seen for what it is, a first step/toe into the digital recruitment world. There are many more steps to go and be taken before we reach a satisfactory destination and can arrive at a real effective recruitment solution.

'Level of comfort' is an interesting phrase and one we’ve been subject to as an online media planners over the years when trying to convince clients to migrate online. That fact that digital provides a level of unprecedented accountability has enabled us to provide levels of comfort to our clients; however there are varying degrees of ‘comfort’/accountability that we as an industry can offer clients.

It’s the degrees of 'comfort' and accountability we offer and provide clients with that are key in this debate and to this industry.

Applications through a clients website are important however its data showing which channels (within the digital environment and outside) are actually producing hires which is critical and more insightful.

As long as we recognise that Job boards, the bread and butter of a recruitment solution, are still just one strand and touch point in a recruitment solution then the easier such research NORAS will be to consider as part of a wider process.

John mentioned earlier our need to 'keep on pushing the market' this is absolutely critical and what I hope this blog in some way contributes to doing.

I find the contributions on this blog about how certain members of this industry have been pushing the market for a number of years and have therefore a wealth of experience and 'sophistication' in accountability and planning, complimentary but a bit of a cop out.

If we were able to do third party ad serving five years ago (and yes that's for both public and private sector clients) I find it incredibly difficult to believe that this is not possible to do this today in an environment that is so digitally focused.

Five, four, three, two and one year ago we convinced clients to do track their online advertising and see in black and white exactly what they are getting for their money. This wasn't easy and if anything it was internal clients/colleagues who were the toughest to convince. Clients were no more open or online savvy than other clients at other agencies at the time, however they were clever enough to embrace common sense and a good service when they were offered it.

Today we have so many stats in terms on Internet usage at our disposal to make this a no brainer for clients and I know the appetite for digital is out there. In effect the onus is and should be on ad agencies to do show how clients should approach online recruitment in the most effective innovative and accountable way possible. With online tracking and cost per hire MI at our client's disposal (this can be for both small or large campaigns) the debate about NORAS becomes`what it is; a red herring. We had an excuse years ago as to why online accountabilty and considered online media planning didn't happen and some of us didn't accept it.

Today no excuses exist.

[Peter Gold]

My god, you lot do leave long comment posts!

NORAS is not representative of the industry so it ain't worth the page it's hosted on.

Job boards just put up the price if they get to see how many hires they deliver; don't blame them but it's a vicious circle.

Starting up: stay niche, be different.

Great blog, keep the lurkers, they let you know someone is reading!

[Alex]

Brevity is a lesson I think we all missed in school ;-)

Maybe as we get more experienced bloggers we'll feel more able to quickly get to the point and court controversy without the heavy layers of justification - although of course personal employment situations sometimes necessitate consideration being paid to maintaining smooth waters. Ahhhh, the sweet politics of agency life.

:-)

you see - still not particularly succict!

[Richard Grimer]

Hi Ben and Sinead,

I’m the research manager at Enhance Media, responsible for NORAS. I wanted to throw a couple of things into the debate.

Ben – I think you might find NORAS more useful than you think.

It's true, that when the NORAS results are covered in the press that the headlines tend to concentrate on top line findings – the size of sites, the increased use of online etc, but there is a lot of detail within the results that specialists such as you and your colleagues could find useful.

The aggregated data in NORAS shines a light on user behaviour and motivation - such as why visitors choose to visit job boards, why repeat visitors return to job boards and the features / functionality that they use. You can also compare this data across industry sectors, roles, locations etc.

New data is also coming on stream all the time, for example showing that 52% of online job seekers have access to an internet enabled mobile device.
I’d be happy to come in and see you and any of your colleagues to explore how you could get more value from the research. Please let me know if this is of interest.

I’d also be interested to find out about any other research that focused on user habits, candidate or client needs that you’d be interested in looking at, please let me know if you’ve got any specific ideas and we can meet up and talk them through.

[Ben]

You may be right, I may find it more useful than i thought - but for some reason I'm just not so sure.

Whenever we hear about NORAS, it is not for its findings on user behaviour, it is about participants' size or positioning in the market. And it's not just the participating job boards that present it in this way.

I have seen it positioned as helping employers to know 'which are the best sites to use for X, Y or Z jobs' time and time again. Often by yourselves.

And the problem is that the data within is really not going to do that. Because online recruitment is not that simple. There are so many variables that affect matters that a simple table of research results carried out once every 6 months and containing 25+ Job boards is not in any way representative of the UK market. I'm not knocking the process, or the frequency of the research, I am just saying I don't think it's truly representative - and that's why I don't consider it worthy of the positioning it receives.

Back when no-one really 'got online', then maybe it helped. Maybe. But the market has moved so far beyond this now. And I'm not really sure that the research, or the findings, have.

I'll be honest, like I say I haven't spent time recently looking at the data, but it just seems interesting that we have before us here several of the largest recruitment advertising agencies in the UK - all of whom have made a very sizeable commitment to online recruitment, and all of whom seem to share the same opinion.

Your statistic about online job seekers having access to an internet enabled mobile device is definitely just that - a statistic, but what are you saying here?

I know I can draw a speedy conclusion from this - but it would not necessarily be an accurate one.

My question would be, of this 52%, how many actually actively use the internet via their device? Regularly? And what for?

My dad has a phone that is internet enabled. He uses it to phone me on. He has no idea how to access anything other than his phonebook. I don't think he's alone.

[Sinead]

Excellent Richard.

Glad you have joined in the discussion.

I look forward to meeting up with you and hearing the insights into job seeker behaviour that NORAS provides.

[Sinead]

Excellent Richard.

Glad you have joined in the discussion.

I look forward to meeting up with you and hearing the insights into job seeker behaviour that NORAS provides.

[Alastair]

Hi Ben,

What do you use to choose job boards and other media?

Do you draw on your experience?

Gut feeling?

Chase those free trials, like Peter?

Or is it some black art that only a chosen few are privy too?

[Matt]

Hi Richard

Thanks for the joining in, good to get things from Enhance's viewpoint. There is another issue though that hasn't really been touched on. With Enhance offering planning, buying and search engine services as well as NORAS that makes you a competitor to me. While I welcome any contribution from my competitors that is useful to all of us and the industry (that after all is what this blog is about) there has to be limit beyond which I can't engage. I wouldn't expect to sit down with Sinead or Ben for them to do a research project for me as they have such a conflict of interests. Its one of the biggest issues I have with NORAS I'm afraid....if you were just a research company then fine but the fact you use it to promote your planning and buying services makes it impossible for me to see it as independent

[Ben]

Hi there Alastair – long time no speak!

Sorry for the delay, been a bit busy today.

In response to your questions:

1. A broad spectrum of factors go into making a media selection – especially online with all the data we can source - you know that. No point in listing them here – the people reading/contributing to this blog already seem to know what sources they need to use to make an informed decision…
2. Sometimes – certainly forms part of anyone’s decision making process I would suspect
3. Not really – we are building up more and more data that means we no longer need to go for this approach – it takes time, but we are getting there.
4. Free trials? Not sure what you mean here Alastair. Strange thing to say…
5. Black art – I think that’s over dramatising it really. It’s no black art, it’s just a case of a lot of hard work, learning and using those worthwhile tools available to us to make sure that as we develop, we learn and advise our clients accordingly.

Hope that answers your questions.

Hey, I just wanted to say that I really enjoyed reading the report on Ri5 about the recent Enhance Media conference – seriously - that was an impressive list of speakers covering what look to be a really broad and interesting array of subjects. That sounded like a really useful conference.

Nice one, and hats off to you guys for organising that. That must take some work…