Job board traffic statements – what price integrity?

So here’s the thing. I recently became aware of how some job boards (well one, but I'm not trawling through every audited traffic statement to check all - at this stage) seek to massage their figures in a manner that is really sticking in my throat. So I’ll ask this question:

When you read about a site’s Unique Users and Page Impressions do you too take that as a face value statement of the overall traffic that comes to the site and has the potential for coming across your job? An indication of the audience that you’re paying for access to?

I did – fool that I am!

Now of course those of us who’ve been dabbling in this world for some time know full well that the UU & PI figures have long since proved any significant value in terms of predicting potential success of a job board and its audience and therefore plays little if any part in online media planning today. The insistence of placing such importance on such a measurement comes from days past when all a media planner had to understand to assist in / justify their planning was ABC & VFD figures. It also panders to the egos of grown men boasting about how big their share is to an, unfortunately, sometimes not very educated / savvy buying audience. But for me this isn’t about how useful the UU and PI figures are – this is about the fact that when any metric is presented, especially when presented in a manner that’s making the most of its independent verification, it needs to have integrity behind it.

So when I discover that a job board is also counting the traffic visiting sites they’ve established that are, in my understanding of it, closed to advertising my vacancies on, then I start to get very annoyed and feel pretty cheated. And, just to clarify further, I'm not talking about a job board with a network of sites which have specialist niche job boards attached that are counted as part of the overall network UU & PI figures - I'm talking about direct recruiting entities for direct recruiting activity (be that client or agency) that I could only appear on were I recruiting within that organisation.

Now I don’t want to get into the game of naming and shaming (been there and scrapped over that one before), and I don’t particularly want anyone else to (but it is a free blogosphere – well kinda). I’d rather see if anyone else has an opinion on how sacrosanct you view the integrity of audited statements and whether you feel at least a little narked that you too may have been taken for a ride by someone hoping you wouldn’t turn to the 'network domains' element of the ABCe audit report either.  Then perhaps if enough people are at least a little miffed by this practise and the people responsible are reading this they’ll get the message and next ABCe audit they’ll do the decent thing (no matter how painful in terms of shedding some reality on their figures) and only count sites that are truly reflective of the audience I, or my clients at least, believe they are buying into when we're quoted their figures.

I know things are seldom absolutely black or white, but am I really being ridiculously naïve? again?! Or perhaps it's just all a misunderstanding and when the mistake is realised there'll be quick moves to correct the over-submission of domains error.

:-]

Comments

[Matt]

Interesting post Alex. I don't normally pay much attention to overall traffic figures but if you are going to go to the trouble of paying to audit them it seems very questionable practice to include sub domains that your advertisers can't appear on. Though you haven't named and shamed, a quick scan of the ABCe site reveals who the culprint is. OK only 7 external organisation's web sites are included but there are some very big names in there. Big names who no doubt spend a fortune promoting their sites....I wonder if the job site in question is giving them a discounted rate for all the free traffic it gets to audit!! If I was one of the third parties' agencies (which I'm not in this case but I know someone who is!) I'd be asking questions...

[Simon Devitt]

Interesting subject and great to see that these certificates are finally getting some greater scrutiny.

Some of the best specialist media buyers I have seen in action in other media have not only kept the publishers in line but also extracted great deals for their customers by taking the time to get detailed knowledge of how the ABC audit works.

Don't for one minute believe that the reducing of bulk sales to flatter newspaper sales figures was stimulated by the desire of publishers to improve quality it was the demands of a small number of buyers who exposed it and pressured publishers to act.

I still commend the website owners that have had the courage to be audited. I understand even more will be audited this year than last as NORAS grows. Surely the site concerned or ABC should act to correct this mistake! Otherwise there is little point in others aspiring to achieve the standard.

A weak ABC would mean that we would be judged by even less reliable panel based benchmarks. Surely a step backwards.

Simon

[DomSumners]

I think audits are great but...
My sense from both sides of the fence was that the ABC would not be that rigorous in policing what is included in statements (certainly in B2B publishing and job boards anyway).If you look further at another massive job boards' sub domains you do wonder if the total traffic of these sites (as opposed to the jobs board part of sites)should really be counted.
So we have a dangerous combination of publishers paying for their own audit and people like us not calling them to account on it - and that is going to lead to if not abuse then certainly "creativity" with numbers.
Let me be clear - sites that audit should be applauded and its great that we can see this sort of info - so hopefully these remarks can be taken in that light.

[Jamie Leonard]

I fully agree with Alex, job boards that include company specific sites within their ABCE audit should be looked at, as other clients will NEVER benefit from that part of the traffic. On the flip side we DO count our networks traffic, but to clarify, ABCE count the recruitment part of the regional paper sites as that’s the only part we build and therefore can be audited on. All of these sites are open to any company to use and therefore the figures are justified. I have also heard a few rumours that we also count the Homes and Cars traffic. If we did our ABCE number would be around 4 million, so no, we don’t.

The most annoying part is that as a media I long for the day that ABCE gains credibility in the eyes of the advertising agencies, but it’s issues like this that will prevent that.

[DomSumners]

jamie
thanks for clarifying that - i agree with your comments too

[Alex]

From ABCe's perspective (if I can speak for them as they clearly don’t want to be drawn on this) they see themselves as very much the paid servants of their masters (in this instance the media) - my words not theirs. So they get told / invited to audit what the site wants audited - they have no input or sway in regards to monitoring the greater integrity of what they're reporting on. They aren't and have never been appointed as integrity policemen - just that at agencies we kind of took them to be playing that role when we were just talking press publication figures and, of late, as the plethora of the figures web properties generate become so potentially convoluted and confusing so they are clearly needing some input and direction, which clearly the media owners aren't rushing to invest in.

Personally I’ve asked this of a leading media senior manager or two previously – unfortunately the response is pretty much “what’s in it for me?” Well maybe the time is upon us to consider that question so that we can help ABCe answer the questions we really want answered and bring to bear metrics that have real currency within the media planning community.

Perhaps it’s time to watch this space. Perhaps.

[Jamie Leonard]

I agree with you Alex and ask a question; ultimately, the agencies are the end users of this information. Like it or not, agencies control over 60% of media spend in the UK which makes you my best friend because guess what, you have the purse strings and I’m in media sales. What would you (the online agency community) want to see in the 3rd party, non biased audit?

I’m not throwing this back at anyone, I’ve just heard of what people don’t want and I feel if we want to move forward with this we have to establish what the end users want.

Alex – you’ll be pleased to know that I agree with your points here. If you use ABC E figures for determining the possible reach of advertising then the current ABC E rules as to what can and can’t be included on an audit certificate need updating.

Broadly speaking, you can include any URL that you own / operate in an ABC E audit as long as you then list it on the certificate – for example if I wanted to audit www.noras.co.uk I could also count traffic that went to www.howeru.com as long as I listed this URL on the certificate. The certificate might then say 20,000 Unique Users, but there’d be no clue as to the breakdown of traffic between these URLs.

The same principle applies to job boards audited by ABC E, they can include traffic from any source, as long as it’s listed on the certificate and as you rightly point out, this can include sources that are closed to advertisers – thus giving potentially misleading top line traffic figures. It would obviously be more useful for advertisers if this wasn’t the case. We tried to get these rules changed for NORAS 2007, but weren’t able to do so. At the time the consensus between the sites involved and ABC E was that any rule change would be too confusing for the market as sites would be forced to produce two different types of certificate – one for NORAS (that only included traffic from areas that you could advertise on) and one for the rest of the market (that included traffic to all URLs they operated).

One thing to bear in mind is that not all ABC E audits are used for the purposes of assessing the possible reach of advertising (some are used to demonstrate total traffic hosted for investors etc) and so changing the rules for everyone isn’t necessarily the right thing to do. However, I think now would be a good time for a rule change – maybe ABC E could produce two classes of certificate – an ‘advertising certificate’ that only included traffic that was accessible to all advertisers and a ‘total traffic’ certificate that included all traffic to URLs operated by the site (this could even work in a similar way to full price sales and bulk sales on press certificates)? The best way to get this or a similar change through is to lobby ABC E, who, after all, are an industry owned body who aim to represent the views of the industry – we just need to express our views clearly to them so that they can represent them.

Finally, it’s worth saying that if it wasn’t for the transparency that ABC E adopt by listing all URLs on their audit certificates then we wouldn’t know about this at all, so well done ABC E for being transparent and to the sites that audit their traffic figures, but I think we should now move on and pressure the sites involved to only include relevant traffic and together we should lobby ABC E for a rule change.

[Tim Elkington]

Sorry, just to let you know that the last comment was from me, I got so excited about posting a comment I forgot to add my name! Tim :-)

[Alex]

Well that's all voices so far in agreement then – which is good.

I'll tell you something for nothing though - I'll be looking at that page of URLs audited in the future. I hadn’t thought of it as being anything other than a "wool pulling" exercise to be fair, so thanks for the other perspective Tim (although I fear you're being a little generous as it's so few who seem to have adopted this practice) - but I'll be certain that anyone pimping such dodgy stats at my teams' door is given the message loud and clear that this practice is only doing them a serious disservice.

As I mentioned - ABCe say that they're only able to audit what they're asked to audit - well if it's that important to have two sets of figures (one for share holder purposes) then I suggest they find the cash for that second audit (am sure there’s a cost efficiency they can realise with ABCe), because right now I will be advising my clients to take anything such sites say with a handful of salt. Perception is so often reality, whether rightly or wrongly - and I perceive that I'm smelling a rat.

Alex

[Matt]

Yes a lot of the above makes sense and as mentioned earlier I'm also concerned that people aren't misled. There is quite a lot of presuppostion though that audited generic traffic figures are a useful or even an integral part of media planning for recruitment advertising purposes. They aren't, it is as simple as that!

[Sinead]

Firstly, thanks to Alex for highlighting the fact that some of the job board ABC audit figures are misleading and do not provide a true reflection of site reach. I, like a lot of people presumed that the unique users and page impression figures were from users looking for a job within the open job board environment where my clients paid-for advertising will have appeared.

I can't help feel disappointed that the reporting of irrelevant domain names was included in the audit. One of the positive things universal auditing would bring is the establishment of a level playing field between all job boards, however with this revelation of dubious traffic inclusion, this throws this aspiration into some doubt.

This isn't to say these issues can't be resolved, I'm sure they can and will be. Of course its time as Jamie points out, for people to state clearly what they want instead of what they don't want. However while its important to ensure the stats in this area have complete integrity its also important (as Matt notes) to not get too bogged down in the importance of these figures in the overall scheme of media planning and choices. They provide an important and useful indicator of site relevance as a first step, however it is other tools and further tracking and insight in terms of applications and hires that are and will be key moving forward. (especially as digital recruitment diversifies into arena's beyond job boards).

Its interesting to note the wider debate around the importance of page impressions in determining the reach of a website. With the onset of Web 2.0 and the dynamic content of sites, it is time spent by a user on a site and thus their likelihood to interact with a brand that is now seen as the key metric.

Its clear we all agree there is room for improvement and that the integrity of all stats and metrics should be sound. This is especially important for smaller sized recruiters who may look solely to an ABC audit and make their media choices as a result.

Sinead

[Matt]

Lots of interesting comments on this one. Shame the job board involved hasn't yet joined in to give their side of the story...maybe they are waiting for someone to actually name them! Come on then Monster...what's your take on it all?

[John]

Love the comments here, love the call for commitment to be independently audited with transparency....but....recruitment advertising needs, and has always needed more than audience. Audience is one factor, and of course you need a significant audience, however what is really needed is marketplace. How do you form a marketplace? plenty of Jobs, from a broad customer and skills/industry base. Plenty of Jobseekers, Responding to the jobs they have found and want to fulfil.

You wouldn’t judge the success of a restaurant by the number of people who read the menu. So why would you judge the success of a recruitment website by page impressions and unique users alone? All this shows is that people have visited the website, with no proof that the content has been read or acted on; a fact that is currently being missed or ignored by recruiters. The measure of online recruitment success is in effectiveness or performance; that is in the form of online job applications. Of course, it is not possible for a jobsite to audit it self past that point without the co-operation of the recruiters and the agencies that work with them, and that day can’t come to soon, but until then, lets at least audit ourselves as far as we are able.

Judging a site solely on the number of unique users only gives the recruiter part of the story; and is the same as judging a newspaper purely on circulation. Whilst NORAS has made significant inroads into increasing understanding of the market, and should be applauded for it; the survey lacks a vital ingredient – audited job application figures. Recruiters have access to widely accepted measures of recruitment website reach, performance and user profile information: such as Hitwise, comScore and ABCe audits. These reports are contextualised and capture data throughout the year: Hitwise tracks visits market share to eg UK ‘Employment and Training’ websites, comScore monitors ‘employment internet audience size’ activity and socio-demographic profile. At totaljobs.com we believe it is the responsibility of the online recruitment company to provide recruiters with reach and response information; to help them make the best media decisions based on robust and transparent information.

And the good news for recruiters is that ABCe can now audit online job applications – with a definition of ‘application’ they find acceptable; so recruiters can now accurately assess where they are most likely to get the response they need. Why don’t NORAS insist on making it compulsory to audit applications? Why don’t job boards want to be judged on applications as well as audience as well as jobs as well as recruiters – that sounds like a recruitment marketplace to me.

And please no old pony about the majority of people applying by phone and letter…….just audit, regularly, transparently and for all metrics that combine to make the recruitment offering that is provided by you as a job board – and if you don’t then agencies should withdraw their clients spend.

[Matt]

Some interesting points John and many I agree with. I'm going to pull some of my thoughts together and start a new post as I think it's a slight shift in topic from this post. What I will say here though is link to the previous debate about aduted application figures. I still think it causes more confusion than it solves

http://digitalrecruiting.typepad.co.uk/digital_recruiting/2007/06/yet-more-confus.html

[Tim Elkington]

I agree with a lot of what you say John - just to clarify, including online applications, online job referrals and email job referrals is optional for NORAS 2008, but the intention is to make auditing at least one of these metrics compulsory for NORAS 2009.

Because there's such a variety of sites in NORAS (in terms of the way that responses are generated) the participants decided to make the metrics optional this year to give those auditing them a chance to get used to the process and the results before they were made compulsory, I think this seems sensible. Another thing to bear in mind (as with all new metrics) is that we might use the current definitions for a year, decide they're not quite right or need refining and then change them based on experience.

So that's why they're optional this year, but I fully agree that it would be more useful for the market if everyone audited application figures and that's certainly our intention for NORAS 2009.

[John Whitehurst]

Just want to say a few things on this ...

1. The metrics

Unique User = reach over time
Visitors = frequency over time
Page Impression = impacts
Applications = response

The figures do relate to each other and should be used together. Looking at the figues can give an idea of the health of a site. e.g. a site should have 10-15 page impressions to users.

2. Methodology

All measurement systems have issues - the trick is to understand what the information says to you. e.g. ABC makes monster look bigger and Comscore makes fish4 look smaller. To be honnest it is not the number - but the relation of the number to your tracking metrics that is important. That is what planning does. Take what you can but one thing is never the answer.

3. Grouping people

It is very easy to create groups of people to measure. The importance for a client is moving from understanding groups to individual difference.

DON'T TRY AND PUT PEOPLE IN BOXES ALL THE TIME - IT HAS A VALIDITY. BUT IT IS NOT THE ANSWER.

[Dave Martin]

Before I started Allthetopbananas.cm I ran the digital business for over 250+ regional newspaper sites and as so spent quite alot of time involved in discussions about ABC/e and other such audits.

It is very positive that ABC list the urls included in the audit - previously many classified sites made large figures from included various sub domains.

The audit process whilst being very detailed is flawed by the pace of technology and companies eagerness to have the biggest numbers.

One post suggests that reviewing the page impressions per unique user is a good metric to the health of a site, if the two figures in question were "real" then this would indeed be revealing. However they are not always quite so "real". One of the biggest techno buzz words of the web for the last 24 months has been AJAX, to the none technical this means a site can go and get some data that you need quickly and provide you a better experience. We have all seen the power of Gmail or Google Maps which are all based on AJAX. The problem is this user benifical technology reduces the number of new pages loaded and a page loaded is a page impression. If a site had lots of Ajax loadin in new data the user requests from searhes the site would only get 1 PI but if they made the site less enjoyable for the user it could be many PIs.

So ABC updated the audit process to stay with times. A Page Impression is a User Click for Information. With this definition and the very high complexity of AJAX web sites it is very easy for the PI metric to be over inflated.

Even prior to such techniques small changes to a site navigation and flow could inflate the Page Impressions, for example a click from a search to a details page is a PI, the click back to the Search results is another PI (even though the user has already seen this information), compare this to the details page popping up in a new window, the user closes the window in order to see the previous results and the site looses out on a PI.

These stats are not meaningful.

Today is the age of the classified search site - there are hundreds of them. The metric surely needs to specific to the application - editorial driven sites may need the typical ABC style figure but a classified site should measure unique users, unique searches per user and unique details clicks per user. This is the information needed to evaluate such a product.

We dont care about PI we really care about searches and response. Well I do when I advertise anyway!

Interestingly with such a metric high figures will not always be good - if users have to perform lots of search and click on low amounts it highlights the search product is not good.

With the ever growing Google Analytics such metrics are very easy to measure - perhaps ABC should go this direction and use tag based auditing. Then it wont be based on just one months stats a year - which is always the sites busiest peak month or follows on from a marketing campaign.

[John Whitehurst]

Interesting point ... but to try and say a page impression is not relevant - does not really get the point of the post.

All metrics are relevant - but you need to understand the limitations. You point out the limitation of PI. But forget the importance of PI and the relation to advertising impressions. The PI number gives the indication of the level of impact your campaign will have whn buying creative executions.

Unique users and searches are great for postings or PPC campaigns. But creative executions, e.g. banners, need page impressions to help in the media planning process.

[Dave Martin]

You are totally correct my main focus was on advertising vacancy posts within a classified search (it my business so forgive me.)

I did mention that traditional PI style metrics are suitable for the more traditional editorial based sites - I suggested a different set of metrics for classified search sites.

Even so your point in regards of display style advertising creating awareness such as editorial based websites has to focus on PI with a far higher importance than I considered. I totally agree.

You mention that I missed the point of the blog - I read the title as "Job Board Traffic Statements..." Whilst I am more than aware that many Job Boards carry a large inventory of display advertising the majority of job boards also charge on a per job rate to advertise a vacancy within their classified search - this I am very sure is their main revenue stream along with CV database sales. The discussion of how relevant PI is to the core sales model of a job board may well be a refreshing comment for this blog - but missing the point, eh No.

Quite the opposite, I am sure that if the market of companies and agencies paying for job posts could select job boards based on metrics closer related to receiving quality CVs the industry would lap it up. Surly the dream is to know if you will get candidates to fill the post quickly!

BTW - NICE BLOG POST fantastic discussion area, really good to see such active, relevant and interesting comments.

[John Whitehurst]

Just a point of clarity for Dave ... i was talking about missing the point of my post - not that you missed the main topic for discussion.

[Shaz]

I work for a job board, currently we're not ABCe audited although our main competitor is. Upon enquiring in 2007 i was given some insight on how to boost the ABCe statistics to "make ourselves look better".
ABCe collects its figures twice a year, the site being audited can predetermine the two, one month intervals, that the auditing is run. We were advised to launch and run huge online & offline marketing campaigns during the audited months to boost traffic stats and user numbers to create a better reflection on the audit. As only quantity rather than quality is measured we were recommended numerous avenues for advertising for pure traffic generation, which would have been mainly irrelevant and of no use to our clients, but would have reflected very well in our published figures.
As a relatively small job board we have chosen not to go down the route of auditing in the near future, as we would rather spend our resources on long term sustainable marketing avenues to attract quality candidates, and on developing our service and brand for the benefit of our clients. We do not have the financial backing of our main competitor who are owned by a large media group, so cannot justify throwing money at a stunt just to keep up with the jones' and inflate our statistics.
in my 5 years here i have only ever once been asked if we were ABCe audited, we lost the revenue from the advertiser that month because we werent, however the following month on direct request from the client and following recommendations from candidates we attained the business and have retained it for over 18 months - because we produce results!
I really enjoyed reading this thread, the auditing system as i see it is flawed, however im unsure if there is any way to distinguish quality targeted traffic from random passing visitors attracted by bright lights and special offers.

[Alex Hens]

Hi Shaz,

welcome to the debate - and quite timely too, as although this was started at the end of last year, there's a meeting of ABCe with their job board partners tomorrow.

I was kindly invited to attend as an observer (as media pay for ABCe's existence only they are invited as participants), as were my blogging colleagues. Unfortunately I can't attend due to a prior speaking engagement with http://efinancialcareers.co.uk/, but I know a few who are attending including my agency's media manager.

But I do think you're on the money and am not surpised by your tale at all - I still think ABCe is in danger of marginalising or indeed making itself completely irrelevant if it doesn't look to sort out its role - particularly in regards to taking responsiblity for it's own integrity through policing any misrepresentation of what it reports. It's not what it was set up to do I know - but as some of its participants quite blantantly take the p*ss it's like the FA saying they are powerless to control the on field behaviour of the spoilt brats of football. they can - they must - they just, perhaps, are too scared to.

In the new world people are, or at least flippin well should be, better placed to evaluate success for themselves (or through their chosen advertising partner) - so boasting about mass user traffic increasingly should be an irrelevance, maybe even a comple turn off. And whilst media refuse to audit anymore than surface level information (for fear of it not showing up so well whilst being compared against competitors) then anyone making a buying decision will care less and less. Anyway - surely it's pretty standard practice nowadays to set up a trial first with any site that even looks like it might be relevant and then judge for yourself on the response?

I think metrics with integrity are good, whilst metrics without are frustrating and at times an annoying distraction. I've recently heard of another ABCe audited job board using their content network to bolster their general figures. I know we've frustratingly become a world with as much spin as substance - I just hope that ABCe can encourage their media backers to see sense before the whole things becomes a sad memory of what might have been, but something that greed and arrogance ensured never was.

Over to you ABCe.

[Tim Elkington]

Just as an update on this, we had a good meeting this morning looking at current ABCe rules with valuable contributions from everyone (both ABCe subscribers and observers, it was very egalitarian ;-)). ABCe are now preparing some notes following the meeting which will form the start of the process that addresses the issues highlighted in this debate - you can read more about it over at my blog.

In relation to Shaz's points about auditing twice a year and spiking traffic in order to give unusually high figures, the best answer to this is for more people to audit more regularly and for the media buying community to push for this - credit should go to Total Jobs who's commitment to audit 12 times a year means that their ABCe figures can be trusted both as reliable and realistic in terms of the traffic they generate on a consistent basis.

[Local Jobs]

Job finder websites are needed to advertise their websites that's why they resort to some ideas such as what is stated above. However, it also has some advantages to the job seekers, you gotta admit that. :)