Who hasn’t heard the line “do as I say, not as I do”? It’s a handy device for many situations. A “free pass” for a parent admonishing a child to eat more vegetables. An “insurance policy” for a teacher confident only of his theoretical grasp of a subject. A “get out of jail free card” for someone providing cheap advice. In each of these situations, what the speaker says is much more important than how he or she acts. Indeed, you would be foolish to put any stock on their actions – you’ve been warned in no uncertain terms not to!
However, there are situations in which we ought to put more faith in what people actually do than in what they say they will do. This is especially true if we are trying to get an indication of what people would do before they have the opportunity to do anything at all. Surveys about firms’ and employees’ intentions fit into this category. These include, for example, surveys about what executives feel is their company’s most challenging HR issue; surveys about how much firms are going to spend on salary increases next year and whether employees are engaged or not. We ought to take such survey results with heaps, not grains, of salt.
The problem is that such surveys are the life-blood of any discussion about human resources issues. You cannot open an HR related publication or attend an HR conference without being bombarded by results of some survey or the other. It seems as if arguments only have credibility if they are backed up by some survey result, without regard to the quality of the survey or the applicability of its results — or perhaps worse, that issues not covered by a survey are not worth raising. Indeed, most HR discussions open with an anecdote, which quickly qualifies as a trend when it invariably draws some nodding heads. From there, only a survey result will do to affirm the item as a fact.
There’s no arguing that surveys are extremely useful devices and can intelligently inform us about peoples’ perceptions or likely courses of action. There are many thoughtfully designed and carefully executed surveys out there. Regrettably, there are many more poorly conceived and frivolous surveys extant, too, and they can garner more attention with their “sensational” findings. How do you distinguish between the two? There are a number of ways to assess the quality and reliability of survey results, including sampling technique, sample size and survey instrument design.
I’ll cover all of these in subsequent posts, but to whet your appetite, let’s take a look at the following examples, both reported in the Facts and Figures section of the November issue of Human Resources Executive Magazine. Read the results and think about them for a few minutes. How was the sample selected? Who exactly is in the sample? Are there enough respondents to be able to make reliable inferences about the broader population? About you or your firm? Are the questions framed properly?
I’m confident that you will agree that you might think twice before using the survey results, as published at least, in any way.
In a new survey of 533 employed Americans, 53 percent say they would prefer to stay with their current company, even if they were offered a good job in another organization. Source: GfK Custom Research North America, New York
Ninety percent of 466 large, U.S. companies surveyed say they expect to lose the grandfathered status on their existing health plans by 2014. Source: Hewitt Associates, Lincolnshire, Ill.
Hidden Job Market
Just 22 percent of all positions with an annual compensation of $200,000 or more are posted on job boards or websites, according to 679 search firm consultants and HR leaders. That figure is down from 24 percent in 2009 and 30% in 2008. Source: Execunet.com, Norwalk, Conn.