If you want to get publicity for some idea, promote a product, or just get in the news, then you should report the results of a meaningless survey. Search on Google using the phrase "a survey has shown that...", and you will see what I mean. You will learn that one in six therapists have tried to cure homosexuals, that more than 70% of people would exchange their computer password for a bar of chocolate, that Americans who attend church are more likely to favour torture than those that do not, and so on. You don't need to bother with getting a good response rate, a representative sample, or even a valid and reliable questionnaire. Just circulate some questions to a few people, and send the most eye-catching result to the press.
There are also plenty of meaningless surveys which never get to the press, but are circulated within companies, government departments and universities. These are often promoted as 'quality assurance', and are even taken seriously by some people. Management boards ponder reasons for a fall in satisfaction ratings by 5% on a survey with a response rate of 20%, without admitting that the whole exercise does not mean very much. Truth to tell, survey results might not mean much even if the response rate was 100%. Many meaningless surveys use ambiguous questions coupled with dubious Likert scales (the kind which assign numerical scores to a range of five or so questions from 'very satisfied' to 'very dissatisfied'). These have the apparent advantage of producing a numerical score and hence allowing statistical analysis. Usually however, people only look at mean scores, and these can be misleading. A survey in which 50% of respondents were 'very satisfied' and 50% 'very dissatisfied' would produce the same mean score as one in which 100% said they were 'neither satisfied or dissatisfied'.
What's the alternative? It is essential for organisations to assess the quality of what they do, and their customers/citizens/students are in a good position to assess this. Rather than assessing mean scores on Likert scales, organisations should concentrate their attention on the causes of satisfaction and dissatisfaction, and ideas for improverment. The best way of doing this is probably to use open-ended interviews or focus groups. Of course, this would require quality assurance staff to be skillful in survey techniques, to be creative, and to be prepare to co-operate with front-line staff rather than stand in judgement over them.
There are also plenty of meaningless surveys which never get to the press, but are circulated within companies, government departments and universities. These are often promoted as 'quality assurance', and are even taken seriously by some people. Management boards ponder reasons for a fall in satisfaction ratings by 5% on a survey with a response rate of 20%, without admitting that the whole exercise does not mean very much. Truth to tell, survey results might not mean much even if the response rate was 100%. Many meaningless surveys use ambiguous questions coupled with dubious Likert scales (the kind which assign numerical scores to a range of five or so questions from 'very satisfied' to 'very dissatisfied'). These have the apparent advantage of producing a numerical score and hence allowing statistical analysis. Usually however, people only look at mean scores, and these can be misleading. A survey in which 50% of respondents were 'very satisfied' and 50% 'very dissatisfied' would produce the same mean score as one in which 100% said they were 'neither satisfied or dissatisfied'.
What's the alternative? It is essential for organisations to assess the quality of what they do, and their customers/citizens/students are in a good position to assess this. Rather than assessing mean scores on Likert scales, organisations should concentrate their attention on the causes of satisfaction and dissatisfaction, and ideas for improverment. The best way of doing this is probably to use open-ended interviews or focus groups. Of course, this would require quality assurance staff to be skillful in survey techniques, to be creative, and to be prepare to co-operate with front-line staff rather than stand in judgement over them.
No comments:
Post a Comment
Comments welcome