PREAMBLE ON DEVELOPING MEASUREMENT STRATEGIES

How do we work out what sorts of measures are needed to help run an enterprise? We’ll use three methods.

One reasonable starting point is to decide on some principles for selecting performance measures. It’s not only reasonable, it’s important because poorly chosen measures can distort people’s behavior, with disastrous and expensive consequences, as we’ll see shortly.

Then there’s a simple way of thinking about how to identify measures that can help you find your way, even in the foggiest circumstances. It’s the Tribus Paradigm, named after a very clear and practical thinker about management: Myron Tribus.

Finally, it’s important to recognize that there’s a hierarchy amongst performance measures: indeed, the Tribus Paradigm leads us inexorably to a performance measurement structure.

So that’s the Framework that will now be developed: a set of Principles for selecting measures, a Paradigm for thinking about which performance measures are needed, and a structure for performance measures.

Small and Medium Enterprises (SMEs) need performance measurement systems too! Indeed, the Performance Measurement Framework described in Chapter 2 was developed initially in the context of seeking to assist SMEs with a widespread weakness in the way they conducted business. And this weakness continues: for example, Antonio Lerro comments

For an SME, adopting a performance measurement system is rather simpler than for a larger enterprise.

1. The starting point is the stakeholder analysis described in Section 11.3: who are the principal parties (sub-groups) in each stakeholder group, and what are their main issues (e.g. Exhibit 11.2)?

2. Given your current business issues, where do you need to focus attention first – Customers? People? Partners? Community? – and which sub-groups of these?

3. For an SME, the number of customers, or the number of people employed, may be too small to warrant full surveys. However, Focus Groups can certainly be conducted to ascertain the key Attributes of the Value drivers, and then the same (and possibly additional) Focus Groups used to gather some basic data on the relative importance of the various Value drivers, and of how your enterprise is rated, relative to your competitors. That should provide a reasonable basis for setting priorities to improve your business.

COMMUNITY VALUE AND BUSINESS IMPACT

PREAMBLE

Is your enterprise regarded as a good corporate citizen?

Does it give your industry a good name or a bad name?

How do you know that the “good deeds” you perform as part of discharging your “Corporate Social Responsibility” are having any impact on the success or otherwise of your business?

In fact, are you doing anything in this area that reflects your company’s concerns about its impact on the local community, or about preserving the environment, or efforts in pursuit of the betterment of humanity locally or globally?

Amazingly, with all the stunning examples of very successful people and corporations who have been putting something back into the community, there are still many enterprises that “don’t get it.”

On the one hand, the Bill and Melinda Gates Foundation provides very considerable financial support for projects directed at improving global health, reducing poverty and aiding development. Warren Buffett, investor extraordinaire, has announced that his accumulated wealth will be donated to the Foundation.

Introduction

In the past, many surveys have reported the response rate (see Chapter 20) as the only measure of the quality of the data from a survey. Although the response rate could be considered a necessary measure of data quality, it is by no means a sufficient definition of data quality. The response rate, after all, measures only the proportion of the attempted sample from which what are regarded as complete responses were obtained. It is certainly true that one can immediately assume that a low response rate is likely to mean that the data will be subject to substantial nonresponse bias and may be far from a representative sample of the target population. However, it does not follow that a high response rate indicates good-quality data.

As a general rule, there are a few measures of data quality that can be applied to all surveys, and there are a few measures of data quality that are specific to specific types of surveys. In this chapter, I deal first with some general measures of quality, and then discuss some specific measures that may apply to only a specific type of survey. The latter are provided only as examples, to assist the reader to develop such measures for the specific type of survey with which he or she may be concerned. Following this the chapter deals with validation surveys, and it concludes by proposing a checklist of measures that would assess the overall quality of the survey effort.

Introduction

The preceding chapters have provided guidance on all the aspects of a survey up to the point of actually implementing the survey. In this chapter, a few suggestions and some guidance are provided on the implementation of the survey itself. The chapter begins with a discussion on the selection and training of interviewers and supervisors, then continues with the fielding of the survey, and concludes with discussion about issues that are likely to arise in the execution of many surveys of human populations.

Interviewer selection

Somewhat tongue in cheek, it is not uncommon to hear university lecturers talk about using students as interviewers, largely because they are comparatively inexpensive and readily available to most research projects. However, the use of students is often not a good idea for the role of interviewers. All surveys, except fully postal surveys, require interviewers. Even postal surveys require survey staff to pack survey materials and ensure that survey packages are sent out to the correct sample of interviewees. For the next few paragraphs, the focus is on interviewers, rather than other survey staff.

Some definitions in statistics

Statistics is defined by the Oxford Dictionary of English Etymology as ‘the political science concerned with the facts of a state or community’, and the word is derived from the German statistisch. The beginning of modern statistics was in the sixteenth century, when large amounts of data began to be collected on the populations of countries in Europe, and the task was to make sense of these vast amounts of data. As statistics has evolved from this beginning, it has become a science concerned with handling large quantities of data, but also with using much smaller amounts of data in an effort to represent entire populations, when the task of handling data on the entire population is too large or expensive. The science of statistics is concerned with providing inputs to political decision making, to the testing of hypotheses (understanding what would happen if …), drawing inferences from limited data, and, considering the data limitations, doing all these things under conditions of uncertainty.

A word used commonly in statistics and surveys is population. The population is defined as the entire collection of elements of concern in a given situation. It is also sometimes referred to as a universe. Thus, if the elements of concern are pre-school children in a state, then the population is all the pre-school children in the state at the time of the study. If the elements of concern are elephants in Africa, then the population consists of all the elephants currently in Africa. If the elements of concern are the vehicles using a particular freeway on a specified day, then the population is all the vehicles that use that particular freeway on that specific day.

Introduction

Before a survey is committed to the field, the various designs and procedures of the survey should be tested. Despite everything that has been written on the topic of the design of surveys, the ways to word questions, the procedures to use, and so forth, each time a new survey is put in the field, new issues will arise and different interpretations and responses will occur among the respondents. Pilot surveys and pretests are the procedures that are used to test and refine the survey before it is actually fielded (see, for example, Freeman, 2003). Unfortunately, those commissioning surveys are often ignorant of the needs for pilot surveys and pretests, or assume that they are unnecessary in this particular case, and therefore do not budget either the time or money to undertake them. However, it has been proven time after time that surveys that are fielded without testing result in problems that could have been avoided, had such prior testing been undertaken (Brace, 2004).

Pilot surveys and pretests can be used for many purposes. Most obviously, they are used to test aspects of the survey, to make sure that everything in the survey works as intended. However, they can also be used to test alternative approaches to various aspects of the survey design and execution, to assess response rates and completion rates of a survey, and to refine the instruments and protocols to be used.

Introduction

The biggest problem with surveys is their cost. Most clients for surveys operate from limited budgets, such that the budget available for a survey is rarely ever sufficient for the desired accuracy and coverage of the survey (Stopher and Metcalf, 1996). As has been discussed in previous chapters of this book, the art of survey design is one of trade-offs or compromise, and this is certainly the case in the area of survey economics. The specific trade-offs involved are those pertaining to producing the greatest accuracy and coverage for the least cost.

Economics in survey design is a serious issue. There are too many instances of surveys having been commissioned and undertaken with too little attention to the economics of the survey design, resulting in one of two outcomes. Either the data from the survey end up never being used and the money spent is effectively wasted, because the data turn out not to be suited to the task at hand, or the firm undertaking the survey produces data fit for the purpose, but only at the cost of a major loss of money in executing the survey – a situation that has, unfortunately, resulted in the bankruptcy of several survey firms around the world over the years.

Introduction

In years gone by it was not uncommon to see on survey forms a dedicated column on each page for coding the responses. As noted elsewhere in this book, this should never be present on a self-administered survey, and it should often be avoided, if possible, on interviewer-administered surveys. In computer-assisted surveys of any type, whether by telephone, self-administered, or internet, it is completely unnecessary, because these surveys will record the computer-coded responses directly to an appropriate computer storage medium.

Coding is a step in which responses provided by the survey sample are converted into codes that permit computer-based analysis to be undertaken. Data entry is then the process of transferring these computer codes from the survey form or other intermediate medium to the computer. Coding is necessary for any responses on the survey form that are provided in a form that is not amenable to analysis. This will include questions that require the respondent to write in a response in words to a particular question, or to a question that offers multiple choices of response, with the possible responses being descriptors of some sort. For example, the set of unordered responses shown in Figure 18.1 (which is the same as Figure 9.10) requires some type of numeric coding to be useful for the analyst.

Introduction

Chapter 6 deals with the alternative methods for conducting surveys of human populations, such as by post, internet, telephone, or in person (face-to-face). Chapter 8 deals with the design of survey instruments, and Chapters 9 and 10 with wording, question design, and layout issues. In this chapter, a number of issues relating to the overall design of data collection procedures are dealt with that have not been the subject of these earlier chapters. These issues include such items as the number and type of contacts, making initial contacts, who should respond to the survey, what constitutes a complete response, an introduction to item and unit nonresponse issues, sample replacement, incentives, and respondent burden.

Contacting respondents

In most sample surveys, it is necessary to contact some sample elements more than once. At the outset of the planning of the survey, it is necessary and appropriate to plan the number, type, and schedule of contacts. In many surveys of households and individuals, there may be a pre-notification contact, a recruitment contact, an interview contact, and a retrieval contact. In addition, there may be various reminder contacts that are also necessary. The actual contacts required will depend on the method by which the survey is being done. In the following subsections, some of the different options are discussed.

Why ethics?

At any time that one is dealing with human populations, it is necessary to consider the ethics of what one may or may not ask members of the population to do. Increasingly, governments and universities alike are becoming extremely sensitive to issues of ethics, especially in relation to the privacy of individuals, and the burden that may be placed upon them by surveys. Several market and survey research organisations around the world have set up codes of practice, regulations, or ethical standards that they request or require their members to practise.

The dictionary defines ethics as relating to morals, or treating of morality or duty, or a set of moral principles. In the context of surveys of human populations, this clearly has to do with what it is morally correct to ask people to do, and how to treat those who are asked to participate in surveys. Perhaps one of the greatest threats to the ability of governments and agencies to undertake survey research that will ultimately benefit people is the unscrupulous and immoral use of surveys by entities that seek to profit from the use of such devices. It is, therefore, perhaps of even greater importance that those agencies engaging in survey research of an altruistic nature adhere to correct moral principles in all matters concerning interaction with the potential participants in a survey.