Data collection

Various factors influencing a margin of error - When someone have to gather data, it is likely he will try to build a quantitative study. The logic of this choice is: it will be more precise. Indeed, it will be possible to have an evaluation of the statistical precision of the results. Everything will seem so perfect ! Unfortunately, things are not so simple.

When someone have to gather data, it is likely he will try to build a quantitative study. The logic of this choice is: it will be more precise. Indeed, it will be possible to have an evaluation of the statistical precision of the results. Everything will seem so perfect ! Unfortunately, things are not so simple.

First of all, the statistical error measures only the error of statistical origin. This error is the one which results from the proportion of chosen people and the number of chosen people. But, this truism is often forgotten. As of consequence, many will react to the statistical error just like if nothing else enters the calculation of the error. So, it is with a false feeling of confidence that many will believe to have done a precise study.

Numerous other sources of errors can occur. In a previous article: The issues related to practical assessment of a margin of error, I had made an overview of some of these sources of errors. They have in common to be related with non-quantifiable factors. So, they seem to be invisible and as of consequence : inexistent. In fact, the only thing which can render those visible, is the meticulousness of the researcher. He should be careful about every detail in order to localize and neutralize any problem. Here are some of the places where it might be useful to look at :

Bias related to the researcher. By distributing its questionnaire, the researcher can give (by its attitude) to the respondent various clues on the expected answer. It is as if the researcher gives to the respondent what should be the "good answers", or in other words : what he wishes to read or listen. Obviously, this effect can occur particularly in the case of an interview. At this moment, the verbal attitudes and various gestures are a significant part of the relation researcher - respondent. In the case of a questionnaire, the phase of contacting the respondent contains also a risk to influence the respondent. Sometimes, the researcher might wants to present the questionnaire so as to stimulate the interest of the respondent and ensure a better rate of answer. In wanting to interest the respondent, the researcher might talk a little bit too much about what it is expected.

Bias related to the respondent. Everyone who answers a survey will tend to give a positive image of itself. It is good old reflex, more or less conscious of giving the good answer. And this "good answer" is often influenced by what is socially desirable or acceptable. This biais can, partially, be circonvent by clearly specify the procedures of confidentiality of the survey. I use expression "partially" because there will always be some cases where this reflex will be very strong, even impossible to circumvent. One will understand that a questionnaire related to a sensitive matter (socially or otherwise) will be more at risk to generate the reflex of the socially desirable answer.

Paradoxically, a similar problem is also possible if the respondent considers that subject is too trivial. At this moment, he might not make the effort to answer seriously. The answer will rather be some general ones with no specific relation with the opinion of the respondent. An example of a situation of this kind is explained in the article: Keeping your objectivity .

Bias related to the non-respondent and the undecided. This error can occur at the time of the analysis. Some think that people who do not answer, or those who are undecided, have no precise characteristics. Then, some believes to be able to ignore them, to make as if they do not exist. There is nothing more mistaken ! This topic had been already approached in detail in the article Les indécis. Those who do not answer, often have precise characteristics. To identify these people, you can try to compare the characteristics of those who do not answered some questions with the characteristics of those who answered all the questionnaire. You will sometimes realize that some groups (age, salary, sex, schooling ...) tend to answer in a different way.

As for those who do not answer the questionnaire at all, it is more difficult to localize them. They become "mixed" with another source of error which is related with the sample. There is no efficent way to make the distinction between the persons who are not in our results because they do not want to answer and the persons who are not in our results because they are not in the sample.

The question of bias related to the sample as well as various other problems will be the main topic of the second part of this article.

As we saw in the previous article, the margin of error of a survey does not lies in the sole statistical error. Often, non-statistical errors have a significant impact. Result is very simple, by calculating the statistical margin of error, one can easily have a false sense of security. The causes of error which we talked about were:

Bias related to the sample. A sample should be a scaled model of the population from which you seek data. But to reach this goal is not an easy task. A sample sometimes can contain some distortions. It is rare that a sample is really neutral. In a previous article: The phone book, I talked about distortions that might be induced by this kind of lists. To identify people who are not in the sample, you can compare the characteristics of the respondents with statistical data from a reliable source. In doing so, you might sometimes discover that some parts of the population (age, salary, sex, schooling) are not represented in the good proportions.

Regrettably, it might also happen that some individuals will refuse to answer your questionnaire. It is then difficult to separate the persons who do not answer because they are not a part of the sample from those who do not answer because they completely refused to answer it.

Bias related with the structure of a questionnaire. The structure of a questionnaire must be planed with care. It can happened that this structure suggests answers to the respondent. This topic was talked about in more details in the article : The issues related to practical assessment of a margin of error ? A questionnaire can be seen as a "thinking sequence" in which the respondent should engage himself. This sequence must give the respondent a feeling of continuity. At the same time, it must avoid the risk of giving clues to the respondent about the "right answers". Let us imagine that in the first questions, your respondent read questions related to the consequences of the problem " X ". Then, following these questions, is a one related to the seriousness of the problem " X ". The answers to this last question will then be modified. The proportion of respondents saying that " x " is an awful problem will increase.

There is also another factor which is not directly related with the structure of the questionnaire but rather with the topic of the questions. Some topics which seem too trivial from the respondent point of view, can make your questionnaire looks "silly". Let us imagine a survey on the new decoration of an establishment. Some characteristics of this decoration can seem superfluous for the customer. In contrast, too sensitive subjects can make your respondent lie. Let us imagine a study on the alcohol consumption or prostitution. The respondent can tend to answer according to what is socially desirable. This subject was already approached on the previous article in the subtitle ways connected with the respondent.

Let us take the example of a survey where one tries to assess the importance of various factors for consumers. It is what can happened in a marketing research where one wants to identify what drive a consumer toward the bying of "bubble gum", or anything else that can be sold. Sometimes, you may see choices of answers like this one :

But, there is a problem with this choice. It is inadequate. Why ? this will be the topic of our next article. You will have then more details about this issue. Furthermore, we shall examine problems related with the methods.

In our previous article, our topic was the issues was the assessment of the importance of a criterion for a respondent. For that purpose, it can be efficient to use some kind of scale in our questionnaire. Thus a researcher can obtain information related to the level of interest for such or such factor. It is often in the wording of this scale that lies a source of errors. For the sake of the argument, lets assume the following scale is used:

At this point, it does not seems to show any problem. The goal is to assess the importance of such or such factor. And it is what this scale is doing by using a simple choice of answers. It is a simple scale, but it obviously serves its purpose. But, take a look closer ! You might uncover a difficulty which might easily escape the unkeen mind. Let us imagine an individual who does not like what we can name "criterion 1". What might he answer ? In the mind of this respondent, this criterion can be a factor very important to avoid. Now, the context suggest rather that the expression "very important" has to have a positive overtone. What will our respondent do ? He is likely to choose : not important.

One can notices that a respondent answering the "not important." category can either be:

In the same logic, a respondent choosing the category " very important " can eater be:

As you can see, there is a significant problem related to the interpretation of the answers. This error is important to the point of jeopardizing the credibility of a survey which would use this kind of scale without any precaution.

You can notice that margin of error is much more than the sole statistical error. In all the steps of a research (by survey or otherwise) bias and errors can occur. This will result a poor quality of gathered data.

Another place where a source of errors can show up, arise from the research method used. Indeed, some methods can leave more place to the "common sense" of the researcher. It is mainly the case when a researcher is engaged in a procedure of interviews. This method often implies a strong interaction between the researcher and the respondent. So, there is a possibility that the exchanges of information might take the researcher out of his frame of search. Because of this interaction, the researcher can be brought to ask some questions in a different fashion from a respondent to another. Then it might be difficult to compare the answers. Solution to this problem lies in the follow-up of a rigorous plan of interview. So, the researcher is enable to track down the elements which can lead the searcher away from its target. It is then easier to decide whether a new element is a risk to be lead away or an occasion to look at an unforseen factor.

This kind of risk can be encountered if someone use the method called discussion group (or focus group). In that case, there are issues related to the animation which can lead you towards intricated situations.

It is extremely difficult to be exhaustive in enumerating problems which can have an influence on the precision of the results of a study. The framework of this series of articles, was to suggest clues so that he can have a better understanding of those issues that will need a particular attention. Obviously, searching in the field of human facts will always be tricky.

On one hand, you must keep in mind that statistical error is not the only one. Even though it is not possible to compute errors others than fo statistical origin, they are nevertheless present.

On the other hand, they are the qualitative methods which present a particular risks because of the interaction between the researcher and respondent. However, quantitative studies are not free of any problem. The stakes in the structure of a questionnaire are sufficient to derails the results of a study. Too often, one will only trust the computations of the statistical error and forgetting quite other cause. This attitude will usually happened during a quantitative study where the presence of calculated results will lure you into a false sens of security.