Sample Design: Part 1
Slide 1
The goal of this lecture on sample design is to introduce you to basic samples and terminology, to discuss the various stages of selecting a sample, to introduce you to two basic types of samples —probability and non-probability—and to explain the different types of samples in each of those categories and when you might use them, to introduce you to the notion of sampling error, and to close with a brief discussion of Internet and panel samples.
Slide 2
As the caption to this cartoon states , “In sampling you examine less than the whole body of data to reach a conclusion about that body. Thus, when you taste your first bite of something and say
‘It needs salt’, you’re sampling.”
Slide 3
Perhaps a non-food context might help. In the four photos shown on this slide, the first one is at normal resolution and the subsequent three photos are of lower and lower resolution. The face remains recognizable when going from photo #1 to #2. By photo #4 , it’s almost impossible to discern a face. This photo analogy hints at issues about sample size and representativeness.
Slide 4
I’ll begin by introducing you to some basic sampling terms: population or universe, population element, census, and sample.
Slide 5
A population or universe is any complete group. It could be a group of people, sales territories, or stores. It’s the total group from which information is needed. Note that a population need not be huge. For example, the students enrolled in this course represent the population of students enrolled in this course.
Slide 6
To take a census means studying all the elements comprising a population. The U.S.
Government attempts a population census every 10 years. Of course, the population could be small, like the students enrolled in this course. If I surveyed all those students, then I’d be taking a census.
Slide 7
In contrast to a census, a sample is a subset of a larger population. Cost and time are the main reasons for drawing a sample. A census is expensive unless the population is small and accessible. Typically, a population is large and not readily accessible, in which case drawing a representative sample is the only cost- and time-effective way to assess the population.
Slide 8
Researchers progress through the following stages as they draw a sample: (1) defining the target population, (2) selecting a sample frame, (3) deciding whether to use a probability or non-
Page | 1

probability sampling method, (4) planning the procedure for selecting sampling units, (5) determining the sample size, (6) drawing the sample, and (7) conducting the field work.
Slide 9
The first step in defining a target population is to consider the research objectives. For example, if I’m interested in conducting a study on different advertising appeals or attempting to fine tune a new questionnaire scale, then an appropriate population is students. Alternatively, if I was a marketing manager interested in adding a product to my existing product line, the population is unlikely to be students, unless the product is sweatshirts or PCs. To operationally define a population like NMSU alumni, I might contact the alumni office and request a complete list of alumni. If the population is NMSU marketing majors, then I might acquire a list of those students from the COB. Although s uch a list won’t be perfect—it will include some non-majors (and nonstudents) and exclude some current majors —I can assume that all current marketing majors appear on that list. When selecting a target population, the convenience and cost of alternative samples should be considered. An operationally defined population that ’s less expensive to access should be used, ceteris paribus, because the savings per completed questionnaire could be put toward collecting a larger sample.
Slide 10
The second step is to select a sampling frame. This will only be relevant for probability or scientific samples. A sampling frame is the list of elements from which a sample may be drawn.
Alternative sampling frames might include mailing and commercial lists, but they can be problematic if they ’re not representative of the target population. For example, for my dissertation research I needed to identify people who relocated within the last six months. At the time, R.L. Pope maintained the ‘new movers list’, so I paid $4,000 for contact information for
40,000 households. Unfortunately, the list was not as advertised, i.e., 80% of households had relocated in the last six months. In fact, one person I contacted hadn’t moved in 57 years! Sadly,
I would have been better off randomly phoning people across the U.S. Although other commercially available lists may perform better, all such lists can be problematic.
Slide 11
The third step is to identify the sampling units. Sampling units can be persons, household, businesses, et cetera. Several years ago, I conducted a study for the NMSU Marketing
Department to assess the skills of and needed by undergraduate marketing majors, so I surveyed several groups. One group was Fortune 500 companies —you can’t survey the company but you can survey the personnel director —and few opted to respond. Yet, the sampling unit was personnel directors for those companies. Sometimes the primary sampling unit is of interest; other times it’s the secondary sampling unit (as per cluster sampling).
Slide 12
The fourth step is to decide if it’s best to draw a probability (scientific) or non-probability (nonscientific) sample. For a probability sample, every population element has a known non-zero probability of selection; for a non-probability sample, the probability of selecting a particular member of the population is unknown. This known or unknown probability of selecting population elements differentiates the two types of samples. A sample is scientific if the probability of selecting any member is known, and non-scientific if the probability of selecting any member is unknown; the latter case makes it difficult to extrapolate from the sample to the population.
Page | 2

Slide 13 (No Audio)
Slide 14
When are non-probability samples preferred to probability samples, and vice versa? Here are five considerations:

Cost is an issue and non-probability samples are far cheaper than probability samples.
For exploratory research, non-probability samples often are serviceable. For conclusive research —from which managers draw conclusions and act accordingly—probability samples should be representative and hence reliable.

If non-sampling error are a major error component, then non-probability sampling is acceptable because sampling error is a lesser error source. Alternatively, if sampling errors are the largest error component, then a probability sample is preferred.

If the population is relatively homogeneous
—members are relatively similar to one another, then intra-population variability is low and a non-probability sample may be sufficiently representative of the larger population. Hence, it may be possible to draw a representative sample without probability sampling. Alternatively, if the population is highly variable or heterogeneous, then a probability sample is preferred because there’s more control over its representativeness.

If statistics —which are irrelevant for non-probability samples—are needed to extrapolate from a sample to the population, then probability sampling is required.

If operational considerations are critical, then a non-probability sample is preferred. Such samples are easier and faster to collect, more accessible, and less expensive.
Probability samples tend to require great care in selection and are more expensive because creating a sample frame is an expensive proposition.
In summary, non-probability samples are great for exploratory research, great if non-sampling errors are larger than sampling errors, the population is similar across members, statistical analyses are not required, and operational considerations are critical. In contrast, probability samples are preferred if decisions will be made based on the research, sampling error is the largest component of total error, population elements tend to be heterogeneous, statistical analyses are important to extrapolate to a larger population, and unfavorable operational considerations are less critical.
Slide 15
At this point, a discussion on the different types of non-probability and probability samples is warranted. I’ll start with non-probability samples because it’s easier to contrast the probability samples after discussing non-probability samples. The four types of non-probability samples are co nvenience, judgmental, quota, and snowball. Then I’ll talk about probability samples: random, systematic, stratified, and cluster.
Slide 16 (No Audio)
Slide 17
I’ve included this cartoon to illustrate the problem with televised polls that news networks run, even when they ’re indicated to be non-scientific, as viewers tend to tune out that caveat
Page | 3

because they don’t know what it means. The cartoon shows a person who cares a lot about the outcome of a television-program-instigated survey and will continue to dial the 900 number to register his vote. Baseball fans are encouraged to submit multiple All-Star ballots; in fact,
Internet-based ballots allow voting for the starting team and the last player elected to the All-
Star team. As a result, votes for the starting team are weighted by fan interest; it isn’t a one person, one vote election.
Slide 18
Convenience samples also are called haphazard or accidental samples. Convenience sampling is a procedure for obtaining people or units that are convenient to the researcher. Convenience is not necessarily a problem and convenience sampling is desirable for some types of research.
In particular, convenience samples are very useful for pre-testing questionnaires because they are inexpensive and can help to identify poorly worded questions and poorly formatted questionnaires.
Slide 19
The sampling error introduced by convenience sampling is twofold. First, certain elements of the target population will be systematically excluded. Second, elements that are not members of the target population will be included. As a result, the sample will only include a biased subset of the target population.
Slide 20
This classic Pogo cartoon makes the point once more that a convenience sample may be great for some purposes but not for trying to generalize to a larger population.
Slide 21
A second type of non-probability sample is a judgment sample, also referred to as a purposive sample. The judgment in this case is the judgment of the researcher; an experienced person selects the sample based on his or her judgment about appropriate characteristics required of sample members. One example of such samples is very familiar: television networks use judgment sampling to forecast the winner before all votes are tallied. These forecasts are based on exit interviews of voters as they leave the polling precinct. It is cost prohibitive to place interviewers at every precinct in the U.S. However, there ’s historical data on the districts whose voters are most predictive of the election. With historical data, the networks can identify these key precincts and station interviewers at key polling places. The judgment here is informed judgment based on historical voting patterns. If conducted properly, exit polling provides accurate forecast; the misleading exit polls for the Bush-Kerry presidential election were a byproduct of poor questioning rather than poor sampling
Slide 22
Unlike convenience sampling, in which people who are and aren’t members of a population are surveyed, the judgment sample includes only relevant member of a population. However, if certain members of the population are systematically excluded from the sample, then the survey results may be biased.
Page | 4

Slide 23
A quota sample is a sample in which various population subgroups are representative on pertinent sample characteristics to the extent desired by researchers.
Slide 24
Sex and age are two characteristics that a researcher might want to control. For example, the researchers for this chewing gum study controlled respondents such that 45% are male and
55% are female. The same can be done for age of respondents and mouthwash use or gender and razor blade consumption. For mouthwash, the field service could be required to solicit responses so that 45% of the sample was male and 55% was female, 20% were between the ages of 16-24, 25% were between 25 and 34, et cetera.
Slide 25
The last of the four non-probability samples is called a snowball sample. For such samples, a small initial set of respondents is selected by a probability method and additional respondents are recommended by these initial respondents. Assume I want to survey a group that is rare in the population and there’s no good mailing or phone list available. I might randomly dial the phone, looking for people who qualify as members of the target population. If I dial enough numbers, I’ll eventually find one person. It’s likely that one person knows other similar people. At the end of the interview , I’d ask that person for three other people who are qualified to participate in my study due to similar interests, predispositions, and behaviors. For example, suppose there was no good list of NMSU alumni. I could start dialing phone numbers in New
Mexico until I contacted a NMSU alum. At the end of the interview, I would ask for three more alums, then contact those three people and ask each of them for three more alums, and so on.
Basically, the sample would snowball from one or a handful of initial respondents who would identify other qualified respondents, who would in turn identify more respondents, et cetera.
Page | 5