PAF 501 - Chapter 1: Science and Scientific Research Natural Science This science of natural occurring objects or phenomena, such as light, objects, matter, earth, celestial bodies, or the human body. Three categories of Natural Science: Physical Science (physics), Earth Science (geology), Life Science (biology). Precise, accurate, deterministic (ex. Physics testing speed of sound) Social Science This science of people or collections of people, such as groups, firms, societies, or economics, and their individual or collective behaviors. Three categories of Social Sciences: Psychology (study of human behavior), sociology (study of social groups), economics (study of firms, markets, economics). Less accurate, less deterministic, and have a high degree of measurement error (ex. Measuring someone’s happiness) Basic Science (Pure Science) Explains the most basic objects and forces, relationships between them, and laws governing them. (ex. Physics, mathematics, and biology). Applied Science (Practical Science) Are sciences that apply scientific knowledge from basic sciences in a physical environment. (ex. Engineering) Scientific Knowledge Refers to a generalized body of laws and theories to explain a phenomenon or behavior of interest that are acquired using the scientific method. Laws Are observed patterns of phenomena or behaviors (ex. Newton’s Three Laws of Motion) Theories Are systematic explanations of the underlying phenomenon or behavior Scientific Evidence Only based on two things, logic (theory) and evidence (observations). The two are interconnected as theories provide meaning and significance to what we observe, and observations help validate or refine existing theories or construct new ones. Theoretical Level Concerned with developing abstract concepts about a natural or social phenomenon and relationships between those concepts. Empirical Level Concerned with testing the theoretical concepts and relationships to see how well they reflect our observations of reality, with the goal of ultimately building better theories. Inductive Research (Theory-Building) The goal of a researcher is to infer theoretical concepts and pattens from observed data. Most useful when there are few prior theories or explanations. (Generalization of observations) Deductive Research (Theory-Testing) The goal of the researcher is to text concepts and patterns known from theory using new empirical data. Not only are you testing the theory, but you’re hoping to possibly refine, improve, and extend it. Most useful when there are many competing theories of the same phenomenon. (Test hypothesis) Methodological “Know how” to conduct the right type of scientific research Theoretical “Know what’ to measure and look at for scientific research Scientific Method Refers to a standardized set of techniques for building scientific knowledge, such as how to make valid observations, how to interpret results, and how to generalize those results. Must satisfy four characteristics: - Logical: Scientific inferences must be based on logical principles of reasoning Confirmable: Inferences derived must match with observed evidence Repeatable: Other scientists should be able to independently replicate or repeat a scientific study and obtain similar, if not identical results. Scruntinizable: The procedures used and in the inferences derived must withstand critical scrutiny (peer review) by other scientists. Exploratory Research Often conducted in new areas of inquiry, where the goals of research are (1) to scope out the magnitude or extent of a particular phenomenon problem, or behavior, (2) to generate some initial ideas about the phenomenon or, (3) to test the feasibility of undertaking a more extensive study regarding that phenomenon. Descriptive Research Is directed at making careful observations and detailed documentation of a phenomenon of interest. These observations must be based on the scientific method. (ex. Population growth, demographic statistics). Looking for the “what”, “where”, “when” of the phenomenon. Explanatory Research Seeks explanations of observed phenomena, problems, or behaviors. Looking for the “why” and “how” of the phenomenon. Rationalism Coined by Greek philosophers, it suggests that the fundamental nature of being and the world can be understood more accurately through a process of systematic logical reasoning. Empiricism Coined by Francis Bacon, suggested that knowledge can only be derived from observations in the real world. Eventually led to the scientific method Natural Philosophy The idea of fusing Rationalism and Empiricism together. Focused specifically on understanding nature and the physical universe. (Galileo and Sir Isaac Newton). German Idealism Argued that experience is purely subjective and processing them using pure reason without first delving into the subjective nature of experiences will lead to theoretical illusions. (Immanuel Kant) Positivism Suggested that theory and observations have circular dependence on each other. While theories may be created via reasoning, they are only authentic if they can be verified through observations. Was usually tied to quantitative research methods. (Auguste Comte) Antipositivism Employed qualitative methods such as unstructured interviews and participant observation. Postpositivism Amends positivism by suggesting that it is impossible to verify the truth although it is possible to reject false beliefs, though it retains the positivist notion of an objective truth and its emphasis on the scientific method. Critical Research (Critical Theory) Retains similar ideas of critiquing and resolving social inequality and adds that people can and should consciously act to change their social and economic circumstances, although their ability to do so is constrained by various forms of social, cultural, and political domination. PAF 501 - Chapter 2: Thinking Like a Researcher Unit of Analysis Refers to the person, collective, or object that is the target of the investigation. (Individuals, groups, organizations, countries, technologies, and objects). Concepts Generalizable properties or characteristics associated with objects, events, or people. Construct An abstract concept that is specifically chosen or created to explain a given phenomenon. Broken down into several underlying concepts: - Unidimensional: ex. Someone’s weight Multi-Dimensional: ex. Someone’s communication skills Operational Definitions Define constructs in terms of how they will be empirically measured (ex. If you’re measuring temperature, are you using Celsius, Fahrenheit, or Kelvin?). Variable Is a measurable representation of an abstract construct. Nomological Network: - Independent Variable: A variable that explains another variable (the cause) Dependent Variable: Variables that are explained by other variables. (the effect) Mediating Variables: A variable that are explained by independent variables while also explain dependent variables. Moderating Variables: A variables that influences the relationship between independent and dependent variables. Control Variables: A variable that must be controlled for study because if not it has the ability to impact other variables. Proposition Is a tentative and conjectural relationship between constructs that is stated in a declarative form. (ex. An increase in student intelligence causes an increase in their academic achievement) (stated in an theoretical plane) Hypotheses The empirical formulation of propositions, stated as a relationship between variables. (ex. An increase in students’ IQ score causes an increase in their grade point average) (stated in an empirical plane) Theory A set of systematically interrelated constructs and propositions intended to explain and predict a phenomenon or behavior of interest, within certain boundary conditions and assumptions. (explains the phenomenon) Model Is a representation of all or part of a system that is constructed to study that system. . . e.g. how the system works or what triggers the system. (represents the phenomenon) Deduction Is the process of drawing conclusions about a phenomenon or behavior based on theoretical or logical reasons and an initial set of premises. (ex. A bank enforces a strict code of ethics for its employees and Jamie is an employee at that bank, then Jamie can be trusted to follow ethical practices) Induction Is the process of drawing conclusions based on facts or observed evidences. (ex. If a firm spent a lot of money on a promotional campaign, but the sales did not increases, then possibly the promotion campaign was poorly executed). PAF 501 - Chapter 3: The Research Process Paradigms The mental models or frames we use to organize our reasoning and observations (belief systems) Ontology Refers to our assumption about how we see the world e.g., does the world consist mostly of social order or constant change. S Epistemology Refers to our assumptions about the best way to study world, e.g., should we use an objective or subjective approach to study social reality. Functionalism View the world consisting of social order and seek patterns of ordered events or behaviors and believe that the best way to study the world is using objective approach by using standardized data collection like surveys. (Objectivism/Social Order) Interpretivism Study social order through subjective interpretation of participants involved, such as interviewing different participants and reconciling differences among their responses using their own subjective perspectives (Subjectivism/Social Order) Radical Structuralism Believe that the world consists of radical changes and seek to understand or enact change using an objectivist approach (Objectivism/Radical Change) Radical Humanism Wish to understand social change using subjective perspectives of the participants involved (Subjectivism/Radical Change) Research Process Exploration 1) Observation: We observe a natural or social phenomenon, event, or behavior that interest us. 2) Rationalization: We try to make sense of or the observed phenomenon, event or behavior by logically connecting the difference pieces of the puzzle that we observe, which in some cases, may lead to the construction of a theory. 3) Validation: We test our theories using scientific method through a process of data collection and analysis and in dog so, possibly modify or extend our initial theory. The first phase of research. This phase includes exploring and selecting research questions for further investigation, examining the published literature in the area of inquiry to understand the current state of knowledge in the area, and identifying theories that may help answer the research questions of interest. 1) Research Questions: Identifying one or more questions dealing with a specific behavior, event, or phenomena of interest that you wish to seek answers for in your research. (Research questions are more appealing if they can be for a broader population). 2) Literature review: three-fold purpose (1) to survey the current state of knowledge in the area of inquiry (2) to identify key authors, articles, theories, and finding in that area and (3) to identify gaps of knowledge in the research area. 3) Theories: Will help identify which of these constructs is logically relevant to the target phenomenon and how. Research Design This process is concerned with creating a blueprint of the activities to take in order to satisfactorily answer the research questions identified in the exploration phase. 1) Operationalization: Is the process of designing precise measure for abstract theoretical constructs. 2) Research Method: What way are you going to collect data to address their research question(s) of interest. 3) Sampling: Choose the target population from which you wish to collect data. Closing linked to the unit of analysis in a research problem. Research Proposal Details all of the decisions made in the preceding stages of the research process and the rationale behind each decision. What’s your question? Why? What’s the prior knowledge on it? What theories do you wish to employ? What hypothesis do you want to test? How can it be measured? What’s the research method? Why? What’s the desired sampling structure? Etc. Research Execution Arrive here once you decide who to study (subjects), what to measure (concepts), and how to collect data (research method). Now you include pilot testings the measurement instruments, data collection, and data analysis. 1) Pilot Testing: Testing on usually a small subset of the target population that you’re seeing if there’s any problems in your research design. 2) Data Collection: Once your pilot test is successful you can start collecting your qualitative or quantitative data. 3) Research Analysis: Analyze and interpret the data for the purpose of drawing conclusions regarding your research question. 4) Research Report: Document the entire research process and its findings in the form of a research paper, dissertation, or monograph. This should outline in detail all the choices made during the research process and why, and what the outcomes at each phase were. PAF 501 - Chapter 4: Theories in Scientific Research Science Knowledge represented as a collection of “theories” derived using the scientific method. Idiographic Explanations Are those that explain a single situation or event in idiosyncratic detail. (ex. You did poorly on an exam because (1) you forgot tht you had an exam on that day, (2) you arrived late to the exam due to a traffic jam, (3) you panicked midway through that exam). Nomothetic Explanations Seek to explain a class of situations or events rather than a specific situation or event. (ex. Students do poorly in exams do so because they did not spend adequate time preparing for exams or that they suffer from nervousness, attention-deficit, or some other medical disorder). Generalized across situations, events, or people . . . less precise, less complete, less detailed. Theory Four Building Block of Theory: Constructs, propositions, logic, and boundary conditions/assumptions. 1) Concepts: Capture the “what” of theories. Reminder concepts can be unidimensional (embody a single concept . . . height, weight, age) or they can be multi-dimensional (embody multiple underlying concepts . . . culture, personality). 2) Propositions: Capture the “how” of theories. Associations postulated between constructs on deductive logic. Should ideally indicate a causeeffect relationship (if X happens, then Y with follow). Exists only on the theoretical plane (Construct A -> Proposition -> Construct B) o Hypothesis: Is the empirical formulation of a proposition. Exists only on the empirical place. (Independent Variable -> Hypothesis -> Dependent Variable). 3) Logic: Represents the “why” of theories. Provides the basis for justifying the propositions as postulated. Acts as like the glue that connects the theoretical constructs and provides meaning and relevance to the relationships between these constructs. 4) Boundary Conditions/Assumptions: Examines the “who, when, and where” of theories. Govern where the theory can be applied and where it cannot be applied. (ex. Many economic theories assume that human beings are rational and employ utility maximizations based on cost and benefit expectations as a way of understanding human behavior. How can we evaluate the goodness of a given theory? - How do we build a theory? - Logical Consistency: Are the theoretical constructs, propositions, boundary conditions, and assumptions logically consistent with each other? Explanatory Power: How much does a given theory explain (or predict) reality? Falsifiability: Theories cannot be theories unless they can be empirically testable to see if their falsifiable. Parsimony: Examines how much of a phenomenon is explained with how few variables. Grounded Theory Building: Observe patterns of events and behaviors. Bottom-Up Conceptual Analysis: Identify different sets of predictors relevant to the phenomenon of interest using a predefined framework. Agency Theory (Principal-Agent Theory) Classic theory in the organizational economics literature that explains two-party relationships. (ex. Such as those between an employer and its employees). The core assumptions of this theory are that human beings are self-interested individuals, boundedly rational, and risk-averse. (Consists of a principal and agent) Theory of Planned Behavior Presumes that individual behavior represents conscious reasoned choice and is shaped by cognitive thinking and social pressures. Innovative Diffusion Theory Explains how innovations are adopted within a population of potential adopters. Four key elements in this theory are innovation, communications channels, time, and social system. Elaboration Likelihood Model Dual-process theory of attitude formation or change in psychology literature. Explains how individuals can be influenced to change their attitude toward a certain object, events, or behavior and the relative efficacy of such change strategies. General Deterrence Theory Examines why certain individuals engage in deviant, anti-social, or criminal behaviors. PAF 501 - Chapter 16: Research Ethics Ethics Conformance to the standards of conduct of a given profession or graph. - - Example: scientists should not manipulate their data collection, analysis, and interpretation procedures in a way that contradicts the principles of science or scientific method or advances their personal agenda. It’s an moral distinction between right and wrong, not necessarily what’s legal and illegal. Voluntary Participation and Harmlessness Subjects in a research project must be aware that their participation in the study is voluntary, that they have the freedom to withdraw from the study at any time without any unfavorable consequences, and they are not harmed as a result of their participation or non-participation in the project. Informed Consent Form Clearly describes their right to not participate and right to withdraw, before their responses in the study can be recorded. - If under 18, form must be sign by legal guardian. Must retain these forms for often times up to three years after the study concludes. Legal Condition Anonymity Implies that the researcher or readers of the final research report or paper cannot identify a given response with a specific respondent. Confidentiality Which the researcher can identify a person’s responses but promises not to divulge that person’s identify in any report, paper, or public forum. Disclosure Usually the obligation to provide some information about their study to potential subjects before data collection to help them decide whether or not they wish to participate in the study. Analysis & Reporting Researchers have ethical obligations to the scientific community on how data is analyzed and reported in their study. Institutional Review Boards The IRB reviews all research proposal involving human subjects to ensure that the principles of voluntary participation, harmlessness, anonymity, confidentiality. 1) Exempt: o Protected subject identity o Work already done o Normal course of events 2) Expedited o Social Science Data o Medical but not dangerous 3) Full Review o Invasive o Pain / harm Association of Information Systems (AIS) The global professional association of researchers in the information systems discipline, - - Category 1: Includes serious transgressions such as plagiarism and falsification of data, research procedures, or data analysis, which may lead to expulsion from the association, dismissal from employment, legal action, and fatal damage to professional reputation. Category 2: Includes less serious transgression such as not respecting the rights of research subjects, misrepresenting the originality of research projects, and using data published by others without acknowledgement. In-Person Notes (8/30/21): - - Asch Experiment: Taught us about group conformity o College student asked to say which line length was closest to the baseline: A, B, C Confederate in group selected the wrong answer. Milgram Experiment: Authority figure. Will you bend your standards to adhere to authority. o Experimenter directs the teacher to give electric shocks to a student subject. Online Lecture Video: - - Fraud: Drop outliers, bad reporting, plagiarism/theft, set up experiments with known results. Plagiarism: o Reference all material whether it is in print or in-line Human Subject Use: Today’s biological and social scientist are required to ensure the safety and dignity of human subjects. o 18th Century: Edward Jenner: smallpox vaccinations on son and neighborhood children o 20th Century: mustard gas experiment on Irish / Indian soldiers, WW2 Nazi experiments. Nuremburg Code: Demonstrate that research has a purpose, avoid all unnecessary harm, inform subject of potential harm, and subject must volunteer, be provide informed consent, and be empowered to stop the experiment. (The Code is not Law, just a recommendation) - The 1979 Belmont Report: o Autonomy: Respect for persons – each person is given respect, time, and opportunity to make a decision o Beneficence: Obligation of the researcher to protect the subject and to maximize their benefit from the research o Justice: Risk vs. Benefits. PAF 501 - Chapter 5: Research Design Research Design Comprehensive plan for data collection in an empirical research project. It is the “blueprint” for empirical research aimed at answering specific research questions or testing specific hypotheses, and must specify at least three processes 1) The data collection process 2) The instrument development process 3) The sampling process Positive Methods Such as laboratory experiments and survey research, are aimed at theory (or hypotheses) testing, while interpretive methods, such as action research and ethnography, are aimed at theory building. - Interpretive Methods Employ an inductive approach that starts with data and tries to derive a theory about the phenomenon of interest from the observed data. - Mixed-Mode Designs Employ a deductive approach to research, starting with a theory and testing theoretical postulates using empirical data. Conducting multiple interviews and then coming up with a theme from them. A combination of qualitative and quantitative data Quality of Research - - - - Internal Validity (Causality): Examines whether the observed change in a dependent variable is indeed caused by a corresponding change in hypothesized independent variables, and not by variables extraneous to the research context. o Causality requires (1) covariation of cause and effect, (2) temporal precedence: cause must proceed effect in time, (look at the sequence. This happens than that happens afterwards. (3) no plausible alternative explanation. External Validity (Generalizability): Refers to whether the observed associations can be generalized from the sample to the population (population validity), or to other people, organizations, contexts, or time (ecological validity). Construct Validity: Examines how well a given measurement scale is measuring the theoretical construct that it is expected to measure. o Example: Construct validity must assure that a measure of empathy is indeed measuring empathy and not compassion, which may be difficult since theses o constructs are somewhat similar In meaning. Statistical Conclusion Validity: Examines the extent to which conclusions derived using a statistical procedure is valid. (Not applicable for interpretive research designs). Controls required to assure internal validity (causality) of research designs Experimental Studies 1) Manipulation: The researcher manipulates the independent variables in one or more levels (called treatments) and compares the effect of the treatments against a control group where subject do not receive the treatment. 2) Elimination: Technique relies on eliminating extraneous variables by holding them constant across treatments, such as by restricting the study to a single gender or a single socio-economic status. 3) Inclusion Technique: The role of the extraneous variables is considered by including them in the research design and separately estimating their effects on the dependent variable. (Allows for greater generalizability, but requires larger samples). 4) Statistical Control: Extraneous variables are measured and used as covariates during the statistical testing process. 5) Randomization: Is aimed at canceling out the effects of extraneous variables through a process of random sampling, if it can be assured that these effect are of a random (non-systematic) nature. o Random Selection: Where a sample is selected randomly from a population o Random Assignment: Where subjects selected in a non-random manner are randomly assigned to treatment groups. Are intended to test cause-effect relationship (hypotheses) in a tightly controlled setting by separating that cause from the effect in time, administrating the cause to one group of subjects (the “treatment group”) but not to another group (“control group”), and observing how the mean effects vary between subjects in these two groups. - True Experimental Designs: subjects must be randomly assigned between each group. - Quasi-Experimental Designs: When random assignment between each group isn’t followed. Video Lecture Notes: - Studies: o Level of Analysis: What is the focus of the study? o Unit of Analysis: What is being measured? (people, nations, households cells, mRNA, protein, bones - Variables o Level of Measurement: Distinguishes between quantitative and qualitative variables (nominal, ordinal, interval, ratio) Nominal: Categorical data. (Male or Female) Ordinal: Order of Variables (Surveys such as Very Happy, Happy, Fine, Unhappy, and Very Unhappy) (Try to always have 5 categories Interval: Real number data. (Difference between 1 and 2, is the same as 3 and 4. Example: Temperature) Ratio: Must have a clear definition of 0. o Unit of Measurement: The specific unit such as dollars, cents, kilograms, pounds. In-Person Notes (8/30/21): - PAF 501 - Chapter 6: Measurement of Constructs Conceptualization Is the mental process by which fuzzy and imprecise constructs (concepts) and their constituent components are defined in concrete and precise terms. Unidimensional Are constructs that are expected to have a single underlying dimension. Example: person’s weight, wind speed, and probably even complex constructs like selfesteem. Multidimensional Are constructs that consist of two or more underlying dimensions. Example: If we conceptualize a person’s academic aptitude as consisting of two dimensions, such as mathematical and verbal abilities. Operationalization Occurs after a theoretical construct is defined. Refers to the process of developing indicators or items for measuring these constructs. - Constructs: Exist on the theoretical level Indicators: Exist on the empirical level Variable A combination of indicators at the empirical level. - Each indicators may have several attributes (or levels) and each attribute represents a value. Example: Gender variable has two attributes, male and female. o Values of attributes may be qualitative (non-numeric) or quantitative (numeric) Reflective Indicator Is a measure that “reflects” an underlying construct. Example: If religiosity is defined as a construct that measure how religious a person is, then attending religious service may be a reflective indicator of religiosity. (Construct causes the change in the measurement items (indicators)) Formative Indicator Is a measure that “forms” or contributes to an underlying construct. Such indicators may represent different dimensions of the construct of interest. Example: If religiosity is defined as composing of a belief dimension, a devotional dimension, and a ritual dimension, then indicators chosen to measure each of these different dimensions will be considered formative indicators. (Measurement items (indicators) cause the change in the construct) Levels of Measurement (Rating Scale) Refer to the values that an indicator can take (by says nothing about the indicators itself). Example, gender could be measured as M and F or 1 and 2. Scale Central Tendency Statistics Transformations Nominal Mode Chi-square Ordinal Median Interval Arithmetic mean, range, standard deviation Geometric mean, harmonic mean Percentile, nonparametric statistics Correlation, regression, analysis of variance One-to-One (equality) Monotonic Increasing (order) Ratio Nominal (Categorical Scale) Positive linear (affine) Coefficient of Positive variation similarities (multiplicative, logarithmic Measures categorical data. These scales use used for variables or indicators that have mutually exclusive attributes. Example: Gender (two values: male and female). Nominal scale merely offer names or labels for different attribute values. Ordinal Scales Measure rank-ordered data, such as the ranking of students in a class as first, second, third, and so forth. However, the actual or relative values of attributes or difference in attribute values cannot be assessed. For instance, ranking of students in class says nothing about the actual GPA or test scores of the students. Interval Scales Are those where the values measured are not only rank-ordered, but are also equidistant from adjacent attributes. Example, the temperature scale where that difference between 30 and 40 degree F is the same as the difference between 80 and 90 F. Allows us to examine “how much more” is one attribute when compared to another, which is not possible with nominal or ordinal scales. Ratio Scales Are those that have all the qualities of nominal, ordinal, and interval scales, and in addition, also have a “true zero” point (where the value zero implies lack or nonavailability of the underling construct. Example: A firm of size zero means that it has not employees or revenues. Binary Scales Are nominal scales consisting of binary items that assumed one of two possible values, such as yes or no. Video Lecture Notes: - Quality of Measurement: o The indicator represents the concept. - Measuring Concepts: It is usually too complex to represent a construct with only one indicator. Usually a collection of indicators to represent a concept is usually referred to as a scale or latent variable. o Example of multiple indicators making up a construct . . . Social Economic Status (Construct) 1) Indicator: Income 2) Indicator: Education 3) Indicator: Occupation - Quality of Measurement Continued - The indicator is measured properly. (All indicators include a small amount of error) Measurements should be. . . 1. Accurate 2. Unbiased 3. Reliable (Consistency) - Measurement error is random. Statistics can control measurement error. In-Person Notes (8/30/21): - Likert Scales: Strongly agree, strongly disagree, etc PAF 501 - Chapter 7: Scale Reliability and Validity Reliability Is the degree to which the measure of a construct is consistent or dependable. In other words, if we use this scale to measure the same construct multiple times, do we get pretty much the same result every time, assuming the underlying phenomenon is not changing? - Note, reliability implies consistency but not accuracy. Inter-rater Reliability (Inter-observer reliability) Is a measure of consistency between two or more independent raters (observers) of the same construct. Example: If these are two rates rating 1—observations into one of three possible categories, and their rating match from 75% of the observations, then the inter-rater reliability is 0.75. Test-retest Reliability Is a measure of consistency between two measurements (tests) of the same construct administered to the same sample at two difference points in time. The time interval between the two tests is critical Split-half Reliability Split-half reliability is a measure of consistency between two halves of a construct measure. True Score Theory This is a psychometric theory that examines how measurement works, what it measures, and what it does not measure. X (Observed Score = T (True Score) + E (Error) Measurement Errors Random Error: Is the error that can be attributed to a set of unknown and uncontrollable external factors that randomly influence some observations but not others. Adds variability (standard deviation) to the distribution but does not affect its central tendency (mean) Systematic Error: Is an error that is introduced by factors that systematically affect all observations of a construct across an entire sample in a systematic manner. Affects the central tendency but not the variability, this reduces the validity of the measurement. X = T + Er + Es Assessment of Validity PAF 501 - Chapter 8: Sampling Sampling Is the statistical process of selecting a subset of a population of interest for purposes of making observations and statistical inferences about that population. Sampling Process Population: The group you want to generalize to -> Sampling Frame: A list from where you can draw your sample -> Sample: the actual units selected for observation. Probability Sampling Is technique in which every unit in the population has a chance (non-zero probability) of being selected in the sample, and this chance can be accurately determined. Simple Random Sampling In this technique, all possible subset of the population (more accurately, of a sampling frame) are given an equal probability of being select. Systematic Sampling In this technique, the sampling frame is ordered according to some criteria and elements are selected at regular intervals through that ordered list. Stratified Sampling The sampling frame is divided into homogeneous and non-overlapping subgroups (called “strata”), and a simple random sample is drawn within each subgroup. Layered sample. Cluster Sampling If you have a population dispersed over a wide geographic region, it may not be feasible to conduct a simple random sampling of the entire population. In such case, it may be reasonable to divide the population into cluster, randomly sample a few clusters, and measure all units with that cluster. Match-pairs Sampling Sometimes research may want to compare two subgroups within one population based on a specific criterion. Multi-state sampling The probability sampling techniques described previously are all examples of single stage sampling techniques. Depending on your sampling needs, you may combine these single-stage techniques to conduct multi-state sampling. Nonprobability Sampling A sampling technique in which some units of the population have zero chance of selection or where the probability of selection cannot be accurately determined. Does not allow the estimation of sampling errors, and may be subjected to sampling bias. Convenience Sampling Also called accidental or opportunity sampling, this is a technique in which a sample is drawn from that part of the population that is close to hand, readily available, or convenient. Quota Sampling In this technique the population is segmented into mutually-exclusive subgroups )just as in stratified sampling), and then a non-random set of observations is chosen from each subgroup to meet a predefined quota. - Proportional Quote Sampling: The proportion of respondents in each subgroup should match that of the population. Non-proportional Quote Sampling: IS less restrictive in that you don’t have to achieve a proportional representation, but perhaps meet a minimum size in each subgroup. Expert Sampling This is a technique where respondents are chosen in a non-random manner based on their expertise on the phenomenon being studied. Snowballing Sampling You start by identifying a few respondents that match the criteria for inclusion in your study, and then ask them to recommend others they know who also meet your selection criteria. Response Is a measurement value provided by a sampled unit Frequency Distribution Graphed responses from difference respondents to the same item or observation Normal Distribution Large number of responses in a sample where the frequency distribution tends to resemble a bell-shaped curve. Can be used to estimate overall characteristics of the entire sample, such as sample mean or standard deviation. These estimates are called sample statistics. Population Parameters Since the entire population can never be sampled, population characteristics are always unknown and called this Sampling Error Differences between Sample Statistics and Population Parameters. Standard Error The variability or spread of sample statistics in a sampling distribution. (The standard deviation of a sampling statistic). Confidence Interval Is the estimated probability that a population parameter lies within a specific interval of sample statistic values. All normal distributions tend to follow a 68-9599 precent rule, which says that over 68% of the cases in the distribution lie within on standard deviation of the mean, over 95% of the cases in the distribution lie within two standard deviations of the mean value, over 99% of the cases in the distribution lie within three standard deviations of the mean value. Video Lecture Notes: What is Sampling? - Theory of Sampling: To represent a population of interest as a smaller cohort (or grouping) of individuals within a research experiment to test a hypothesis. - Sampling: Demonstrating Causation - Inference: To draw conclusion about a population from a sample o How do we do this? Through statistics. Types of Sampling: - - Random Sampling (Probability) o Simple Random Sampling Every potential subject has an equal chance of being selected Drawing numbers out of hat Random number generator Advantages Findings are very generalizable o Systemic Random Sampling o Stratified Random Sampling o Proportionate Random Sampling o Disproportionate o Cluster o Multi-stage sampling o Catch / re-catch Non-probability Sampling (Not Random) o Purposive Sampling o Accidental, Haphazard o Quota o Snowball sampling Steps for Picking a Sample: PAF 501 - Chapter 10: Experimental Research Experimental Research Often considered the gold standard in research designs. In this design, one of more independent variables are manipulated by the researcher (as treatments), subjects are randomly assigned to different treatment levels (random assignment), and the results of the treatments on outcomes (dependent variables) are observed. - Internal Validity: The unique strength of experimental research is its internal validity (causality) due to its ability to link cause and effect through treatment manipulation, while controlling for the spurious effect of extraneous variable. Laboratory Experiments Conducted in a lab (artificial) setting tend to be high in internal validity, but this comes at the cost of low external validity (generalizability) because the artificial lab setting in which the study is conducted may not reflect the real world Field Experiments Conducted in field settings such as in a real organization, and high in both internal and external validity. These studies are rare because of the difficulties associated with manipulating treatments and controlling for extraneous effects in a field setting. Treatment / Control Groups In order to test the effects of a new drug intended to treat a certain medical condition like dementia, if a sample of dementia patients is randomly divided into three groups, with the first group receiving a high dosage of the drug, the second group receiving a low dosage, and the third group receives a placebo such as a sugar pill (control group), the first two groups are experimental groups, and the third group is a control group. Treatment Manipulation Pretest Measures: Conducted before the treatment is administered Random Selection vs. Random Assignment Random Selection: Is the process of randomly drawing a sample from a population or sampling frame. (Usually employed in survey research because it assures each unit in the population has a positive chance of being selected into the sample). High external validity (generalizability). Posttest Measures: Conducted after the treatment is administered Random Assignment: Is a process in true experimental research to ensure that treatment groups are similar (equivalent) to each other and to the control group, prior to treatment administration. High internal validity. Threats to Internal Validity - - History Threat: A historic event, not related to the experiment mat influence outcomes Maturation Threat: Duration or aging can affect a post-test or outcome - Maturity - Boredom - Strength Testing Threat: A pre-test can affect a post-test (sensitive to issue to a subject response to please the experimenter) Instrumentation Threat: Reliability (or appropriateness) of the tool or qualification of the person used to evaluate the outcome Mortality Threat: Sample reduction that may not be independent from the dependent variable (outcome) - Not actual, mortality, this refers to drop out rate for human subjects, but can also refer to “rodent” death in animal experiments. - Reflects the same idea as attrition, but is considered a group threat because it reveals that the two “groups” might have been different in ways not recognized. Attrition can occur for studies with only one group as some participants leave over time. - Regression Threat: Regression toward the mean (sample chosen for extreme scores – there is nowhere to go but away from the extreme) In-person: time and group effects Discussion post is done on this – find an actual study that control or didn’t control for these threat on internal validity Two Group Experiments: Pretest-Posttest Control Group Designs Two Group Experiments: Subjects are randomly assigned to treatment and control groups, subjected to an initial (pretest) measurement of dependent variables of interests, the treatment group is administered a treatment (representing the independent variable of interest), and the dependent variables measured again (posttest). The treatments effect is measured as the difference in the posttest and pretest scores between the treatment and control groups. This design is simpler version of the pretest-posttest design where pretest measurements are omitted. The treatments effect is measured simply as the difference in the posttest scores between the two groups. Posttest Only Control Group Design Two Group Experiments: Covariance Designs Sometimes, measures of dependent variables may by influenced by extraneous variables called covariates. Covariates are those variables that are not of central interest to an experimental study, buy should nevertheless by controlled in an experimental design in order to eliminate their potential effect on the dependent variables. Covariance Design is a special type of pretest posttest control group design where the pretest measure is essentially a measurement of the variants of interest rather than that of the dependent variables. This effect is measured as the difference in the posttest scores between the treatment and the control group. Factorial Designs Enable the researcher to examine not only the individual effect of each treatment on the dependent variables (called main effects), but also their joint effect (called interaction effects). - - Main Effect: Is said to exist if the dependent variable shows a significant difference between multiple levels of on factors, at all levels of other factors. Interaction Effect: Exists when the effect of differences in one factor depends upon the level of a second factor. Hybrid Experimental Designs: This is a variation of the posttest-only or pretest-[posttest control group design where the subject population can be grouped intro relatively homogeneous subgroups (called blocks) within which the experiment is replicated. Randomized Block Design Hybrid Experimental Designs: Solomon Four-Group Design Hybrid Experimental Designs: Switch Replication Design In this design, the sample is divided into two treatment groups and two control groups. One treatment group and one control group receive the pretest and the other two groups do not. This design represents a combination of post-test-only and pretest-posttest control group designs, and is intended to test for potential biasing effect of pretest measurement on posttest measures that tend to occur in pretest-posttest designs but not in posttest only designs. Two-group design implemented in two phases with three waves of measurement. The treatment group in the first phase serves as the control group in the second phases, and the control group in the first phase serves because the treatment group in the second phases. Video Lecture Notes: Research Continuum - Level of Analysis: What are you studying? - Macro Whole System (In Vivo), Midlevel (In Situ), Micro Control (In Vitro) Video Lecture Notes: Causality - - - - - - Slide 1 - Ideas are theories only when you can test them to be falsifiable We do this testing through a hypothesis 1. Test for hypothesis should be able to be replicated Slide 2: - Theories are statements about relationships between concepts or constructions. A statement about a cause-and-effect association between variables. They are a statement about an association between variables: + or – Remember: a theory is about an association not necessarily a cause. 1. A theory can be tested repeatedly 2. Evidence can accept or refute a hypothesis about an association which strengthens or weakens a theory. Slide 3: - Variables: may be modifiable (weight, calorie intake) or non-modifiable (race, ethnic origin) - Constant the opposite of a variable, these stay the same. - Variables can have multiple groups and in those groups you can have different constants Slide 4: - Independent Variable: Measure of the cause This variable, condition, or “thing” that is manipulated (or changed) by the research in an experiment - Dependent Variable: Measure of the effect This is the variable that is observed (for changes) following change or manipulations in the independent variable. Slide 5 - How do we show causation? (Demonstrating Causation) Co-variation (must be correlated – either + or -) Cause prior to effect Absence of a plausible rival hypothesis Slide 6: - Covariation: A change in one variable (X – manipulation/treatment – what your change) corresponds with a change in another variable (y – outcome/observation) - - Slide 7: - Cause and Effect Names Slide 8: - Reserve Causality - Slide 9: - Pretest-Posttest – Insuring Cause Happens Before Effect - - Slide 10: - Plausible Rival Hypothesis (Empirical Example) Slide 11: - Hypothesis Testing: Operationalization of the cause and effect Video Lecture Notes: The Experimental Method (Random Control Trials) - Slide 1: - Whether a “study” is an experiment depends on its “design” “Design” implies methods including group set-up, required techniques/strategies, and analysis. - Slide 2: - How to demonstrate cause in Random Control Trial Experiment - Slide 3: - Random Control Trials Random Assignment vs Random Sampling Random Assignment vs Matching - - 1. Matching can be influenced by your bias Slide 4: - Elements of a Random Control Trial Slide 5: - Pretest -> Posttest - - - Slide 6: - RCT Design Graphic Slide 7: - RCTS and Causality Slide 8: - Downside of RCT Dealing with group threats through random assignment and time threats through manipulations/treatment Contamination 1. a. Expectancy: The experimenter might expect something and influence the results 1) Fix this by utilizing a blind experiment. Too Focused – control is achieved through the sacrifice of realism - 1. Slide 9: Design Notation - Slide 10: Notation - Notation Continued - Slide 11: The goal of hypothesis testing - In-Person Notes: - Lab experiments (high internal validity, lower external validity/generalizability), field experiments high in both internal and external validity, treatment/control groups, pretest/posttest measures, random select Inferential Statistics Video Lecture - Inferential Statistics: provide a means to generalize the data that you do not have - Underlying Assumption: your sample in unbiased The specifics differ for each statistic - p is the probability that your results are “real” and not by accident (show representative the sample is of the population) - Distributions - A distribution is a description of frequency of occurrences of events, for a range of events. - Histograms, box plots, stem and leaf plot are methods for visualizing empirical distributions - “Normal” distributions are balanced, and the mean=median=mode - Making Apples into Oranges - Means and standard deviations are expressed in original units - Z scores are a conversion to “standard” units representing values as a function of their distance from the normalized mean of zero - Z-Test - Z tests are the first inferential statistical tests taught - Z tests are used to compare a sample to a population with a known mean (or proportion) and variance (standard deviation) - Few studies use Z tests, but they are useful for introducing hypothesis testing, because the Z / Normal / Gaussian distributions have nice properties. Useful to illustrate how to determine if a sample value is significantly different than the population value Depends only on “value”, spread, and sample size - Hypothesis for T-Test - General Null = There is NO difference between the groups (Equal to) - General Alternative = There is a (significant) difference between groups (Not Equal to) - T-Test Comparing Means Assumptions - Distributions are normal - Variances are equal - Random and Independent Samples - Hypothesis Testing - Hypothesis testing is about rejecting the null hypothesis - If your inferential statistics indicate that you can have confidence in rejecting the null (your p value is <.05) your result is significant. a is the probability that your rejection of the null is false If you do reject the null hypothesis when you should not, you are making a Type 1 (a) error. Setting your criteria to alpha 0.01 makes it less likely you will accidently publish finding due to chance. - There is also a risk that you might falsely retain the null hypothesis THis occurs when your data analysis result are not statistically significant, but the alternative hypothesis is true 1. This is called a Type II (B) error Types of Error - - - Sample Size and Power - As a sample size increase, it becomes easier to correctly reject a null hypothesis - Power: Is the probability of correctly rejecting the null hypothesis when the alternative hypothesis is true - If a researcher can reject the null, the he/she can conclude that the results generalize from the sample to the population and can publish a paper Steps to perform a T-test - - fc - Standard Error - Standard Error is the standard deviation of the sampling distribution How likely you are to be wrong PAF 501 - Chapter 11: Case Research Case Research Intensively studying a phenomenon over time within its natural setting. Strengths: - - Can be theory testing or theory building Research questions can be modified during the research process if the original questions are found to be less relevant Can help drive richer, more contextualized, and more authentic interpretation of the phenomenon of interest than most other research methods. The phenomenon of interest can be studied from the perspectives of multiple participants and using multiple level of analysis (individual and organizational) Weaknesses: - - Key Decisions in Case Research Interpretive Case Research - No experimental control, internal validity of inferences remain week. Quality of inferences derived from case research depends heavily on the integrative power of the researcher. (Expert notices pattens vs novice miss them) Because the inferences are heavily contextualized, it may be difficult to generalize inferences from case research to other contexts or other organizations. Is the right method for the research questions being studied? What is the appropriate unit of analysis for a case research study? Should the researcher employ a single-case or multiple-case design? What sites should be chosen for case research? What techniques of data collection should be used in case research? In an inductive technique where evidence collected from one or more case sites is systematically analyzed and synthesized to allow concepts and patterns to emerge for the purpose of building new theories or expanding existing ones. Conducting Case Research - Define research question Select case sites Create instruments and protocols Select respondents Start data collection Conduct within-case data analysis Conduct cross-case analysis Build and test hypotheses Write case research report PAF 501 - Chapter 13: Qualitative Analysis Qualitative Analysis Heavily dependent on the researcher’s analytic and integrative skills and personal knowledge of the social context where the data is collected. The emphasis in qualitative analysis is “sense making” or understanding a phenomenon, rather than predicting or explaining. Grounded Theory An inductive technique of interpreting recorded data about a social phenomenon to build theories about that phenomenon. Coding Techniques for Analyzing Text Data - - - Integration Techniques - Open Coding: is a process aimed at identifying concepts or key ideas that are hidden within textual data, which are potentially related to the phenomenon of interest. - Categories: Tend to be broad and generalizable, and ultimately evolve into constructs in a ground theory. Categories are needed to reduce the amount of concepts the researcher must work with and to build a “big picture” of the issues salient to understanding a social phenomenon. Axial Coding: The categories and subcategories are assembled into causal relationship or hypotheses that can tentatively explain the phenomenon of interest. (little spheres of coded concepts and here you are drawing bigger circles around that). Selective Coding: Which involves identifying a central category or a core variable and systematically and logically relating this central category to other categories. (Looking for central organized principle to explain the phenomenon and linking it to other categories). Storylining: Categories and relationships are used to explicate and/or refine a story of the observed phenomenon. - - Memoing: Is the process of using these memos to discover patterns and relationships between categories using two-by-two tables, diagrams or figures. Concept Mapping: Is a graphical representation of concepts and relationships between those concepts (Using boxes and arrows). In-Person Notes: 9/27/21 - Theoretical Saturation: Coding/data refinement until when additional data does not yield marginal change in core categories/relationships. - Bernard (Participation Observation): - Bernard: (Direct / Indirect Observation): Direct (watching people on the spot) and indirect (archeology of human behavior) - roundedtive/nonreactive: - Continuous monitoring/spot sampling - Unobtrusive observations Video Lecture Notes: Unobtrusive Methods - Unobtrusive Methods: Studies that do not “inconvenience” the subject matter - Do not take them out of their element - Do not place a burden - Do not put subjects at risk (except perhaps a risk of loss of anonymity) - Researchers study them in their natural environment - Problems with Obtrusive Studies: - People change when they know someone is watching - Common Problems with Obtrusive Studies - Reactivity Hawthorne Effect 1. Western Electric Company: If changing the lights in the factory improve productivity, they found out that it did but rather than the lights causing the change it was found that productivity increased because the observers were watching the people. Response set: People responding and behaving as they think the way the researchers would want Placebo effect - The Observer’s Paradox Heisenberg’s Uncertainty Principle 1. It is impossible to measure systems (in quantum physics) without affecting those systems 2. Position and momentum 3. Schrodinger’s Cat - When you’re conducting researcher and directly observing them, you are changing the environment you are studying. Unobtrusive versus Qualitative Observation is unobtrusive. Unobtrusive is a mix of Qualitative and Quantitative Ethnography - - Ethology: Scientific and objective study of animal behavior (under natural conditions) - Jane Goodall (Chimps) - Dian Fossey (Gorillas) Ethnography: Systematic study of people and culture (write and “people”) - Critical versus Realist (purpose) - You can ask questions - Trust building Participant Observation: - Complete emersion in new social setting: fly on the wall. - Completely immersive Observation (behavioral) Open ended Exploratory (not confirmatory) Time consuming All consuming Chance of “going native” - So immersed that you can’t be objective Unstructured does not equal Undisciplined - Researcher must pay close attention and take copious notes Must refrain from bringing in preconceptions - Must refrain from becoming too sympathetic Unstructured does not equal Without Standards - Inductive does not mean anything goes: - Good work conforms to strict standards: Gaining access / Rapport 1. Key informants 2. Gatekeepers Explicit Awareness (mindfulness) Building memory Naivety and Objectivity (value neutrality) - Field notes Jotting / diary / log / formal field notes Theoretical Approach - Inductive (usually) Grounded Theory - Also a “re” search but not for falsifying based on evidence but for gaining additional, deeper and broader knowledge - “Discover new worlds, boldly go where no one has gone before” See questions from new perspectives Ask different questions - Use the new data to bootstrap the theory Completely inductive ^ Theoretical Approach Continued - - Coding Evaluation Standards: Internal Validity External Validity (Generalizability) Reliability Objectivity Creditable / Richness of Data / Free from bias Transferability / Indicator of elements of the population to know limit Dependable / Reliable insofar as context is constant Theoretical Orientation / Detail / Confirmability Caution - Analytical Induction - Includes a search for falsifying evidence - Flawed approach: no example can either prove or disprove a rule - Fallacy of Composition (exception of composition): Thinking you can make generalized statements just because you studied something or someone. (Individual -> Group) Ecological Fallacy: You know on average what a group does something so you general that to an individual. (Group -> Individual) - Finding out Why - When studying humans we cannot only investigate what they do, behavior, we can find out WHY Researchers can ask people questions about - Motivation - Attitudes - Feelings - Methods to find out why are almost always obtrusive Content Analysis - - - Two processes: - Specification - Application of rules Level / Units of analysis Recording Unit (word, terms, themes, characters, paragraphs, items) - Theme is a sentence (subject / predicate) - Items can be books, articles. - Semiotic focuses on symbols other than “words” Context Unit - Context in which recording unit is found - What does the word mean in the sentence Content Analysis - - - Common categories - Who - What - Where - When - How Manifest vs Latent - Qualitative - Quantitative (Manifest – counting the number of times a certain word appears) Inductive or Deductive Requires - Coding (themes) - Grouping (frames) - Linking to theory Archives - Government records Commercial records Any recorded artifact at any level Data bit and papers Level and Unit of Analysis matter Traces: - Methods used to studying animals can be used to study humans Physical traces of human activity - Where people go (where and wear) Museum exhibits Where to pace - What people eat - What people waste Conclusion: - Reactivity makes it difficult to study people when they are aware they are being studied People behave differently when not in their natural environment To find out what people REALLY do, it is important to be able to observe / record / infer what they really do - Not just ask - Not just assume that what we see is all there is Bernard (RM – Participant Observation): - fieldwork: complete participant/participant observer/complete observer; going native; rapid assessments; validity: impact on researcher on phenomenon (want to minimize reactivity); hanging out/rapport; practice objectivity; complete immersion: becoming the phenomenon; objectivity does not mean value neutrality; indigenous research: objectivity challenge; initial contact/culture shock; mid-fieldwork break important to get distance/objectivity Bernard (RM – Direct and Indirect Observation): - direct (watching people on the spot) and indirect (archeology of human behavior) observation; reactive/nonreactive; continuous monitoring/spot sampling; unobtrusive observations; indirect observations (trace/archival research); Bernard (RM – Text Analysis): - Unobtrusive methods Lecture Slides: unobtrusive (collection methodology) but qualitative/quantitative is the analysis technique; ethology (animal study in natural conditions) & ethnography (systematic study of people/culture), participant observation (become part of the group and observe from the inside and their perspective); unstructured not equal to undisciplined (notes, data) or standards (access/mindfulness/naivety & objectivity; grounded theory (use data inductively to create new theory); inductive grounded theory: code field notes, code content (software), text to themes/themes to theory/validate models; evaluation standards: internal (free from bias)/external (generalizability) validity, reliability, objectivity; caution in generalizing to larger groups; ecological fallacy: assuming individual characteristics based on general population characteristics; fallacy of composition: generalizing to larger groups based on smaller group characteristics; content analysis (who, what, where, when, how): level/unit of analysis, recording unit (word, term, theme, character, paragraph, item), manifest (quantitative, i.e. word frequencies) and latent (qualitative); archived data of any type is unobtrusive data gathering; traces: physical traces of human activity; reactivity: difficult to study people when they are aware of it (reactive); Statistic Videos RSM4 Bivariate Objects of Measurement - Level of Measurement: Distinguishes between quantitative and qualitative variables (nominal, ordinal, interval, ratio) Unit Measurement: The specific unit such as dollar, cents, kg, or ibs, etc. Unit of Analysis: What you are measuring, such as people, nations, households, cells, mRNA, protein, bones. Level of Analysis: The comparison is at what level: individual, city, national. Levels of Measurement: - Nominal Data: - Binary Data: - Any data can be converted into a binary variable: - Interval ratio data -> Threshold - Nominal -> 2 groups - Binary “nominal” data can be used in quantitative analysis coded as 0 or 1 This is sometimes referred to as “dummy coded” Ordinal Data: - - Order of variables is important but not the difference between variables - Utilizes Likert Scales Ex. Very happy, unhappy, fine, happy, very happy Survey creation note: The fact that we have an odd number of possible answer helps our post-hoc analyses to interpret on the central tendency (mean, median, mode) easier/better than with an even number of answers. Statistics Matter - Quality vs. Quantity - Univariates: one variable - Bivariate: two variables Correlation Regression Cross tabulations (contingency tables) - Distributions - Probability (Inferential Statistics Causation Review: - Requires three things - Correlation / Covariation Covariation: 1. When X goes up, Y goes up (positive correlation). 2. When X goes down, Y goes down (positive correlation) 3. When X goes up, Y goes down (negative correlation) - Cause prior to effect - Absence of a plausible rival hypothesis Bi-Variables: Nominal or Ordinal - - Cross Tabular Data - 2-way frequencies - Contingency tables - Bivariate analysis of categorical data Survey Analysis - Quantification -> crosstabs - Are 2 variables related? Does knowing the first provide you with information about the second? Bivariates: Interval - Variance -> co-variance How do you compute variance and what does it mena? - (X – mean deviation from the mean) Correlation Is the standardized covariances - Covariation: Is a measure of the degree two variables vary relative to one another, but the two variables can be at different scales - Correlation: Provides a statistic of covariance for the “standardized” variables. - Standardizing: Converts both variables to the same scale (Z scales) To standardize: 1. Divide each deviation by the variable’s standard deviation 2. Then sum and divide by n-1 Regression: - IF a relationship is linear (as one variables increases so does the second) it can be represented as a line. Review - 3 types of bivariates - Crosstabs Used for nominal or ordinal Two-way frequency/contingency table - Correlation (Covariation) Used for interval or ratio - Regression 1. Shows not only how two variables correlate or covary but also slope. (where that zero value sits) Slope Intercept - PAF 501 – Bernard Chapter 9: Unstructured and Semistructured Informal Interviewing Categorized by a total lack of structure or control. The researcher tries to remember conversations heard during the course of a day in the field. This requires constant jotting and daily sessions in which you sit at a computer, typing away, unburdening your memory, and developing field notes. Unstructured Interviewing There is nothing at all informal about unstructured interviewing and nothing deceptive, either. You sit down with another person and hold an interview. Period. Both of you know what you’re doing and there is no shared feeling tht you’re just engaged in pleasant chitchat. Used when you have lots and lots of time – like when you are doing long-term fieldwork It is used in studies that requires on textual data and in studies that requires both textual and numerical data. This methodology is great for lived experiences (what was it like to survive handto-hand combat, etc) Semistructured Interviewing It has much of the freewheeling quality of unstructured interviewing, and requires all the same skills but semi-structured interviewing is based on the use of an interview guide. - Interview guide: Is a written list of questions and topics that need to be covered in a particular order. The best method in situations where you won’t get more than one chance to interview someone. Also, semistructured interviewing works very well in projects where you are dealing with high-level bureaucrats and elite members of a community. Structured Interviewing People are asked to respond to as nearly identical a set of stimuli as possible. One variety of structured interviews involves use of an interview schedule. - Interview Schedule: An explicit set of instructions to interviewers who administer questionnaires orally. Video Lecture: RM-05 Interviews Finding Out Why - - Obsevation is a great way to discover WHAT people do . . . When studying humans we cannot only investigate what they do (behavior) we can find out WHY. Researchers can ask people QUESTIONS about - Motivation - Attitudes - Feelings Methods to find out why are almost always obtrusive - Although it can be a part of an unobtrusive method like participant observation Way To Ask Questions - There are many techniques for asking questions of respondents to learn about their opinions, attitudes, and behavior. - What sets these techniques apart is how structured they are. 1. 2. 3. All of these methods requires the researcher attempt to stay “neutral” and objective as possible An “interview guide” directs the topics and questions “Interview schedules” are instructions regarding what questions to ask \ The Many Level of Measurement - Researchers have several options for how to ask questions varying in: - The level of “structure” - Whether the focus is on individuals or groups Context - Validity - Measurement: - Ethics - - - For the most part asking questions is - Non-invasive - Obvious (so there is no deception) So IRB’s provide Exempt or Expedited approvals - The only real concern -> confidentiality This is especially true when the objective of the study is to discover information about personal, illegal, or non-normative attitudes or behaviors Still requires consent of respondent Standard (expectations for good studies) - Asking questions well requires researchers - Establish rapport (but not the degree of introducing bias) - Record responses - Do not lead, or inject yourself into, the conversation Listen and probe to elicit a more complete response Stay as neutral as possible to avoid biases responses Stay as objective as possible to avoid biasing the interview. Qualitative Evaluation Standards - Internal, external, reliability, objectivity. More on Threats to Validity - - Respondent accuracy - Memory (people forget) Techniques to jog memory are useful to reduce this Interaction introduced biases (Reactivity . . . again) - Characteristics or presence of the interviewer inhibit the responses Response effects 1. Respondent want to please the interviewer (response set bias) 2. Deference effect (respondent says what he/she think you want to hear) 3. Social desirability effect (people want to look good so they lie or exaggerate positive and hide negatives). Expectancy effects (Expectations of the interviews “lead” responses} A survey is a carefully designed set of questions. Good survey design requires the researcher get 3 things right: - Ask questions in a manner that does not threaten validity. Organize the questions in a manner that does not threaten validity. Administer the sample in a way that does not threaten validity. Conclusion - When studying people asking questions can provide useful information HOWEVER - Regardless of level of structure, there techniques are vulnerable to error Respondent may not be truthful for a variety of reasons - They do not recall or remember accurately - They want to present an image of themselves that is not accurate - The more naturalistic studies are likely to be high on ecological validity, can be high or low on internal validity, but may not be generalizable The more structured interviews (surveys) are likely to be generalizable (if the sample was properly obtained) but ecological validity is low. Video Lecture: RM-05 Survey Research 1 - Why Survery - Contrasted with unstructured or semi-structured interviews, surveys provide a way to “standardize” asking questions Mainly confirmatory (deductive) Rigid in structure 1. Time 2. Questions - Easily replicated - Easier to analyze - Survey Pros - Used to collect data on the feeling, perceptions and motivations of people - Prepared and rehearsed in advance - Respondent options are limited - Can focus on the topic of interest directly - While “obtrusive” not usually too bad (e.g. questions are not usually personally invasive) - Easy to get a good (random) sample - If well-constructed amenable to statistical analysis - Informed consent - Survey Cons - Not as sensitive to the broad range of feeling as more opened ended approaches are. - Limited by the preconceptions of designers Respondent option are limited (con as well as pro) Can miss important elements of a topic Types of Surveys - Longitudinal - Panel - Same respondents many times Trend - Different respondents many times Cohort - Trend study (different respondents) focusing on change during the life course Required to separate the effect of age, cohort and time. Cohort vs. Age - - Everybody is 18 once - Is a characteristic the result of “age” or the shared life experience that occurred to the group that were born a certain year Liberal youth in the 1960s - Was it their age? (Are you 18 year olds liberal?) - Was it their cohort (Are people born in 1948 liberal when they are 18 and 50?) - Is it a combination of age and cohort? Video Lecture: RM-05 Survey Research 2 Why Survey? - - Contrasted with unstructured or semi structured interviews surveys provide a way to “standardize” asking questions - Mainly confirmatory (deductive) - Rigid in structure Time Questions Easily replicated Easier to analyze Survey Challenges - - All types of surveys: - Completion rates (surveys not completed) - Refusal rates (survey’s refused by some) May or may not be random (introduce bias) - People lie / people forget / people misunderstand Response set Response style Panel Studies have additional issues - Attrition (People drop out) - Pretest sensitization (people learn how to answer). Threats to Good Surveys - - Biases - Sample bias - Collection bias - Respondent / Subject bias Question wording - Bad questions (garbage in / garbage out) Survey order Sample Bias - Sampling procedure is not representative - Only the rich, only the sick, only the ones who return the survey, etc. Over-sampling is sometimes important - To get information on underrepresented groups - Or because the researcher is only interested in that one group Data Collection Bias - 2 Types - Observation Bias: What you ask 1. How the questions are chosen / worded / ordered How you collect the data (phone vs face to face / interviewer characteristics) - Non-observation bias Who / what you DON’T ask = what you miss Subject Bias - - Response Style - Measures the style of the subject rather than the construct of interest Some people will never give anything a 10 out of 10 Other always exaggerate the positive 1. Such extreme answers can be corrected for with statistics. Response Set - Social desirability bias - What is “acceptable” rather than what is a true response - Answers seek to please the investigator or fall within societal norms. 2 Main Challenges - Challenge 1: Getting the questions right - Be clear and unambiguous: short, simple, to the point Define “difficult” or ambiguous terms (don’t assume a mutual understanding) Avoid double barrel items (ask one question at a time Favor closed ended questions with mutually exclusive and exhaustive categories - Avoid biasing answers Avoid leading questions Provide unbiased response option Be cautious of Order Effects - Ask questions so that they can be quantified Seven Deadly Sins 1) Leading words / questions 1. Example: Would you agree with responsible parents that car seats should be required for infants? 2) No “opt out” option (prefer not to answer) 1. What are your religious beliefs? (Too personal). 3) Ambiguous: Not asking direct questions 1. What should government do to improve commute time? 4) Not giving mutually exclusive choices 5) Failing to provide an exhaustive set of options 6) Unbalanced (biased) scales 7) Double (triple) barreled questions - Challenge 2: Getting the question order right - Vetted Social Science Scale Types - Scales refer to “multiple indicators” to describe a construct - Types include Binary 1. Capturing attitudes as binary a. Yes / No (1 / 0) 2. Multiple indicators can be added together (summed) to create a quantitative measure Likert 1. A likert scale is the aggregation of many likert items 2. A likert item expand questions (items) to a 3-9 (odd number) point scale a. Each question is composed of a “stem” with corresponding anchor b. Anchors are coded as ascending or descending numbers - - c. 3. Is a summative scale Semantic Differential 1. Multiple pair of polar opposites 2. Respondent is asked to select where they are on the continuum 3. Same anchor, different types of contrasts 4. Summative Guttman 1. Cumulative scale of tolerances 2. Assumptions: Preferences on item are ordinal a. If you agree with on you will agree with all that follow b. Listing of “tolerances” from least to most extreme c. Value of first “yes” indicates attitude of tolerance 3. Example: Bogardus Social Distance scale Thurstone Bogardus 2 element scales Question structure Scaling -> Consolidation of multiple questions This is a means for quantifying qualitative data. Semantic Differential - Multiple pairs of polar opposites presented Respondent is asked to select where they are on the continuum Same anchor, different types of contrasts Summative - Scale Validation (Thurston Scale) - There are many ways to validate scales, these include - Criteria validity (comparison with a validated measure) - Correlation with onther similar concepts - Expect Judges Example: Thurstone 11 point scale construction 1. Clear conceptual definition 2. Generation of 80-100 candidate questions 3. Panel of judges rank questions 1 (poor) to 11 (good) depending on closeness to concept 4. Statistical analysis of judges ranks result in 11 point scale 4 main types of Validity - - - Face: common sense - Does the question make sense using common sense Content: used mainly when questions wording is the concern – subjective judgement of content - Used mainly when question wording is the concern – need to ensure that all the words being used in a survey (content) mean the same to all people being survery - Deals with question design Criterion: Tested against a standard - Ask a new question (e.g. variation on a theme) that can be compared to a standard (other questions or physical attribute) that has been already and deemed to be a valid standard. Construct: Closeness to target (statistical) - Relation among items on test - Relation of the test to other supposed measures Like criterion validity, but neither measure is assumed to be a “standard” - Relation to indictors of other construct of theory. - Idea the cannot be measured with one variable so construct validity is a series of statistical test that determine how close your question(s) are to the idea you are measure (sometimes referred to as “closeness to target”). Question Ordering - Not as well studied as question wording the order of questions matters as well - Framing - Prompting - Be aware of the order of questions and how the order of topics might influence how subjects respond. 10/18/21: In-Person Notes . 10/29/21: Research Deigns Revisited Study Design: - Fancy techniques in a laboratory do not constitute an experiment What you can conclude from a “study” depends on its “design” “Design” implies methods, which includes, sampling, group set-up, required techniques/strategies, and analyses. Causation - X -> Y - Three factors to demonstrate causation Co-variation Cause prior to effect Absence of a plausible rival hypothesis. Random Control Trials - - One of the ways to demonstrate causality - Random: Assignment/distribution of samples you are testing - Control: Interventions/treatments - Trials: Repeat trials (not just an n=1 for each group) Elements of the RTC - Random Assignment of 9 into groups Control vs. Treatment (Experimental Group) - Pre-test: Pre-treatment measurements between groups Both groups should be indistinguishable Ensures cause (treatment) precedes the effect (outcome) Provide time-specific evidence that the effect occurred after the treatment No difference between pretest groups is further evidence that nothing else differs between the groups. - Explicit treatment given to only one group The introduction of a treatment, controlling for everything else, ensures that the treatment is the reason for the difference between groups - Post-test: Measurable difference of outcome / effect between treatment and control groups following the treatment. True Experimental Design (RCT) - Group 1: No Y -> Treatment X -> Y Group 2: No Y -> _____________ -> No Y Components of an Experiment - R = Random Assignment X = Exposure to a Treatment or Event O = Observation Column = Time ordered from left to right; two events in the same column happen at the same time Row = Experimental Unit or Subject Experimental Designs: Post-Tests - - Random Control Trial: (O = O . . . Null Hypothesis) (O does not equal O . . . Alternative Hypothesis) - ROXO - RO_O One Shot (Least experimental design) - XO Designs: Main Types - - - One Shot: Correlation. Measures dependent variable. (post-hoc analysis) - Not control - Not treatment - No pre-test One Group: Look for changes within each subject - Manipulate dependent variable - Pre-test / post-test - Advantages: Control of event O measured before and after event - Disadvantages: O may have occurred anyway Something other than X may have occurred to generate O Static Group: Look for change between difference subjects or groups - The difference between individuals or groups might be the result of something other than what is being studied (plausible rival hypothesis). - Advantages: Comparison of the presence and absence of X - Disadvantages: - Differences not measured may have accounted for O Differential recruitment or mortality) Experimental (RCT). - Advantages: Control, comparison, pretest/posttest, random assignment Called the Solomon 4 Quasi-Experiment (NEGD) (many policy questions use non-equilivinant group designs (quasi)) - NOXO NO_O Interrupted Time Series - OXO...OX...O Downside to RTC - - Experimenter expectancy - Self-fulfilling prophesy Solved by having a blind experimenter - Demand characteristics (response set) Treatment group demonstrates and effect that is not real (placebo) Control group is impacted by experiment in unanticipated ways 1. Compensation 2. Exaggerated contamination RTC focuses on “control” at the expense of realism in many ways - IT is always important to ask if A is Necessary or Sufficient to cause B 10/29/21: Figures - Figures - Highlight and must be referred to by the text. They are useful if . . . Only a picture can tell the story There is too much information to include in a table What is subtle in a table is obvious in a picture - Things to consider to include: Volume of data you are presenting Time intervals Qualitative or Quantitative data Comparisons Raw vs. summarized - In the end ask yourself Does the table/graph help the reader? Does it help you (the writer) - Things to consider for your figure - Do no duplicate the information from a Table - Size . . . make sure it fits well in the paper - Color 10/29/21: Table - Tables - Highlight and must be referred to by the test. They are useful if: There is too much information to include in a paragraph There are too many dimensions to a concept to describe clearly in a paragraph - Thing to consider include: Volume of data you are presenting Time intervals or continuous (depends on your experiment) Space Qualitative or Quantitative Comparisons Raw vs. summarized In the City Lights and American Red Cross Nutrition Behavior Study researchers recognized a shared issue with their sample. What was the main, common concern? - Baseline (N = 1461) Posttest (N=1031) Another name for a Pretest-Posttest Comparison Group Design is: - Both RTC and Non-equivalent group design can be described as pretest-posttest What is the main difference between an experiment and a quasi-experiment? - Doesn’t sampling fall under assignment procedures? So it could be either or. Hey Professor, I hope you’re having a good weekend so far! I’m reaching out this morning to hopefully get some clarification on a few questions I missed on this week’s quiz. 1) Another name for a Pre-test-Posttest Comparison Group Design is: - My Answer: Random Control Trial - Correct Answer: Non-equivalent Group Design - I was a little confused here because both the RCT and Non-equivalent Group Design utilize the pre-testposttest method. 2) What is the main difference between an experiment and a quasi-experiment? - My Answer: Sampling - Correct Answer: Assignment Procedures - I was going back and forth between both sampling and assignment procedures because in my mind sampling and assignment procedures fall underneath the same bucket of describing the subject not being randomly selected. 11/1/21 : Module 7 11/15/21: Evidence leading up to a policy recommendation Policy issue/problem, this is how I organized it, these are the solutions <- Put that all in executive summary. Important element(s) Conclusion: Here’s what we talked about and here’s why all these matter
