Research Methods of Applied Linguistics and Statistics (3) Types of research Constructing Research Designs Research Types ◼ ◼ ◼ ◼ Data sources: primary, secondary Objectives: descriptive, exploratory, correlational, explanatory Time for data collection: cross-sectional, longitudinal (panel, trend) Data types: qualitative, quantitative (survey, experimental) ◼ ◼ Primary: studies based on primary, or original, data sources, such as classroom observations or real students, or their test scores, or their responses to a questionnaire. Secondary: studies based on secondary sources such as other researhers’ books and articles, including library research and literature reviews. ◼ ◼ Descriptive: studies involving the collection of data in order to test hypotheses or to answer questions concerning the current status of the subjects of the study. Exploratory: studies conducted into an issue or problem where there are few or no earlier studies to refer to. The focus is on gaining insights and familiarity for later investigation. ◼ ◼ Correlational: The basic question for descriptive research is - "What are the values of a number of variables for a given sample of subjects. The basic research question for correlation research is - What is the relationship between two or more variables for a given set of subjects. Notice that we said relationship between variables and not the effect of one variable on another variable. In descriptive research we are just describing our subjects in terms of one or more variables, while in correlational research we are looking at the relationship between the variables. ◼ ◼ Explanatory: studies conducted in order to explain any behaviour in L2 learning and teaching. It could be done by using questionnaires, group discussions, interviews, etc. Cross-sectional: studies in which groups of participants of different ages are observed and compared at a given time ◼ Longitudinal: It is research that studies a person or group over a set period of time, normally to track the effect of some variable and as such permits causal pathways to be determined . For example, when trying to discover the effect of language aptitude on L2 proficiency, a researcher may track 50 children over a 30 year period in order to find a common factor between those children who are quicker and effective L2 learners. Constructing Research Designs ◼ Determining the design ◼ Classification of designs Research Design 1: Quantitative / Qualitative ◼ Quantitative research is empirical research involving analysis of numerical data Its aim is to determine the relationship between one thing (an independent variable) and another (a dependent or outcome variable) in a population. Quantitative research designs are either descriptive (subjects usually measured once) or experimental (subjects measured before and after a treatment). A descriptive study establishes only associations between variables. An experiment establishes causality. ◼ Qualitative research is empirical research involving analysis of data such as words (e.g., from interviews), pictures (e.g., video), or objects (e.g., an artifact). Its aim is building theory. It is generally inductive in approach, is based originally on the naturalistic assumption that reality is mind dependent (i.e., can only be known as it is interpreted and has "meaning" for the observer), is usually of single-subject design, and generally deals with nominal data Philosophical base Phenomenology (quali.) Logical positivism (quan.) Data from personal involvement Subjective Data from experiment Empathy Distance Data in the form of string Data in form of number Objective Methodology Synthetic Analytic From part to whole From whole to part Internal structure External structure Process-oriented Product-oriented Dynamic reality hypothesis Static reality hypothesis Purpose Induction Deduction Hypothesis generating Hypothesis confirming Exploratory, descriptive Inferential, confirmatory, explanatory Result of pattern or Formation of theory hypothesis Features for both designs High validity, low reliability Preference of content to form Low generalizing capacity High explanatory power Low validity, high reliability Preference of form to content High generalizing capacity Low explanatory power Mixed design: Balanced Mixed design: Unbalanced Literature review of quantitative and qualitative studies by Lazaraton (2000) ◼ Current Trends in Research Methodology and Statistics in Applied Linguistics Research Design: experimental research An experimental study is a study in which the researcher manipulates one or more independent variables and measures their effect(s) on one or more dependent variables while controlling the effect of extraneous variables. Major components of experimental research The population A comparison which involves at least 2 groups or 2 conditions. ◼ The treatment A treatment in which one or more independent variables are manipulated. ◼ The measurement of the treatment The measurement of dependent variable(s) as the result of the treatment. ◼ Letters standing for components ◼ ◼ ◼ ◼ X: an experimental treatment (teaching method, exposure to a material…) －: no treatment O: measurement of the effects of the treatment or a test R: randomization, or the random assignment of subjects to groups in order to control for extraneous variables. Types of experimental studies Pre-experimental One-group posttest only study: X O1 One-group pre/post test study: O1 X O 2 No control group; no randomization Quasi-experimental True-experimental Posttest only nonequivalent groups: EG: X O1 CG: O2 Pre/post test nonequivalent groups: EG: O1 X O2 CG: O3 O4 Posttest only equivalent groups EG: R X O1 CG: R O2 Pre/post test equivalent groups: EG: R O1 X O2 CG: R O3 O4 a control group; no randomization a control group + randomization Pre-experimental: single group design One-shot design ◼ One group pretest-posttest design ◼ Time-sampling design ◼ Example for one-shot design An English teacher used the immersion teaching method for teaching English communicative use. After 3 weeks, a test is administered to the class. The class performs well on the test. ◼XO ◼ It is also called a pilot study design ◼ One group pretest-posttest design It is also called repeated measures. It uses ◼ ◼ the subjects as their own controls and eliminates the need for a control group design. Strength: It is efficient because some extraneous variables, attrition and loss of subjects controlled. Weakness: validity might be affected by history, attrition and maturation. Time-sampling design ◼ ◼ ◼ It is also referred to as time-series designs. O 1 O 2 O n X O4 O 5 O n O1 X O2 → O3 -- O4 →O5 X O6 →O7 -- O8 Quasi-experimental: posttest only nonequivalent groups • EG: X O1 CG: O2 • O1-O2 • The treatment effect could also be attributed to a series of extraneous variables. Posttest/posttest nonequivalent groups EG: O1 X O2 CG: O3 O4 ◼ (O2-O1) – (O4-O3) ◼ Without randomization, the confounding effects cannot be fully controlled. True-experimental: posttest only equivalent groups EG: R X O1 CG: R O2 O1-O2 The design assumes that the 2 groups are identical on all relevant features, but it is still sensitive to selection bias, mortality, Hawthorn effects. Pre/post test equivalent groups EG: R O1 X O2 CG: R O3 O4 Factorial Designs ◼ Factorial designs are similar to the true experimental designs and include all of the elements found in those designs, such as randomization, pre- and post-testing, and treatments. The difference is that the effects of independent variables and a moderating variable may be tested at the same time. Example for a factorial design ◼ ◼ In a study to test for the effect of language laboratory training on pronunciation, it is also decided to measure for the effect of such training on learners of different proficiency levels, as well as the effect of different kinds of language laboratory practice. Types of practice (contextualized / decontextualized drills) , proficiency levels of the learner (beginners / advanced learners ) Exemplary Figure for Factorial Design Experimental study by Robb, et al. (1986) ◼ Salience of feedback on error and its effect on EFL writing quality ◼ Types of feedback: complete correction, coded feedback; yellow highlighting of errors, listing the number of errors The compositions were graded on 19 features which were collapsed into 7 categories. ◼