CONSULTING SESSION SUMMARY REPORT
Bilingual Decision Making
CLIENT: Mr. Zachary Brooks
CONSULTANTS: Dean Billheimer, dean.billheimer@arizona.edu
Sean Ashley (author), sashley@email.arizona.edu
Meng Lu, menglu@email.arizona.edu
Jing Zhai, jingzhai@email.arizona.edu
DATE: 16 November 2015
BACKGROUND
The consulting team met with Mr. Zachary Brooks on Monday 9 November 2015.
Some parts of this background and the study section are largely paraphrased from a
written description provided by the client, Mr. Zachary Brooks.
Mr. Zachary Brooks’ dissertation topic is Bilingual Decision Making and he is
completing his PhD in the Graduate Interdisciplinary Program in Second Language
Acquisition & Teaching (SLAT) with minors in Cognitive Science and Management
with a concentration in decision making. Given the option of writing a monograph or
writing three distinct yet related papers, Zachary chose to write three papers. Tom
Bever is his advisor. Zachary is collecting data from first-language (L1) and secondlanguage (L2) speakers of English.
THE STUDY
Zachary uses the surveying software Qualtrics to collect response data. At the time
of our meeting, at least 1025 subjects had taken at least one of the three surveys. At
least 400 had taken all three surveys. Each survey is associated with one of the three
papers. The L2 subjects are from the Center for English as a Second Language (CESL)
and from ENGL-107, an English course designed for English-as-a-Second-Language
(ESL) students, classes. The L1 subjects are sampled from ENGL-101, an English
course designed for native English speakers, classes. The data includes the
competency levels of the L2 subjects and their native languages. The native
languages represented include Spanish, Portuguese, Chinese, and Arabic. Whether
native dialect, such Mandarin vs Cantonese, was recorded was not addressed in the
meeting. Fluent bilinguals were recorded.
Zachary’s three papers address three research questions listed in three studies.
Study 1: How do the decisions of L1 and L2 compare using verbal probability
assessments as a measure? What discernable patterns are evident in the decisions
of intermediate and advanced L2 English speakers versus L1 English speakers?
Verbal probability assessment is how an individual maps words or phrases of
probability to quantifiable probabilities. An individual is asked to map a word or
phrase such as “probably” or “likely” or “rare” to numerical probabilities such as
“90%” or “70% or “20%” chances of occurrence. Different mappings are expected to
depend on native language. Zachary provided the example that native Arabic
speakers tend to associating similar probabilities to lower-probability English phrases
while associating a wider range of probabilities to higher-probability English phrases.
This could be because of a richer gradient of words for higher-probability phrases in
the Arabic language and a more sparse range of words for lower-probability phrases
in the Arabic language.
Study 2: How do the decisions of L1 and L2 compare using verbal Trolly problems
assessments as a measure? What discernable patterns are evident in the decisions
of intermediate and advanced L2 English speakers versus L1 English speakers?
The Trolly problem is an enlightening ethics thought experiment. The hypothesis is
that L1 speakers will take a more principle-based decision-making process while L2
speakers will take a more utilitarian approach.
Study 3: How do the decisions of L1 and L2 compare using estimation as a measure?
What discernable patterns are evident in the decisions of intermediate and advanced
L2 English speakers versus L1 English speakers?
The estimation measure is also referred to as a quick-response measure and is a
Cognitive Reflection Test (CRT). An example includes briefly displaying a scatter of
dots to the subject and asking the subject to tell the number of dots. Zachary
mentioned he spoke with a couple lawyers and that the estimation measure is an
accepted measure in the study of decision-making. Zachary plans to use the
estimation measure as a type of control variable to account for variations within
language groups.
DISCUSSION
The major questions brought forth, as understood by the team of consultants, are:
1. Is there a difference? Is there a difference in decision making between L1
and L2 groups?
2. Are there patterns? If so, what are the patterns? The patterns are of
decision making between and among the L1 and L2 groups.
3. Are the discovered patterns more than statistical anomalies?
4. Are there covariates that were not considered?
One of the major statistical concerns in the study is whether the analysis of the data
is accounting for multiplicity. Multiplicity is the statistical concept that a larger
composite statement composed of many smaller constituent statements will require
the constituent statements to be stronger for the composite statement to be of a
specified strength. For example, in order to be 90% confident in the statement “all
cats are green and all dogs are blue”, we may have to be 95% confident in the
individual statement “all cats are green” and 95% confident in the individual
statement “all dogs are blue”. The concept of multiplicity is relevant because
Zachary’s research method is survey based with a number of questions per survey.
Statistical tests can be performed on each question individually. However,
multiplicity correction should be considered when amalgamating results from
multiple tests simultaneously. There are various methods for multiplicity correction.
While the Bonferroni Adjustment is most simple, it is very conservative and could
result in false negative decisions. Dr. Billheimer suggested looking into the Benjimini
& Hochberg method for multiplicity correction.
It was suggested to analyze variations, trends, and patterns within the native
language groups before observing the differences between them. This may allow for
a sequential analysis which gradually builds up the bigger picture by understanding
portions of the puzzle at a time.
There were further questions on the CRT (Study 3) and on the use of the estimation
measure as a baseline. This is not a statistical question, but more a question out of
curiosity on how the estimation measure would serve as a good baseline.
NEXT STEP
Sean Ashley will write up the meeting summary (this document).
The team will write up a report for the client.
The team should include an explanation of multiplicity correction and its relevance
and application to Zachary’s research. The team could look into more sophisticated
methods for multiplicity correction, such as Benjimini & Hochberg, in the context of
Zachary’s research.
Organization and delegation on roles, who’s going to write what, for writing the
client report need to be done.
Will the group be discussing the Benjimini & Hochberg method for multiplicity
correction?