International Diversity in Graduate Engineering
Education: Development of a Multi Institutional
Survey
Erin Crede and Maura Borrego
Virginia Tech, ecrede@vt.edu, mborrego@vt.edu
Abstract - This study examines the experiences of one of
engineering graduate schools largest populations:
international students. This paper discusses the use of
ethnographically guided observations and interviews in
the development of a multi institution survey on
graduate student experiences. The results of qualitative
analysis will be briefly discussed to contextualize the
survey sections and creation of individual questions.
The development of this survey was the second part of a
multi phase mixed methods study whose purpose was to
investigate the role of international diversity in graduate
engineering communities. Results of this study will help
shape the future of graduate engineering education by
bringing the experiences of both international and
domestic graduate engineering students to the forefront.
We conclude with implications for the current state of
engineering graduate education as well as directions for
future research.
Index Terms – graduate education, international students,
survey design, mixed methods
INTRODUCTION
The high percentage of international students is a defining
feature of many graduate engineering programs. To date,
much of the research regarding the recruitment and retention
of doctoral students has focused on examining student
characteristics and factors external to the university, such as
undergraduate GPA, gender and marital status [1, 2].
Additional studies have focused on the interaction between
the student and the department or discipline, but did not
incorporate engineering students [3, 4]. Previous studies
have included a broad range of disciplines, examining the
statistically significant differences in program completion
rates, and identifying characteristics that distinguish one
discipline from another. Paramount among these cited
differences was the large proportion of international
students attending US graduate engineering programs[5].
While these quantitative studies have yielded a broad
overview of possible factors that influence the decision to
leave doctoral study, further exploration of what constitutes
the graduate student experience will illuminate additional
factors not previously studied. In order to more fully
understand patterns of doctoral student attrition we must
first develop a deeper understanding of the lived
experiences of engineering students during their graduate
education.
Previous research has concluded that
international diversity is a defining feature of engineering
graduate programs, but there is little work on how this
makes engineering different from other disciplines.
In their 1984 article, Stone and Campbell warn
researchers that survey data alone can produce misleading
results when researchers work outside of the area on which
they are knowledgeable[6]. Yet another problem arises
when there are cultural differences between survey
populations [7]. When creating survey items, it is important
to note that cultural norms, values, and experiences
influence the respondents understanding of the relevant set
of constructs [8]. This will consequently influence the
range of behaviors and ideas that are valid indicators of the
constructs, and how respondents interpret items used to
assess these constructs [8]. For social research to be good
science, methods must be merged with knowledge of a
group's culture [9]. The choice of methodological tools
should facilitate researchers' knowledge about the
experiences and perspectives of the group under study, as
well as the development of research instruments that are
useful and relevant [10].
With this in mind, we set out to develop an instrument
that was grounded in the experiences of graduate students
paying special consideration to cultural differences among
the different student groups. If we want to quantitatively
examine these experiences, we must first develop an
understanding of foreign and domestic students in a smaller
research group setting, and use these results to shape our
larger study. This paper focuses on the findings of an
ethnographically guided investigation that were used to
develop a multi-institution survey. Using observations and
interviews of internationally diverse graduate engineering
research groups, we were able to better characterize possible
factors that would contribute to students overall graduate
school experience, which will enable us to measure the
softer sides of student engagement and commitment to
graduate school. The remainder of this article discusses the
initial qualitative data collection and analysis procedures,
followed by how we used these to inform the development
of a quantitative instrument. This is especially important for
researchers developing an instrument that will be
administered to participants from various cultures, or when
little is known about the constructs in question. In
developing this instrument, we were guided by the aim of
the larger study which is to develop a clearer understanding
how a high percentage of international students influence
the culture of a graduate engineering research group.
QUALITATIVE RESEARCH METHODS
Creation and administration of the survey instrument was
the second phase of a larger study. The overall research
methodology for this project was exploratory mixed
methods [11, 12]. This type of study design is appropriate
when the research being undertaken is exploratory in design
and theory building and verification are necessary [13]. The
qualitative phase of the study utilizes ethnographically
guided research methods, including formal and informal
interviews, lengthy periods of observation and participation
in most research group activities. Ethnography is a strategy
of inquiry where the researcher studies an intact cultural
group in a naturalistic setting over a prolonged period of
time [14], and is necessary when the purpose of the research
is to understand how people’s beliefs and values guide their
actions and understanding of those actions [15]. This was
deemed especially important due to the social and cultural
nature of our research question.
Findings from the
ethnography were used to inform the development of an
electronic survey.
I. Participants and Setting
The setting for the ethnography was three graduate
engineering research groups and several individual students
in two engineering departments at a large public university.
These were purposefully selected to represent different
types of research groups, overall size, and proportions of
international students. Aerospace and Ocean Engineering
currently enrolls 119 graduate students; 26% are
international, and 73% are Caucasian domestic students.
Electrical and Computer Engineering currently enrolls 486
graduate students; 67% are international, and 29% are
Caucasian domestic students. International students
represent a wide variety of countries including China, India,
South Korea, Bangladesh, Indonesia, Iran, Sweden and the
United Kingdom.
One research group was
multidisciplinary, featuring students and faculty from both
departments.
II. Ethnographic Data Collection
Data collection for this study contained ethnographic
observations, interviews and diary entries. A total of 25
interviews were collected from graduate students and
faculty advisors. Semi structured formal interviews were
conducted following the procedures prescribed by
Hermanowicz and Seidman [16, 17]. Informal interviews
were also accomplished as part of the ethnographic
fieldwork. Observation data was collected via a graduate
research assistant who shadowed selected research groups
for up to 20 hours per week, paying special attention to be
present at lab group meetings and other key events such as
social gatherings, and seminars. The methods prescribed by
Emerson et al [18] for taking ethnographic field notes were
used along with data collection procedures discussed in
Fetterman [19]. In addition to interview and observation
data, select participants completed weekly diary entries
describing the events that had occurred in the lab and within
the research group during that week. They were encouraged
to elaborate on their interactions with other group members
and their feelings of participation and acceptance within the
research group.
Methods for diary creation and
administration were accomplished using the methods
prescribed by Bolger et al [20].
III. Ethnographic Data Analysis
As with most qualitative research traditions, data analysis in
ethnography is integrated into the data collection phase
rather than undertaken at the end. The aim in ethnography is
to describe the culture, specifically the beliefs and values
that guide people’s behavior and understanding [15]. Both
data collection and analysis focus on identifying what is
important in that culture. In graduate education, this might
mean: How are new students expected to spend their time
getting up to speed? What is considered a stupid question?
When and for how long should students be in the lab or
office? How much guidance should students expect from
their advisor? Who is viewed as an authority, and why?
What networks operate to “get things done?” Which
projects and research topics are considered most
prestigious? Who interacts socially with whom, and what
additional information does this relationship afford?
Reflecting on ethnographic data captured in field notes is
important to both capture insights and understanding as they
occur, as well as to ensure that gaps in the data can be
addressed while the ethnographer still has access to the
participants.
During the qualitative data analysis, several working
hypotheses suggested by the literature were explored. First,
expectations that international and domestic students
develop regarding the graduate institution impact their level
of engagement within their research group and department
[21]. Second, the presence of large numbers of international
students increases the awareness of diverse perspectives
among all graduate students (foreign and domestic), thus
shaping the academic community. In addition to the
suggestions presented in the literature, the researcher
allowed for additional phenomena relating to graduate
student experiences of international and domestic students
to emerge.
LESSONS FROM THE QUALITATIVE FINDINGS
The purpose of using ethnographically guided methods was
to enable the research team to better conceptualize the
graduate student experience. This technique has been used
by researchers in the past, specifically in cases where there
are cultural factors that the researchers may not comprehend
at the outset of the study[7]. Through these interviews and
observations we were able to strengthen our conceptual
understanding, our awareness of different cultural
considerations, and generate new hypotheses to test.
I. Conceptual Understanding
The important question to consider here is did we learn
anything from the interviews and observations that we could
not have learned from the instrument alone? This might
also be interpreted as, “were there any import cultural,
experiential or theoretical concepts whose importance or
even existence was realized as a result of the qualitative
work?”
We had an initial plan that included many of the main
constructs that we were interested in measuring from a
thorough review of the existing theoretical literature [22],
but were undecided about how to measure individual
components of these constructs. We originally planned to
investigate student identity development, socialization,
enculturation (both to graduate school and the host country)
and career aspirations. Through our observations and
interviews with research groups and individual students, we
were able to identify several ways that identity was socially
constructed, and explore how students interact in social and
work settings. While participating in many of the research
group activities we were able to observe interactions
between students and faculty members, along with the
general group dynamic. Additionally, exposure to graduate
students in a research group setting shed light on other
factors that contribute to the graduate student experience we
had not previously considered, such as how motivation and
informal learning practices contribute to socialization and
professional identity development.
One of the most important considerations regarding our
conceptual understanding was the constant interaction with
students and faculty in nontraditional settings. Using an
ethnographic framework enabled the research team to attend
social gatherings both in and outside of the academic setting
which deepened our understanding of the range of factors
that comprise the graduate school experience. Had we not
attended group lunches, happy hour, movies and even a
wedding, we would not have obtained an understanding of
the lesser researched areas of student learning and
development. Our constant presence in both the social and
academic environment helped ensure that the details
students might not find relevant were available to the
research team, allowing us to paint a more complete picture.
Finally we were able to observe the language of the
participants, and consequently could use this wording in the
questionnaire design. This was beneficial not only for the
terms used by domestic students and faculty members, but
especially so for the international students.
II. Cultural Considerations
There are two components of the cultural considerations that
are relevant to this study. The first is the social construction
of the culture of graduate study in an engineering
department, and the second was the more traditional cultural
differences among international students. The overarching
goal of the larger study is to explore the effect of
international diversity on engineering graduate education.
Through observing a highly diverse research group, we were
able to capture the interactions between domestic and
international students that might not have been accurately
captured in a questionnaire.
Interviews with several international students enabled
the research team to explore variations in cultural
understanding. Observations of students confirmed findings
from the literature, as well as revealed some other examples
of differences in communication patterns of students from
various nationalities. Perhaps most importantly we were
able to confirm that the variables we were interested in were
salient to students from countries other than the United
States. Finally, one of the observed research teams had
students from nine different countries, allowing the research
team to ensure that the instrument will be understood by
individuals from a wide range of backgrounds.
III. Hypothesis Formulation
We began the ethnographic data collection with a few
working hypotheses suggested by the literature. First, we
hypothesized that prior expectations regarding the graduate
institution held by international and domestic students
impact socialization within their research group and
department [21]. Second, the presence of large numbers of
international students increases the awareness of diverse
perspectives among all engineering graduate students
(foreign and domestic), thus shaping the academic
community. Several research studies have investigated
related topics, including Litzler et al [23] who investigated
the climate for graduate students in science and engineering
departments. They found that the degree of competition is
highly and negatively correlated to degree progress.
While conducting the initial interviews and
observations of the research groups we were using a highly
inductive approach, attempting to allow themes and
additional constructs to emerge from the data. After a few
months of pure data collection, we began to simultaneously
collect and analysis data, which is common practice with
ethnographically guided observations and interviews [18,
24]. From this analysis we were able to develop some
additional working hypotheses that the newly designed
instrument would be able to test. Examples of these include
the importance and influence of various groups (friends,
peers, faculty members) on students from different
backgrounds, and the salience of different identities for
students from different countries at various stages of their
graduate career. The collection and analysis of the
interview and observation data allowed the research team to
design the survey with an analysis plan already in place,
which greatly facilitates expedient design and pilot testing.
SURVEY DESIGN AND ANALYSIS
After thoroughly considering all of the ethnographic data we
were ready to develop the instrument. It is important to note
that not all of the qualitative data was incorporated into the
questionnaire. However, because we now had a more
complete understanding of the experiences of graduate
students, we were able to prioritize hypotheses that we
intended to test, and reserve additional research questions
for future instruments.
The questionnaire was designed to measure student
experiences and perceptions, and thus contains 50, five
point Likert scale questions designed to factors relating to
identity salience, cultural expectations, and socialization..
Additionally there are several questions soliciting
demographic information, specifically relating to
nationality, gender, degree program, year in school, and
academic discipline among others. These factors would be
used to compare overall engagement and commitment to
graduate school we well as determine if there were
statistically significant differences between international and
domestic students. Several of these demographic variables
were chosen to enable the data to also be compared to other
studies. Using our newly acquired knowledge of the
population in questions, we followed the recommendations
of Fowler [25] to create the survey questions, and the
guidance of Dillman, Smyth and Christian [26] and Groves,
Fowler et al [27] regarding sampling techniques.
Descriptive statistics such as means, medians, and
standard deviations will be computed for most survey items.
Analysis of variance (ANOVA) will be used to determine if
there are statistically significant differences in academic
departments/disciplines, program size, public versus private
universities, and across student demographic groups. Linear
regression models will be explored to uncover the
combination of factors contributing to the previously
mentioned constructs among both international and
domestic graduate students.
Following completion of the survey construction we
member checked the instrument within the research groups
that participated in the interviews and observations. This
was accomplished primarily to confirm that the language
and terminology was understood by the wide range of
international participants. Currently the instrument is being
pilot tested using a larger audience of approximately 50
graduate engineering students from a range of disciplines.
Once the instrument meets sufficient validity and reliability
measures locally, and the research team is satisfied that the
data analysis techniques and software are running smoothly,
we will launch the instrument to the target population. Four
additional universities have agreed to participate from all
across the United States, totaling 7,000-8,000 potential
respondents. We anticipate complete results of the pilot test
(validity and reliability analysis) as well as preliminary
findings from the instrument by November 2010.
IMPLICATIONS FOR SURVEY RESEARCHERS
Perhaps the most important implication this data provides is
the importance of understanding the social, cultural and
personal factors about a population prior to conducting
quantitative research. Using the literature, we could have
created a valid and reliable instrument, but still missed out
on several important constructs that greatly increase our
understanding of the experiences we are attempting to study.
The ethnographically guided investigation we have
undertaken is not to be confused with a pilot test, which is
generally smaller in scope and scale. Without a thorough
investigation of the variables that comprised our constructs,
we would not have known what items to include in the
initial construction of the questionnaire, how to word them,
and what would be of interest to our participants. While
pilot testing would have enabled us to fix many of the
pitfalls encountered by survey researchers, the deep
understanding of our participants through several months of
observations and countless hours of interviews could not
have been obtained through pilot testing alone.
Through the use of ethnographically guided interviews
and observations we were able to obtain insight into the way
that graduate students think and behave in internationally
diverse research groups.
Collection and analysis of
qualitative data contributes to the design of the
questionnaire in several ways. The first is the actual
development of the questions. We were able to uncover
several measureable variables through the prolonged
ethnographic investigation of the constructs of interest. We
were also able to investigate firsthand how and where
interactions might occur in the data and generate new
hypotheses that data from the questionnaire would be able to
test. Second, interviews with various participants showcased
how the research was viewed by the different types of
participants. The diversity of the target population for this
questionnaire required careful attention to the values, norms
and cultural expectations of vastly different groups. Had we
not experienced lengthy periods of contact with several
different cultures we might not have obtained the full
benefits of the instrument. Additionally the exposure to
such a diverse sample of our target population helped to
reduce potential measurement error, increasing the overall
validity of the instrument.
The ethnographic methods used by this research team
represent only one type of qualitative inquiry that may guide
instrument development. There are many other qualitative
methods that may serve interested survey researchers,
including focus groups, interviews, case studies,
observations, document analysis and diary methods [13, 14,
17, 20, 28]. Researchers interested in using qualitative
methods to better design survey instruments should consider
several questions when choosing a method:
1.
2.
3.
4.
What is your overall research question and how
does it relate to what you are trying to measure?
What type and level of background information is
available on your topic?
How easy will it be to gain access to participants,
and for how long can you gain access to them?
What don’t you know about the population of
interest that may influence your choice of
questions?
5.
How large and diverse is the population you are
trying to generalize to?
The availability of background information can be a
driving factor in the length of your qualitative investigation.
If relatively little is known about the population in question,
you may have to spend more time with participants than if
you are merely trying to translate an instrument from one
population to another. Focus groups can provide an
excellent medium for gathering information on the basic
terminology used and the basic level of understanding held
by the participants. Interviews provide a similar forum for
gathering information to develop the wording of specific
questions, as well as what questions are appropriate to ask
of the population.
Using this type of study design opens the researcher to
using mixed methods data analysis techniques at the
conclusion of the study. Depending on the level of
qualitative data acquired a research team could use these
data to triangulate results, or further explore the results of
the quantitative findings. The use of qualitative research
methods prior to designing and administering a survey will
help ensure a more valid instrument, open the researcher up
to new hypotheses to test, and provide a means of exploring
other analysis techniques, such as mixed methods research
designs.
ACKNOWLEDGMENT
This work was part of a larger project sponsored by the
National Science Foundation, funded under EEC #0934643
REFERENCES
[1].
Nettles, M.T. and C.M. Millett, Three Magic Letters-Getting to
PhD. 2006, Baltimore, MD: The Johns Hopkins University Press. 329.
[2].
Abedi, J. and E. Benkin, "The effects of students' academic,
financial, and demographic variables on time to the doctorate", Research in
Higher Education, 1987. 27(1): p. 3-14.
[3].
Golde, C.M., "The Role of the Department and Discipline in
Doctoral Student Attrition: Lessons from Four Departments", The Journal
of Higher Education, 2005. 76(6): p. 669-700.
[4].
Golde, C.M., "Beginning Graduate School: Explaining FirstYear Doctoral Attrition", New Directions for Higher Education, 1998.
1998(101): p. 55-64.
[5].
Gardner, S., "Student and faculty attributions of attrition in high
and low-completing doctoral programs in the United States", Higher
Education, 2009. 58(1): p. 97-112.
[6].
Stone, L. and G. Campbell, "The Use and Misuse of Surveys in
International Development: An Experiment From Nepal", Human
Organization, 1984. 43(1): p. 27-37.
[7].
Fuller, T.D., et al., Using Focus Groups to Adapt Survey
Instruments to New Populations, in Successful Focus Groups: Advancing
the State of the Art, D.L. Morgan, Editor. 1993, Sage Publications:
Newbury Park, Ca.
[8].
Hui, C.H. and H.C. Triandis, "Effects of culture and response
format on extreme response styles", Journal of Cross-Cultural Psychology,
1989. 20: p. 296-309.
[9].
Bulmer, M., "Introduction: The Problem of Exporting Social
Survey Research", American Behavioral Scientist, 1998. 42(2): p. 153-167.
[10].
Hughes, D. and K. DuMont, "Using focus groups to facilitate
culturally anchored research", American Journal of Community
Psychology, 1993. 21(6): p. 775-806.
[11].
Creswell, J.W. and V.L. PlanoClark, Designing and Conducting
Mixed Methods Research. 2007, Thousand Oaks, Ca: Sage Publications.
[12].
Tashakkori, A. and C. Teddlie, eds. Handbook of mixed
methods in social and behavioral research. 2003, Sage Publications:
Thousand Oaks, CA.
[13].
Patton, M.Q., Qualitative Research & Evaluation Methods. 3
ed. 2002, Thousand Oaks, Ca: Sage Publications Inc.
[14].
Creswell, J.W., Research design: qualitative, quantitative, and
mixed methods approaches. 3 ed. 2009, Thousand Oaks: Sage
[15].
Rossman, G.B. and S.F. Rallis, Learning in the Field, An
Introduction to Qualitative Research. 2 ed. 2003, Thousand Oaks: Sage
Publications.
[16].
Hermanowicz, J.C., "The Great Interview: 25 Strategies for
Studying People in Bed", Qualitative Sociology, 2002. 25(4).
[17].
Seidman, I., Interviewing for Qualitative Research: A Guide for
Researchers in Education and Social Sciences. 3rd ed. 2005, New York:
Teachers College Press. 162.
[18].
Emerson, R.M., R.I. Fretz, and L.L. Shaw, Writing
Ethnographic Fieldnotes. 1995, Chicaco: University of Chicago Press.
[19].
Fetterman, D.M., Ethnography. Second ed. Applied Social
Research Methods Series. Vol. 17. 1998, Thousand Oaks, CA: Sage
Publications Inc.
[20].
Bolger, N., A. Davis, and E. Rafaeli, "Diary Methods:
Capturing Life as it is Lived", Annual Review of Psychology, 2003. 54: p.
579-616.
[21].
Lovitts, B.E. Who Is Responsible for Graduate Student
Attrition--The Individual or the Institution? Toward an Explanation of the
High and Persistent Rate of Attrition. in Annual Meeting of the American
Educational Research Association. 1996. New York.
[22].
Crede, E. and M. Borrego. The Effect of International Diversity
on Graduate Engineering Education: A Literature Review. in 2010
American Society of Engineering Education Annual Conference and
Exposition. 2010. Louisville, KY.
[23].
Litzler, E., S.E. Lange, and S.G. Brainard. CLimate for
Graduate Students in Science and Engineering Departments. in 2005
American Society of Engineering Education Annual Conference and
Exposition. 2005.
[24].
Charmaz, K., Constructing Grounded Theory, A Practical
Guide to Qualitative Analysis. 2006, Los Angeles, CA: Sage Publications.
[25].
Fowler Jr, F.J., Improving Survey Questions. Applied Social
Research Methods Series, ed. L. Bickman and D.J. Rog. Vol. 38. 1995,
Thousand Oaks, CA: Sage Publications Inc.
[26].
Dillman, D.A., J.D. Smith, and L.M. Christian, Internet, Mail
and Mixed Mode Surveys. 3rd ed. 2009, Hoboken, NJ: John Wiley and
Sons.
[27].
Groves, R.M., et al., Survey Methodology. 2nd ed. 2009,
Hoboken, NJ: Wiley and Sons, Inc.
[28].
Yin, R.K., Case Study Research: Design and Methods. Third ed.
2003, Thousand Oaks, CA: SAGE Publications.
AUTHOR INFORMATION
Erin Crede, Doctoral Student, Department of Engineering
Education, Virginia Tech, 660 McBryde Hall (0218),
Blacksburg VA 24061, ecrede@vt.edu
Maura Borrego, Assistant Professor and Director of the
Graduate Program, Department of Engineering Education
Virginia Tech, mborrego@vt.edu