Guiding Principles 4,5 & 6

advertisement
Six Guiding Principles
Sally Fincher
Building Research in Australasian Computing
Education: Second Workshop
26th-29th January 2005: Sydney
www.kent.ac.uk
Remember these …?
• Pose significant questions that can be answered
empirically
• Link research to relevant theory
• Use methods that permit direct investigation of the
question
• Provide a coherent and explicit chain of reasoning
• Replicate and generalize across studies
• Disclose research to encourage professional scrutiny
and critique
‹#›
For this workshop
• Last year, we concentrated on the first three. For this
workshop, we’ll be mostly concerned with the last
three:
• Provide a coherent and explicit chain of reasoning
• Replicate and generalize across studies
• Disclose research to encourage professional scrutiny
and critique
‹#›
Provide a coherent and explicit chain of
reasoning
• Provide a coherent and explicit chain of reasoning?
Sure, yes, of course ...
... but what does it mean? And how do we do it?
‹#›
Provide a coherent and explicit chain of
reasoning: what does it mean?
• It doesn’t mean that this sort of approach is
inexplicable, or that no-one else does it.
• In fact, we all have a good idea both of what “coherent”
and “explicit” mean.
• Let’s take an example, to explore this further …
‹#›
Street navigation
•We all know what we need to have (and to
have explained) to navigate through city
streets.
•The coherent and explicit chain of reasoning
we need depends on our context
•For example, in London, streets are named,
and signs tend to have a consistent “look and
feel” and to be placed about 3 feet off the
ground
‹#›
Street signs
•In the US, streets are also identified (although
not always with names) and, as a bonus, most
towns are laid out on a grid.
•However the signs tend to be placed above
eye level – on posts, or in the case of large
streets and highways, suspended over the
road itself
•The navigational chain of reasoning remains
clear, but we have to be explicit.
‹#›
Street signs
‹#›
Street signs
• Each village, town, city or ward is divided into "machi,"
the size of large neighbourhoods; each machi is
divided into "chome," which usually consist of several
dozen blocks. Each "chome" has numbered blocks,
and each block has numbered houses. Since only
important avenues have names, a house may not be
identified by which street it is on, rather by which block,
chome, and machi it is in
‹#›
Street signs
• Although there are no street signs in Japan, there are usually
placards on every other telephone pole that have the block
address on them. For example, one sign might read "Suginami-ku,
Asagaya-Minami 3-14." That means the telephone pole you are
looking at is located on the 14th block in the 3rd section of the
neighborhood called Asagaya-Minami in Suginami ward. If you
have a map book of Tokyo, you can look up the page for
Suginami, find Asagaya-Minami, find the 3rd section within it, and
the block within that with the number 14 on it. To see which side of
the block you are on, look at a telephone pole across the street; if
it says "Suginami-ku, Asagaya-Minami 3-22," then you know you
are between blocks 14 and 22
‹#›
Coherent and explicit
• So, Japanese street navigation is totally different from
UK or US street navigation – except in the principle of
navigation and in the fact that we all understand what
has to be accomplished and that there is a
systematised way to go about it.
• That is why we have to be careful to explain all the
steps in our reasoning – and what they mean to us –
even if they seem “obvious”
• If we don’t, then it’s all too easy to pervert the reader’s
view …
‹#›
Replicate and generalize across studies
• Well, yes, of course …
… but what does it mean? And how do we do it?
• It’s a twin problem of “input” and “output”
• Input: what previous work we have based our work on
and what we have used to inform our approach
• Output: giving sufficient information, in the right way,
for others to be able to replicate or generalize from our
work.
‹#›
Input
• In BRACE, what have we replicated & generalized?
• What is important, for instance, in these papers:
• M. McCracken, V. Almstrum, D. Diaz, M. Guzdial, D. Hagan,
Y.B.-D. Kolikant, C. Laxer, L. Thomas, I. Utting, and T. Wilusz.
(2001) A multinational, multi-institutional study of assessment
of programming skills of first-year CS students. Proceedings of
ITiCSE.
• Nathan Rountree, Janet Rountree and Anthony Robins
Predictors of Success and Failure in a CS1 Course (2002)
SIGCSE Bulletin vol. 34, no. 4.
• ETS Kit of Factor-Referenced Cognitive Tests, Educational
Testing Service, Princeton, NJ
• What is important in other papers?
‹#›
Output: Replication in Chemistry: Angwandte
Chemie
Angwandte Chemie
• [a typical paper] … is about three pages long. Almost
one page contains experimental detail. Half a page is
endnotes. The body of the article is then about one and
a half printed pages, of which roughly a third consists
of graphics
• The remarkable density of graphical material, most of
it quasi-iconic representations of microscopic
molecular structure.
‹#›
Angwandte Chemie
“The authors speak, as chemists today do, of molecules
that they do not see, but for which they have excellent
indirect evidence … I have written of this incredible
process, and the way that the chemists’ necessity to
move simultaneously in macroscopic and microscopic
worlds forces chemists to use a mixture of symbolic
and iconic representation of compounds/molecules”
Roald Hoffman (2002) Writing (and Drawing) Chemistry in Jonathan Monroe
(ed) Writing and Revising the Disciplines, Cornell University
‹#›
Note especially …
A particular feature of
Angwandte Chemie is
the mandatory inclusion
at the end of any
experimental paper is an
“Experimental Section”,
detailing procedures for
the experiments carried
out
Experimental Section
“This is a general statement that, in
effect, states that anyone, anytime,
anywhere who treats the same
ingredients in the same way as I did,
will make the same chemical
compound”
And yet, it is based on a single
emprical study at one specific time
and in one specific place
Within the positivist paradigm of
physical science this was, and is,
acceptable
‹#›
Contrast with this …
Referential footnotes & truth-claims
“The note is the correlate of research … The research
paper is written replete with referential notes, ideally, in
a restricted research paradigm, a note per statement in
the principal text. (More subjective moments are
confined to a preface or coda or perhaps to notes not
serving as references.) … “
‹#›
Referential footnotes & truth-claims
“ … to be professional historiography must have notes
that provide references for statements that function
referentially and make truth-claims.”
Dominick LaCapra (2002) Writing History, Writing Trauma in Jonathan
Monroe (ed) Writing and Revising the Disciplines, Cornell University
‹#›
For Computer Science Education
• We are neither Chemists nor Historians, so we don’t
have paradigmatic norms for giving replication and
generalization information.
• For BRACE
• We have two particular issues (common to work of the same
nature)
• And a particular problem (possible unique)
‹#›
Issue 1: “A double crisis of representation
and legitimation” Denzin & Lincoln, Handbook of Qualitative Research
• Some of what we are reporting is qualitative data.That
means we have to accurately present what we know.
• In doing so, we will make (often implicit) claims for its
status as knowledge and its veracity
• By representing what someone says … or interpreting
what a mapping exercise shows … we are embedding
what we believe it to represent by its very presentation
• So, we have to include enough information on our
representation of that knowledge to allow someone
else to judge it’s legitimacy
• There are different legitimacies for different types of
data and different claims (even within BRACE)
‹#›
Issue 1: “fair dealing”
• It is important to estimate error, and the possibility of
unseen error (resting on research design & conduct)
• It is important to conduct the practice of fair-dealing
‘fair dealing’-wariness of presenting the perspective of
one group as if it defined a single truth about the
phenomenon, while paying scant attention to other
perspectives”
Murphy, E. et al, (1988) Qualitative research methods in health technology
assessment: a review of the literature Health Technology Assessment 2 (16)
‹#›
Issue 2: How general are generalizations?
Following the work of Michael Bassey – see references at end
• How can we generalize from any empirical study?
• Scientific generalization
• A positivistic, Popperian view: if x happens in y circumstances
then z will occur in all cases (recall the “experimental section”,
above)
• Probabilistic generalization
• Reliant on sample size and statistical power: if x happens in y
circumstances then z will occur in about p% of cases (recall we
characterised these as “empirical laws” last year)
• Fuzzy generalization
• A class of statements which are imprecisely probable: if x
happens in y circumstances then z may occur
‹#›
Issue 2: Fuzzy generalizations
The fuzzy principle state that everything is a matter of
degree … Fuzziness has a formal name in science:
multivalence. The opposite of fuzziness is bivalence or
two-valuedness, two ways to answer each question,
true or false, 1 or 0. Fuzziness means multivalence. It
means three or more options, perhaps an infinite
spectrum of options, instead of just two extremes. It
means analogue instead of binary, infinite shades of
grey between black and white. It means all that the trial
lawyer or judge tries to rule out when she says,
“Answer just yes or no”.
Bart Kosko (1994 ) Fuzzy Thinking, p.18
‹#›
Issue 2: fuzzy generalisation, the necessary
corollary: BET
• If x happens in y circumstances then z may occur
Well, yes, but then equally:
• If x happens in y circumstances then z may not occur
• Something else is needed. Bassey suggests Best
Estimate of Trustworthiness (BET) which is an
additional statement, an “educated guess” based on
the experience, insight and professional judgement of
the researcher(s)
• Expressed as a range of likelihood (1-10%, perhaps)
‹#›
Problem: generalization
• You can think of the BRACE project as a single piece
of work collaboratively conducted by 15 researchers
• You can think of the BRACE project as an aggregation
of 15 replicated studies
• What issues of replication and generalization change
with these two views?
• What part does “institutional context” play?
• What might the fact that some of the raw data is from a UK
institution mean?
• Is it important that different institutions have contributed
different quantities of data?
‹#›
Domain where
assertions may be true
or is partly true
To infinity
Conditioned generalisation
Domains where
assertion is wholly or
nearly untrue
Domain in which the
assertion is true or at
least a very good
approximation
Stephen Jay Kline (1995) Conceptual Foundations for
Multi-Disciplinary Thinking Stanford University Press
Disclose research to encourage
professional scrutiny and critique
• Yes, indeed. Warts and all, we’ve got a paper to write.
• Ladies and gentlemen, start your engines.
‹#›
Michael Bassey references
• Bassey, M (1999) Case Study Research in Educational
Settings Open University Press
• Bassey, M (2001) A Solution to the Problem of
Generalisation in Educational Research: fuzzy
prediction Oxford Review of Education 27 (1) pp 5-22
• Bassey, M (2003) How general are generalizations? In
Joanna Swann and John Pratt, eds Educational
Research in Practice Continuum Books, London pp.
163-171
‹#›
Download