Critical evaluation (web version)
1. 1. Critical Evaluation of Research Information James Bisset
(james.bisset@durham.ac.uk ) Academic Liaison Librarian (Research Support)
2. 2. Session outline - Importance of evaluation - Forms of value - Group activity Evaluating Research Information
3. 3. Session outline - Importance of evaluation - Forms of value - Group activity Evaluating Research Information
4. 4. Session outline - Importance of evaluation - Forms of value - Group activity Evaluating Research Information
5. 5. Part 1 Importance of evaluation
6. 6. The need to evaluate information • Much training is about directing you to the right
information = searching and retrieval • As postgraduate researchers you have to be
critical and reflect on what you find. • Be aware of your impact on your own research,
and the research of others. • What defines your evaluative criteria?
7. 7. Ecology • Resources are interconnected and they evolve • Information resources are
transformed into knowledge • Knowledge becomes a resource • Therefore prior
knowledge shapes what we go on to create
8. 8. Role of the • In theory we can select almost any information to complete a task • In
practice we filter it by selecting resources we think most appropriate • Motivation affected by the learning we have already done
9. 9. Other factors • But, filtering is done for us BEFORE we get the chance to make a
judgement • People • Technologies • Cost • Skills • Copyright, IP
10. 10. Other factors • But, filtering is done for us BEFORE we get the chance to make a
judgement • People • Technologies • Cost • Skills • Copyright, IP
11. 11. Other factors • But, filtering is done for us BEFORE we get the chance to make a
judgement • People • Technologies • Cost • Skills • Copyright, IP
12. 12. Other factors • But, filtering is done for us BEFORE we get the chance to make a
judgement • People • Technologies • Cost • Skills • Copyright, IP
13. 13. Other factors • But, filtering is done for us BEFORE we get the chance to make a
judgement • People • Technologies • Cost • Skills • Copyright, IP
14. 14. Part 1 Summary • You need to be critical and reflect on all of the sources you find
and use. • You have a professional responsibility, as your research will impact on others.
• You are creating knowledge… … which evolves, and will shape what others create… …
similarly, the information you discover will shape the knowledge you create. • You need
to be aware of the filters already impacting upon the information you use.
15. 15. Forms of value
16. 16. Forms of Value • Filtering process = value judgement – By researcher – Made on
their behalf • What forms of value are there and how do they work together to create
information literate researcher?
17. 17. Objective form of value • Scientific measures of validity or reliability • Exists so that
personal values don‟t unduly influence work • Omit this scheme of value and we risk
information (and knowledge formed from it) becoming counterknowledge (Thompson,
2008).
18. 18. Intersubjective form of value • Based on the shared values of a community e.g.
morals, ethics, laws, economics • Allows for discussion of scientific method as it can‟t
explain everything • Acceptance in a community • If we omit this then values are
relativist
19. 19. Subjective form of value • Decisions you make – Is this what I want, do I need this, is
it relevant? • Privileges you as the researcher in the decision making process • If we
omit it we get groupthink (Janis, 1972) or battery cognition (Blaug, 2007) • Importance
therefore of asserting individual criticality
20. 20. Groupthink
21. 21. Cognitive biases • Subjectivity is vulnerable to bias & hunches • Concept of cognitive
bias was developed in 1970s by Tversky and Kahneman • Four main groups - Social Probability/belief - Memory - Decision making
22. 22. Social biases • Ascribe positive or negative traits to self, individuals or groups •
Loading values or anticipating action based on prior experience or a bias against self,
individuals or groups • Academic impact: need to verify information and not rely on
own views; important to remember when analysing human subjects
23. 23. Memory biases • How you perceive past events • False memory, positive memory,
imbalanced memory • E.g. A Photo, a Suggestion, a False Memory • Academic impact:
importance of accurate record keeping and note taking
24. 24. Memory biases
25. 25. Memory biases
26. 26. Probability and belief • To disregard or to pay too much attention to probability •
Academic impact: need to treat each research finding as distinct and to judge it in its
own right
27. 27. Decision-making biases • Influences on your decisions by own biases or those of a
group • Academic impact: need to be objective and consider all possible routes of
enquiry and treat all research findings as valid until proved otherwise e.g. Semmelweis
reflex
28. 28. Cognitive biases • On your table, group the forty cards into four piles of ten • Social •
Memory • Probability • Decision
29. 29. Three forms of value Subjective Inter- Objective subjective
30. 30. Part 2 Summary • Different concepts of „value‟ and the need to evaluate in terms of
each of these. - need to be objective and look at the measurable facts - need to be intersubjective and apply to the wider context - need to be subjective, and assess the value
based upon our own needs • Explored some of the key bias which may subconsciously
be impacting upon how your (subjective) measure of „value‟ might effect how you filter
information and your evaluative criteria.#
31. 31. Part 3 Evaluation of Research Information
32. 32. Evaluating information In a literature review you need to evaluate: • Relevance to
the topic • Authority of the author, publisher etc • Objectivity • Presentation • Method of
production and methodology • Currency
33. 33. Evaluating information In a literature review you need to evaluate: • Relevance to
the topic • Authority of the author, publisher etc • Objectivity • Presentation • Method of
production and methodology • Currency
34. 34. Relevance to the topic • Read the abstract, introduction or summary. • Scan the
bibliographic information which may highlight key subject areas not specifically alluded
to. • Emphasis may not be clear until you read in full.
35. 35. Relevance to the topic • Read the abstract, introduction or summary. • Scan the
bibliographic information which may highlight key subject areas not specifically alluded
to. • Emphasis may not be clear until you read in full. Be aware of what is filtering your
choices… - Vocabulary and broadness of interpretation. Are you under- estimating the
value of a source because it doesn‟t match your choice of keywords precisely?
36. 36. Authority • Are the authors acknowledged experts in the field? - frequently cited? have you or colleagues heard of them? - do they have an h-index? - can you find any
profile information where they work? • Where is it published? - Impact factors for a
journal (not always an accurate measure of quality, but potentially one of prestige)
37. 37. Authority • Are the authors acknowledged experts in the field? - frequently cited? have you or colleagues heard of them? - do they have an h-index? - can you find any
profile information where they work? • Where is it published? - Impact factors for a
journal (not always an accurate measure of quality, but potentially one of prestige) Be
38.
39.
40.
41.
42.
43.
44.
45.
46.
aware of what is filtering your choices y - Is the prestige of the author impacting on how
you evaluate the content?
38. Objectivity • Is the subject controversial? • If there are differing views on the subject
area, does the author consistently fall into one „camp‟?
39. Objectivity • Is the subject controversial? • If there are differing views on the subject
area, does the author consistently fall into one „camp‟? Be aware of what is filtering
your choices - Does the author demonstrate any hidden bias on the topic? - Evaluate
yourself? Are you subconsciously over-valuing the resource because it confirms your
own prejudices? Are you being objective?
40. Method of production • Is the article peer-reviewed? • Can you identify the
editor/editorial board for the publication?
41. Method of production • Is the article peer-reviewed? • Can you identify the
editor/editorial board for the publication? Be aware of what is filtering your choices… Be aware of editorial policy which may decide what is published.
42. 2.23.39 – 2.25.56 http://news.bbc.co.uk/1/hi/help/3681938.stm
43. Part 3 Summary • Various criteria you can assess a resource by. - a lot more „citation
‟ tools available for journal literature. • How much time do you realistically have?
44. Bibliography • Blaug, R. (2007) „Cognition in a hierarchy‟, Contemporary Political
Theory. 6: 24–44 • Goldacre, B. (2008) Bad science. London: Harper Press. • Janis, I.
(1972) Victims of groupthink: a psychological study of foreign-policy decisions and
fiascoes. Boston: Houghton Mifflin. • Kahneman, D. and Amos, A. (1972) „Subjective
probability: a judgment of representativeness‟, Cognitive Psychology. 3(3): 430-454. •
Luckin, R. (2010) Redesigning learning contexts: technology-rich, learner-centred
ecologies. Abingdon: Routledge. • Strange, D., Hayne, H. and Garry, M. (2008) ‟A photo, a
suggestion, a false memory‟, Applied Cognitive Psychology. 22: 587–603. • Thompson,
D. (2008) Counterknowledge: How We Surrendered to Conspiracy Theories, Quack
Medicine, Bogus Science and Fake History. London: Atlantic Books. • Whitworth, A.
(2009) Information Obesity. Oxford, UK: Chandos. In particular chapter 2. • Whitworth,
D (2010) “The three domains of value: Why IL practitioners must take a holistic
approach” Available at: http://prezi.com/rxqnzpoooolb/the-three-domains-of-valuewhy-il-practitioners-must-take-a- holistic-approach/ •
http://www.informationliteracy.ie/
45. Image Credits [Slide 5, 38] Via Flickr Creative Commons, by Martin LaBar. Original
available at http://www.flickr.com/photos/32454422@N00/163107859/ [Slide 3] Via
Flickr Creative Commons, by Kevin Dooley. Original available at
http://www.flickr.com/photos/12836528@N00/2577006675 [Slide 10] Via Flickr
Creative Commons, by shellorz. Original available at
http://www.flickr.com/photos/59198719@N00/2192821345 [Slide 13] Via Flickr
Creative Commons, by Richard Cocks. Original available at
http://www.flickr.com/photos/richardland/3999234316/ [Slide 12] Via Flickr Creative
Commons, by Photo Extremist. Original available at
http://www.flickr.com/photos/thevlue/4839060646/ [Slide 11] Via Flickr Creative
Commons, by What Dave Sees. Original available at
http://www.flickr.com/photos/whatdavesees/2487875504/
46. Image Credits [Slide 14] Via Flickr Creative Commons, by vl8189. Original available
at http://www.flickr.com/photos/27630470@N03/ [Slide 15] Via Flickr Creative
Commons, by opensourceway. Original available at
http://www.flickr.com/photos/47691521@N07/4371001458/ [Slide 17] Via Flickr
Creative Commons, by otherthings. Original available at
http://www.flickr.com/photos/18619970@N00/3057937540 [Slide 30-31] Photo
provided by colleague [Slide 54] ‘Vitae®, © 2010 Careers Research and Advisory Centre
(CRAC) Limited‘ Available at www.vitae.ac.uk/rdf
47. 47. Vitae Researcher Development Framework [see image credits] Measuring
Researcher Development
Critical Evaluation: Critical Reading & Critical Thinking
1. 1. Critical Evaluation Critical Reading Critical Thinking James Bisset
james.bisset@durham.ac.uk Academic Liaison Librarian (Research Support)
2. 2. Session outline - What is Critical Reading / Critical Thinking? - Definitions, Three
types of reading a text - Approaching a process for critical reading - Scanning/Skimming,
Critical Reading, Critical Thinking, SQ3R - Evaluation of Research Information - What to
look for as a critical reader when evaluating a text - Recognising your evaluative criteria
in your role as a researcher - What you bring to the table, self awareness and cognitive
bias
3. 3. Exercise 1 • Spend 5 minutes to read the short extract on your desks, and make some
brief notes which you would find useful to return to later to re-appraise yourself of the
text. Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available at
http://www.flickr.com/photos/theblackcanvas/2945878325/
4. 4. Part 1 What do we mean by critical reading & thinking?
5. 5. The non-critical reader - Reads a text as a source for... - memorising facts &
statements - repeating facts & statements - building a narrative around facts &
statements without analysing validity, reliability or applicability
6. 6. The critical reader - Reads a text as... - One interpretation of facts - Recognises the
importance of... - what a text says - how the text evidences and portrays the subject
matter
7. 7. Critical Reading “ Critical Reading involves understanding the content of a text as well
as how the subject matter is developed. Critical reading takes in the facts, but goes
further. “ http://www.rimt.edu.au/studyandlearningcentre/
8. 8. The critical thinker - Reads a text as... - One interpretation of facts - Recognises what a
text says and does… - applies own knowledge & values - to evaluate and interpret a
text’s overall meaning.
9. 9. Critical Thinking “Critical thinking involves reflecting on the validity of what you have
read in light of our prior knowledge and understanding of the
world.“ http://www.criticalreading.com/critical_reading_thinking.htm
10. 10. What a text says… (Restatement) What a text does… (Description) (how it says what
it says) What a text means… (Interpretation) (what it means to you, as the reader)
11. 11. What a text says… (Restatement) Restate the same topics and facts. What a text
does… (Description) Discuss the topics & facts within the context of how the original
argument was made. What a text means… (Interpretation) Interprets an overall
meaning within the wider context of the readers prior knowledge and values.
12. 12. Exercise 2 • Spend 5 minutes to read the following short [edited] extract and think
about: - what the text says - what the text does - what the text means Via Flickr Creative
Commons, by © Stuti Sakhalkar. Original available at
http://www.flickr.com/photos/theblackcanvas/2945878325/
13. 13. Goals of critical reading - recognise author's purpose - what is within the scope of
their writing, and what isn’t - what they are trying to do; does it match what you are
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
looking for - understand tone & persuasive elements of the argument - in contrast to the
objective data and evidence - what are they trying to sell you; what are you actually
being sold - recognise bias - identifying patterns of choice of content and language (eg
negative vs positive language, repeated omission or discounting)
14. Critical Reading & Thinking - Is not about: - being negative or finding fault. - It is
about: - assessing the strength of the evidence and the argument presented
15. Part 2 An efficient approach to critical reading:
16. Critical Reading & Thinking - "If we sense that assertations are ridiculous or
irresponsible (critical thinking), we examine the text more closely to test our
understanding (critical
reading)“ https://www.york.ac.uk/media/biology/documents/careers/critical_reading
_handout.pdf - Conversely, you can only think about a text critically if you have
understood it (critical reading) - to understand why we agree or disagree with an
alternative opinion, statement or conclusion. - to understand which issues we agree
and/or disagree with in an argument.
17. Critical Reading: Myth busting - You do not have the time to read everything. - You
do not have the time to read everything critically. - You must be selective. - Stay
focussed: get the info you need.
18. Adopt an efficient approach - Start with some basic principles - Quickly scan/skim
the material - [Critical Reading] Read more thoroughly and make notes - [Critical
Thinking] Consider/Review against your prior knowledge and understanding of the
topic
19. Some basics
20. Some basics - Most readers have an attention span of 15-20 minutes. - Be clear about
why you are reading the text. - Pause for thought - Don’t (always) read in isolation.
Discuss authors, publications, ideas and arguments with colleagues. - Usually, reading
the text once is not enough.
21. SQ3R
22. SQRRR - Survey - Question - Read - Recall - Review Scan / Skim text to provide an
overall impression of usefulness, scope, structure and argument.
23. SQRRR - Survey - Question - Read - Recall - Review Formulate questions you want to
answer / points you want to confirm through more in depth reading.
24. SQRRR - Survey - Question - Read - Recall - Review Read the text in a more focussed
way, aiming to answer the questions formulated.
25. SQRRR - Survey - Question - Read - Recall - Review Pause. Then test your
understanding and memory of the text, and if you feel you have addressed all your
questions raised.
26. SQRRR - Survey - Question - Read - Recall - Review You could also ‘test’ your
understanding by discussing with a colleague.
27. SQRRR - Survey - Question - Read - Recall - Review Return to the text. Read in more
detail, taking notes and identifying any further questions raised or left unanswered.
28. Survey Scanning / Skimming the text
29. Scanning text - Before reading in any detail - “Scan” your eye over the text quickly to identify specific words or phrases - to get a feel for structure • headings &
subheadings • figures, data, images • contents pages • index (for keywords) • reference
list • abstract / body text
30. Scanning text – why? - Note – requires you to have thought about why you are
reading / what you are looking for.
31. Scanning text – why? - Evaluate the relevance and usefulness - Make a judgment on
whether you should read further - Can help you to decide what parts of a document you
want to focus time on.
32. 32. Skimming text - Note – scanning the text first can help you decide where to
concentrate your time.
33. 33. Skimming text - Speed read to get an overview of - structure of text - scope and
content of the text • Note key points in summaries / abstract • Read 1st and last
paragraph / section to get main points • look at 1st sentence of each paragraph to get a
feel of content and thrust of argument.
34. 34. Exercise 3 • From a 30 second view, skim the text (focus on first and last lines of
each paragraph) to try and get an impression of the scope and content of the text. •
Holmes, P., Fay, R., Andrews, J., Attia, M., (2013) Researching multilingually: New
theoretical and methodological directions. International Journal of Applied Linguistics
23, 285–299. Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available at
http://www.flickr.com/photos/theblackcanvas/2945878325/
35. 35. Question Why are you reading the text?
36. 36. What are you looking for? - background info? - latest developments? - seek evidence
to support/refute an idea? - to reinforce your own prejudices? - to examine a
methodology or identify how a result was reached? - because you have to?
37. 37. Read
38. 38. Tips whilst reading - Identify core arguments - link evidence to any conclusions
drawn - identify arguments you feel are under-evidenced / purpose behind arguments
+ interpretation of author. - identify alternative conclusions which could have been
drawn.
39. 39. Tips whilst reading - Look for repetitions of argument, phrases or words to give
clues to authors intentions. - what do they consider crucial? - does this match what you
think is crucial?
40. 40. Recall & Review
41. 41. Tips for making notes - Make notes as you read... - throw away your highlighter annotate margins: key issues / questions raised - develop your own symbols... (AP) **
// !!
42. 42. Tips for making notes An argument should: - explain why the authors considered
what they are doing is worthwhile - explain the approach and methodology chosen explain why the data collected/material selected was most appropriate - how
conclusions drawn link to wider context
43. 43. Tips for making notes Broad definition of evidence: - encompasses what you read,
not just the data collected and presented - choice of methodology - context of data
collection / creation (eg sources used, scope of study/experiment) - rationale for
interpretations and conclusions drawn - relevance of theory underpinning argument
44. 44. Part 3 Evaluation of Research Information
45. 45. Evaluating information When reading critically you need to evaluate: • Relevance to
the topic • Authority of the author, publisher etc • Purpose / Objectivity • Presentation •
Methodology • Currency
46. 46. Relevance to the topic • Before reading the text… • Read the abstract, introduction or
summary. • Scan the bibliographic information which may highlight key subject areas
not specifically alluded to. • Emphasis may not be clear until you read in full.
47. 47. Relevance to the topic • Upon reading the text… • What level is the information at? •
Does it contain, and discuss in enough detail the information you are seeking? • Is the
research relevant to the subject domain / geographical area / demographic / time
period you are interested in?
48. 48. Relevance to the topic “In the course of a series of investigations into various aspects
of mental inheritance an intensive study has been made of so-called ‘identical’ twins.
The cases examined fall into two main groups: first, those reared together in their
parents' homes; secondly, those separated in early infancy, and brought up apart. With
the latter, despite wide differences in environmental conditions, the correlations for
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
intelligence, unlike those for school attainments, prove to be surprisingly high. It is
argued that this implies that ‘intelligence’, when adequately assessed, is largely
dependent on genetic constitution.” http://10.1111/j.2044-8295.1966.tb01014.x
“ Burt’s study of monozygotic twins reared apart … involved the largest number of
separated twin pairs at the time and produced the highest estimate of heritability for IQ”
49. Relevance to the topic Restatement: Burke identified a link between IQ and inherited
genes. Description: This article compares the evidence for IQ being determined by
inherited genes as opposed to IQ being affected by external environmental factors.
Interpretation: Evidence for IQ being an inherited trait rather than affected by external
factors has potential implications for the development of social and education policy.
50. Relevance to the topic • Key topics and ideas. • Level of information presented. •
Relevance in terms of location/subject/scope. • Does the content and level match your
needs. Be aware of what is filtering your choices… - Vocabulary and broadness of
interpretation. Are you under-estimating the value of a source because it doesn’t match
your choice of keywords precisely?
51. Authority “The trouble with quotes from the internet is that you never know if they
are genuine.” Abraham Lincoln (1809-1865, President of the United States of America).
52. Authority • Is it clear where the information comes from? • Are the authors
acknowledged experts in the field? - frequently cited? do they have an h-index? - have
you or colleagues heard of them? - can you find any profile information where they
work? - how well respected is the author, and their work, in their related field of
research? • Where is it published? - impact factors for a journal (not always an accurate
measure of quality, but potentially one of prestige) - is it peer reviewed?
53. Authority • Sir Cyril Burt • Fellow of British Academy • Author of over 350 articles
and a number of books. • “pioneer research on the inheritance of mental ability”
54. Authority – Citations
55. ””the most satisfactory attempt” to estimate hereditability of IQ” and “”the most
valuable” of all the separated twin research.” • Professor at the University of California,
Berkeley. • Author of 400+ peer reviewed papers. • In 2002, was listed in the Review of
General Psychology’s top 50 “most eminent psychologists of the 20th century” ”the
largest of its kind and the only one where “the distribution of children into foster homes
was random ” “ • Edge Pierce Professor of Psychology at Harvard University. • Author of
several peer reviewed papers and books. • Credited with discovering and developing
several models and theories as one of the founding researchers in the field of
quantitative analysis of behaviour. Richard J Herrnstein Arthur R Jenson
56. ”the only one of its kind in which the calculation of heritability had any meaning.” •
Professor of Psychology at the Institute of Psychiatry, King’s College, London. • In 2002,
was listed in the Review of General Psychology’s top 100 “most eminent psychologists of
the 20th century” as the most cited living psychologist at the time of his death. ””the best
data“ on separated twins.” • Alexander M. Poniatoff Professor of Engineering and
Applied Science at Stanford University. • Joint awarded Nobel Prize in Physics in 1956. •
1960’s and 1970’s moved also into area of hereditary behaviour. William B Shockley
Hans Eysenck
57. Authority – Impact Factors - British Journal of Mathematical and Statistical
Psychology - Impact Factor 1.258 - 5th of 13 journals in category “Psychology,
Mathematical” - British Journal of Educational Psychology - Impact Factor 2.093 - 11th
of 50 journals in category “Psychology, Educational” - British Journal of Psychology Impact Factor 2.103 - 26th of 126 journals in category “Psychology, Multi-disciplinary”
58. Authority • Are the authors acknowledged experts in the field? - frequently cited? do
they have an h-index? - have you or colleagues heard of them? - can you find any profile
information where they work? - how well respected is the author, and their work, in
their related field of research? • Where is it published? - Impact factors for a journal (not
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
always an accurate measure of quality, but potentially one of prestige) and is it peer
reviewed? Be aware of what is filtering your choices y - Is the prestige of the author or
the publication impacting on how you evaluate the content?
59. Objectivity • Is the subject controversial? • Does the author use emotive language? •
What is the authors purpose in writing the paper? • If there are differing views on the
subject area, does the author consistently fall into one ‘camp’?
60. Objectivity ** This is an over-simplification…. ** IQ is inherited IQ is affected by
external factors Burt, C (1943) “Ability and Income” British Journal of Educational
Psychology Burt, C.L. (1957) “Heredity and Intelligence; A reply to criticisms” British
Journal of Statistical Psychology Burt, C.L. (1958). "The inheritance of mental ability",
American Psychologist, Burt, C.L. (1972). "Inheritance of general intelligence", American
Psychologist, Burt C (1966) “The Genetic Determination of Differences in Intelligence: A
Study of Monozygotic Twins Reared Apart and Together.” British Journal of Psychology
61. Objectivity • Is the subject controversial? • Does the author use emotive language? •
What is the authors purpose in writing the paper? • If there are differing views on the
subject area, does the author consistently fall into one ‘camp’? Be aware of what is
filtering your choices - Does the author demonstrate any hidden bias on the topic? Evaluate yourself? Are you subconsciously over-valuing the resource because it
confirms your own prejudices? Are you being objective?
62. Methodology • are the methodology and sources of data used clearly identified or
explained? • what evidence is presented to support the ideas and conclusions
expressed? • is the methodology direct or indirect? • are the conclusions and
assumptions made by the author consistent, logical and justified? • does the research
raise any unanswered questions?
63. Methodology “ tests of the usual type… as a means of estimating genotypic
differences, even the most carefully constructed tests are highly fallible instruments,
and … their verdicts are far less trustworthy than the judgments of the pupil’s own
teachers.” Burt, C. (1957)British Journal of Statistical Psychology “the unaided
judgments even of the most experienced teachers … are nevertheless far less
trustworthy in the long run that the results obtained by properly applied intelligence
tests.” Burt, C. (1943)British Journal of Educational Psychology
64. Part 3 Summary • Various criteria you can assess a resource by. - a lot more ‘citation’
tools available for journal literature. • How much time do you realistically have?
65. Part 3 Summary • Some quotes and opinions were taken from one article: Tucker,
W.H.(1994) “Fact and fiction in the discovery of Sir Cyril Burt’s Flaws” Journal of the
History of the Behavioral Sciences (30). • Does that change your opinion on some of the
previous slides assertions and emphasis?
66. Part 4 Recognising your evaluative criteria
67. Critical Thinking “Critical thinking involves reflecting on the validity of what you
have read in light of our prior knowledge and understanding of the
world.“ http://www.criticalreading.com/critical_reading_thinking.htm
68. Ecology • Resources are interconnected and they evolve • Information resources are
transformed into knowledge • Knowledge becomes a resource • Therefore prior
knowledge shapes what we go on to create
69. Cognitive biases • Subjectivity is vulnerable to bias & hunches • Concept of cognitive
bias was developed in 1970s by Tversky and Kahneman • Four main groups - Social Probability/belief - Memory - Decision making
70. 1) “Hwang faked all research on human stem cells”
71. “Hwang faked all research on human stem cells” Halo effect “The tendency for a
person's positive or negative traits to "spill over" from one personality area to another
in others' perceptions of them”
72. 72. 2) Strange, Hayne & Garry (2008) ‘A photo, a suggestion, a false memory’ Applied
Cognitive Psychology. 22, 587–603.
73. 73. Strange, Hayne & Garry (2008) ‘A photo, a suggestion, a false memory’ Applied
Cognitive Psychology. 22, 587–603. False memory bias “A form of misattribution where
imagination, or incorrect recall, is mistaken for a memory.”
74. 74. Memory biases
75. 75. 3) “Once people become aware of an event or conclusion, they will usually express a
good deal of confidence about the predictability of the outcome.”
76. 76. Fischhoff, B., and Beyth, R. (1975) ‘"I knew it would happen" Remembered
probabilities of once-future things’ Organizational Behaviour and Human Performance.
13, 1-16. Hindsight bias “The inclination to see past events as being more predictable
than they actually were”
77. 77. Exercise 4 • Read through the example “cognitive bias” on the cards, and try to
decide which of the four “groups” of cognitive bias each might sit in. - Social (Ascribe
positive or negative traits to self, individuals or groups) - Probability/belief (disregard
or to pay too much attention to probability) - Memory (How you perceive past events) Decision making (Influences on your decisions by own or group biases) Via Flickr
Creative Commons, by © Stuti Sakhalkar. Original available at
http://www.flickr.com/photos/theblackcanvas/2945878325/
78. 78. Cognitive biases • Shape how we read, interpret and evaluate the information we
receive. • Shape how others read, interpret and evaluate the information they receive. •
Any published research is just one interpretation of the facts available
79. 79. Further Reading • Tucker, W.H.(1994) “Fact and fiction in the discovery of Sir Cyril
Burt’s Flaws” Journal of the History of the Behavioral Sciences (30). • University of
Leicester: What is Critical Reading
http://www2.le.ac.uk/offices/ld/resources/writing/writing-resources/critical-reading
• University of Newcastle: Think Critically http://www.ncl.ac.uk/library/researchsupport/informed-researcher/think-critically/ • Open University: Critical Reading
Techniques http://www.open.ac.uk/skillsforstudy/critical-reading-techniques.php/ •
Dan Kurland’s Critical Reading Website http://www.criticalreading.com/ • The SQ3R
method http://www.ic.arizona.edu/ic/wrightr/other/sq3r.html
80. 80. Image Credits [69] Via Flickr Creative Commons, by Martin LaBar. Original available
at http://www.flickr.com/photos/32454422@N00/163107859/ [4,44,67] Via Flickr
Creative Commons, by Kevin Dooley. Available at
http://www.flickr.com/photos/12836528@N00/2577006675 [75] Via Flickr Creative
Commons, and by Brian Yap: Original available at
http://www.flickr.com/photos/30265340@N00/465957804 [15] Via Flickr Creative
Commons, and by FutUndBeidl: Available at
http://www.flickr.com/photos/61423903@N06/7369580478
For example, one common bias is that women are weak (despite many being very
strong). Another is that blacks are dishonest (when most aren't). Another is that obese
people are lazy (when their weight may be due to any of a range of factors, including
disease). People often are not aware of their biases.
I've flipped heads with this coin five times consecutively, so the chance of tails
coming out on the sixth flip is much greater than heads."
Example: when researching startups, you’ll read about the most profitable new
companies, but because you’re not spending the same amount of time studying the
companies that failed, you’ll come away with an inaccurate idea of the probability of
success.
“all birds have wings, and planes have wings, therefore
planes are birds”.
remembering one's exam grades as being better than they were, or remembering a caught
fish as
An example would be in an election. Early on, members within one political party
typically argue with each other and split into factions supporting different candidates
within the same party.
For example, you might subconsciously make selective use of data, or you might
feel pressured to make a decision by powerful colleagues.
Answer all questions by clicking on the button next to the answer under each
question. Questions without answers will be scored as incorrect. You
may review the Critical Thinking course materials at any time (this link will
open up another window, which you can close when done to return here).
1. What does it mean to think critically?
To criticize things for their shortcomings
To think actively, with an awareness of potential problems in the
information you encounter
To consider things in a logical manner
To imagine what it would be like to be criticized
2. Why is it important to evaluate any information you encounter?
Information may contain defects or may be incomplete
Information may contain answers to my questions
Information may be difficult to understand
Information may not be helpful
3. What question should you have in mind when you encounter
information?
Is it current?
Is it accurate?
Is the person providing the information an authority
All of the above
4. How do you know if information found satisfies your needs?
I found everything I needed by consulting one source
It looks like it could be true
I feel like the author is telling the truth
After consulting several sources, the information I found completely
addresses my research topic
5. What are we concerned with when we ask if a person providing
information is an authority?
Whether that person is in charge of people
If that person knows who the author is
If that person has relevant expertise in the subject being presented
If the author knows something no other author knows
6. What does it mean if information is current?
It's up-to-date
It's part of a new fad
It's a hot topic
It's probably outdated or obsolete
7. Why might commercial sources of information present problems for
researchers?
Most researchers don't have credit cards
Advertisements reduce the amount of information you can see on
the screen
You might be asked to buy something you don't want
You might not get the best information available, only that which
some company has paid to have shown to you
8. What is an example of a motive to providing information?
Someone might be trying to sell you something
Someone believes he or she has valuable knowledge or experience
worth sharing with other people
Someone wants to make information freely available in order to help
people
All of the above
9. Why should you care if the information you're examining is
complete?
You might need to pursue another research topic instead
You wouldn't be able to list it in a bibliography
You might not have all the facts
You really don't need to know all the facts
10. Why should you always try to find better sources of information?
Because if you do not seek out superior sources your understanding
of a subject might be incomplete or incorrect
Because you can never use old or historical information
Because only the newest information available can ever be trusted
Because inferior sources take too long to document in a research
paper
Critical Thinking Quiz (thrall.org)
https://www.thrall.org/criticalthinking/quiz.htm
Answer all questions by clicking on the button next to the answer under each question.
Questions without answers will be scored as incorrect. You may review the Critical Thinking
course materials at any time (this link will open up another window, which you can close when
done to return here).
What does it mean to think critically?
To criticize things for their shortcomings
To think actively, with an awareness of potential problems in the information you encounter
To consider things in a logical manner
To imagine what it would be like to be criticized
Why is it important to evaluate any information you encounter?
Information may contain defects or may be incomplete
Information may contain answers to my questions
Information may be difficult to understand
Information may not be helpful
What question should you have in mind when you encounter information?
Is it current?
Is it accurate?
Is the person providing the information an authority
All of the above
How do you know if information found satisfies your needs?
I found everything I needed by consulting one source
It looks like it could be true
I feel like the author is telling the truth
After consulting several sources, the information I found completely addresses my research
topic
What are we concerned with when we ask if a person providing information is an authority?
Whether that person is in charge of people
If that person knows who the author is
If that person has relevant expertise in the subject being presented
If the author knows something no other author knows
What does it mean if information is current?
It's up-to-date
It's part of a new fad
It's a hot topic
It's probably outdated or obsolete
Why might commercial sources of information present problems for researchers?
Most researchers don't have credit cards
Advertisements reduce the amount of information you can see on the screen
You might be asked to buy something you don't want
You might not get the best information available, only that which some company has paid to
have shown to you
What is an example of a motive to providing information?
Someone might be trying to sell you something
Someone believes he or she has valuable knowledge or experience worth sharing with other
people
Someone wants to make information freely available in order to help people
All of the above
Why should you care if the information you're examining is complete?
You might need to pursue another research topic instead
You wouldn't be able to list it in a bibliography
You might not have all the facts
You really don't need to know all the facts
Why should you always try to find better sources of information?
Because if you do not seek out superior sources your understanding of a subject might be
incomplete or incorrect
Because you can never use old or historical information
Because only the newest information available can ever be trusted
Because inferior sources take too long to document in a research paper