Using Portfolio Assessment to Discover Student Learning

advertisement

Using Portfolio Assessment to

Discover Student Learning

Barbara J. D’Angelo, Ph.D.

Arizona State University

Association for Business Communication

Annual Convention

November 2009

Background: Written Communication for Managers

 Junior-level business writing course

 Multi-section course

7-10 sections per semester + summer

On-campus, hybrid, and online

 Redesigned Summer 2007

Industry focus groups

Business and writing faculty input

Role playing scenario: students create a company

Standardized syllabus and assignments

 Portfolio for assessment

End-of-semester self-evaluation of learning

Scored by 1 writing faculty and 1 business faculty

Barbara J. D'Angelo, ABC Convention November 2009

2

Research Questions

 Which assignments do students choose to include in their portfolios?

 How do students organize their portfolios as an argument to demonstrate their learning?

What persuasive techniques do students use to support their argument for learning?

What tone and style do students use?

Do students present an argument for their learning or to enhance their grade?

Barbara J. D'Angelo, ABC Convention November 2009

3

Methods

Random sampling

 3 portfolios from each section Fall 2008 and Spring 2009

 34 portfolios analyzed

Simple counting of assignments

Content analysis of narrative statements

 Categories derived from

Bower (2003): rhetorical appeals and argumentation

Scott (2005): form, tone, and appeal

Post-semester feedback from instructors

See proceedings for details (or talk to me later or email me)

Barbara J. D'Angelo, ABC Convention November 2009

4

Assignments Included in

Portfolios

Fall 2008 Spring 2009

Memo

Manager Roundtable

Proposal

“Branding”

Letter

Policy

Audience Analysis

Agenda

Email

Peer Review

Research Summary

Minutes

8

9

6

4

11

10

10

9

3

2

4

4

14

10

1

7

14

12

10

11

3

7

5

5

Barbara J. D'Angelo, ABC Convention November 2009

5

Assignments

Assignment inclusion not related to grading scale

Inclusion seems based on

 Substantiality of writing or traditional genres

 Instructor emphasis: instructor feedback indicates this as high

De-emphasis of meeting documents

 Lack of understanding or emphasis

 Increased emphasis during spring = higher inclusion = awareness of importance?

Process documents

 Reluctance to show drafts or other “in progress” work?

Barbara J. D'Angelo, ABC Convention November 2009

6

Portfolio Organization

 26 organized generically

Introduction: reflection about course

Body: described artifacts, claims and evidence

Conclusion: summed up

 7 organized contextually

Body contained more context

 Best vs. worst

 Strengths vs. weaknesses

 Course objectives

 Had no influence on raters’ scores

Barbara J. D'Angelo, ABC Convention November 2009

7

Persuasive techniques

# of Portfolios # of Statements

Ethos

Logos

Pathos

Nomos

14

23

17

21

19

51

29

31

Skills Supported

Skills Unsupported

23

18

64

39

Barbara J. D'Angelo, ABC Convention November 2009

8

Use of Persuasion

 Nomos and pathos not linked to “schmooze”

 Not directly linked to a claim or evidence

 More expression of student “feelings”

 Awareness of audience and course values

 Majority (23) of students supported claims

 23 used logos to create claim with evidence to support it

13 supported all claims

Unlike findings with FYC students

 Indication of more sophisticated writing from upper level students?

 8 did not support any claims

Barbara J. D'Angelo, ABC Convention November 2009

9

Tone and Style

24 students used positive tone

Excitement about role playing (pathos)

 3 students used negative tone

Dis-satisfied with grade or with instructor?

 7 students used intermediate tone

Neutral descriptive tone

 All students changed from “business” style to “student” style when completing the portfolio

Barbara J. D'Angelo, ABC Convention November 2009

10

Learning or enhance grade?

Learning

 28 students made statements related to their growth

 Only 1 student directly referred to their grade

 Use of logos and support for claims indicates argument was for learning

Enhance grade

 19 portfolios potentially included “schmooze”

2 flattered course or instructor

Rest thanked instructor

 2 students appealed directly to instructor (using “you”)

Result differs from FYC studies

Barbara J. D'Angelo, ABC Convention November 2009

11

Use for course improvement

More emphasis on meeting documents

Importance to our employer stakeholders

Additional requirement to write agendas added for each Manager

Roundtable

Requirement to evaluate agendas

 Change MRT topics to more completely support written assignments

Instructors believed not enough time to go over concepts needed for assignments

 Strengthen research summary to emphasize tie between finding information-analysis-use for persuasion

Barbara J. D'Angelo, ABC Convention November 2009

12

Future

 Potential change: portfolio scenario to shareholder or other external audience

 Maintain emphasis on business style

Continue to de-emphasize mechanics

But help needed for ESL students

Continue assessment to aggregate additional data

Confirmation of findings related to use of logos/evidence

 Add industry evaluators to assessment to include external stakeholders

Barbara J. D'Angelo, ABC Convention November 2009

13

Questions?

Presentation available at: http://www.public.asu.edu/~bdangelo/presentations/abc09.ppt

Course syllabus and assignments available at: http://techcomm.asu.edu/curriculum/twc347

Email: bdangelo@asu.edu

Barbara J. D'Angelo, ABC Convention November 2009

14

Download