ANNUAL ASSESSMENT REPORT FOR 2010-11 [Normal due date will be October 1 each year.] College of Business / Masters of Business Administration (MBA): October 1, 2011: Authors: Craig Caldwell, Roberto Curci, Kathy Paulson Gjerde, Robert Mackoy, Bill O’Donnell. (Please list everyone involved in your department’s annual assessment review.) Student Learning Outcome (SLO) For each SLO, list two methodologies and the criteria for successful performance (such as a measurement, rubric or scale that indicates a baseline for competency). Methodology 1 1. Identify the A case in fundamental drivers of MBA545 business, and show how Capstone integrated business Experience knowledge can be used Course to solve business problems. 2. Identify the dynamics A case in affecting businesses in MBA545 the international Capstone economy. Experience Course Team assessment 3. Display the ability to instrument in work effectively in MBA505 and teams. MBA535 Criteria for Success At least 80 percent of MBA students should score at least in the satisfactory category. Methodology 2 At least 80 percent of MBA students should score at least in the satisfactory category. Assignment in Gcredit courses (MBA562, MBA552, MBA581) At least 80 percent of MBA students should score at least in the satisfactory category. M1: F10 M2: S11 At least 80 percent of MBA students should score at least in the satisfactory category. 360 assessment instrument in MBA510 At least 80 percent of MBA students should score at least in the satisfactory category. M1: S11 and Sum11 M2: Not assessed (anticipated F11) Exit interview Criteria for Success At least 80 percent of MBA students should score at least in the satisfactory category. Term Assessed (F10 or S11) M1: F10 M2: Not assessed (anticipated S12) 4. Practice high ethical standards in their decision-making. 5. Accept and support that organizational change is a natural business process. Assessment instrument in MBA515 At least 80 percent of MBA Assessment students should score at least in instrument in the satisfactory category. MBA525 Team assessment instrument in MBA505 and MBA535 At least 80 percent of MBA students should score at least in the satisfactory category. 360 assessment instrument in MBA510 At least 80 percent of MBA M1: Not students should score at least in assessed the satisfactory category. (anticipated F11) M2: Not assessed (anticipated F11) At least 80 percent of MBA M1: S11 and Sum11 students should score at M2: Not least in the satisfactory assessed (anticipated F11) 1. Findings—Summarize the findings from the assessment activities for each SLO that was assessed. Identify the SLO # and append supporting documentation such as rubrics, scales, pass rates, test scores, or other measurements used to assess each SLO. ACTIVITY 1: SOURCE OF DATA: Data are derived from written case reports, which were incorporated as a graded assignment in MBA545. The assignment represented 10% of the course grade and was due in December of Fall 2010 semester. See Appendix A for assignment guidelines and Appendix B for rubrics. SLO 1: Identify fundamental drivers of business, and integrate such business knowledge to solve business problems. # % Students Students Excellent (30-40) 0 0% Satisfactory (15-29) 5 38% Unsatisfactory (0-14) 8 62% SLO 2: Identify the dynamics affecting businesses in the international economy # % Students Students Excellent (18-24) 0 0% Satisfactory (9-17) 4 31% Unsatisfactory (0-8) 9 69% OBSERVATIONS: A comparison of results for Fall 2008 and Fall 2010 suggests that students’ performance on SLO 1 has deteriorated. This is to be expected, however, given that the rubric was revised between these two years in an effort to increase the rigor of the standards and place greater emphasis on integration. Under the new rubric, each case report is evaluated on five characteristics: #1 Identifies and considers appropriate data, contexts, and assumptions relevant to the situation #2 Assesses and analyzes appropriate data/evidence using appropriate set of tools and methods. #3 Integrates across functional areas of business #4 Identifies, assesses, and evaluates alternate options. #5 Identifies and assesses conclusions, implications, and consequences Students, in general, perform in the “Satisfactory” range for characteristics 1, 2, and 5. Where students seem to have difficulty is in integrating across functional areas (characteristic 3) and fully evaluating alternative courses of action (characteristic 4). It is important to note that, overall, the case reports submitted in Fall 2010 were much better than those submitted in Fall 2008 in terms of level of detail, particularly in the area of financial analysis. This improvement is likely the result of the case report being a graded assignment with more detailed guidelines. A comparison of results for Fall 2008 and Fall 2010 suggest that students’ performance on SLO 2 has improved slightly. Under this rubric, each case report is evaluated on three characteristics: #1 Visualizes the world as the potential market for his/her products or services. #2 Recognizes global business and cross-cultural factors applicable in a given context. #3 Applies understanding of global business and cross-cultural influences while generating options/decisions/strategies. Students, in general, perform in the “Satisfactory” range for characteristics 1. Where students seem to have difficulty is recognizing and applying global business and cross-cultural factors (characteristics 2 and 3). What this means is that students typically identify foreign markets as a source of cheap labor (i.e. outsourcing) or as a place to sell their goods and services, but demonstrate almost no understanding of what foreign market entry actually entails. Note that these results are slightly better than those reported for Spring 2011 using Method 2. This is not surprising given the more detailed instructions in the assignment guidelines. ACTIVITY 2: SOURCES OF DATA: Data are derived from peer evaluations. Specifically, team members were asked to evaluate each of their team members using a standardized assessment instrument (see Appendix C). Question 9 was used to evaluate SLO 3 and the mean of Questions 5 and 6 was used to evaluate SLO 5. The average ratings across all team members’ assessment instruments were used as that individual’s score. RUBRIC: For SLO 3, an average rating greater than or equal to 95 was categorized as “Excellent,” and average rating greater than or equal to 85 but less than 95 was categorized as “Satisfactory,” and an average rating less than 85 was categorized as “Unsatisfactory.” For SLO 5, an average rating greater than or equal to 95 was categorized as “Excellent,” and average rating greater than or equal to 85 but less than 95 was categorized as “Satisfactory,” and an average rating less than 85 was categorized as “Unsatisfactory.” SLO 3: Display the ability to work effectively in teams. MBA505 Spring 2011 Excellent Satisfactory Unsatisfactory MBA505 Summer 2011 MBA505 Overall MBA535 Spring 2011 # Students % Students # Students % Students # Students % Students # Students % Students 8 18 1 29.6% 66.7% 3.7% 7 11 1 36.8% 57.9% 5.3% 15 29 2 32.6% 63.0% 4.4% 21 7 2 70.0% 23.3% 6.7% SLO 5: Accept and support that organizational change is a natural business process MBA505 Spring 2011 Excellent Satisfactory Unsatisfactory MBA505 Summer 2011 MBA505 Overall MBA535 Spring 2011 # Students % Students # Students % Students # Students % Students # Students % Students 6 15 6 22.2% 55.6% 22.2% 0 17 2 0.0% 89.5% 10.5% 6 32 8 13.0% 69.6% 17.4% 23 6 1 76.7% 20.0% 3.3% OBSERVATIONS: MBA505 is the first 500-level course taken by MBA students and, thus, represents where students are in terms of SLO3 and SLO5 upon starting their graduate coursework. We note that for both SLO3 and SLO5, the majority of students are evaluated as being in the Satisfactory or Excellent range, with the greatest percentage in the Satisfactory range (63.0% for SLO3 and 69.6% for SLO5). This is not surprising given that most of our MBA students are working professionals with, on average, 6-7 years of work experience. Thus, working in teams and being a part of organizational change are not new experiences for them. Our program goal is to simply help students refine these skills, particularly those students who are evaluated as Unsatisfactory in MBA505. MBA535 is a core course but because our program is not lockstep, the point in their course of study at which students enroll in this course varies. Although some students take this course immediately after MBA505, others may wait several semesters before taking it. Thus, student scores in MBA535 represent where students are on SLO3 and SLO5 mid-program. Again, we note that for both SLO3 and SLO5, the majority of students are evaluated as being in the Satisfactory or Excellent range, with the greatest percentage in the Excellent range (70.0% for SLO3 and 76.7% for SLO5). Part of the reason we observe an increase in the number of students being evaluate as Excellent in MBA535 compared to MBA505 may be that the team experience in this course centers on a dynamic simulation in which students are engaged over the course of an entire semester. In this setting, students are more naturally able to move through the stages of team development and, in addition, are required to adapt to changing market conditions. Nonetheless, this improvement in performance on SLO3 and SLO5 suggests that our curriculum has a positive impact on students’ skills in these two areas. ACTIVITY 3: SOURCE OF DATA: Data are derived from in-class exercise administered in Spring 2011 in two graduate core classes and two International Business electives. The students did not receive class credit for completing this exercise and were not asked to engage in any outside preparation. See Appendix D for the in-class exercise. See Appendix E for the rubric. SLO 2: Identify the dynamics affecting businesses in the international economy # % Students Students Excellent (18-24) 0 0% Satisfactory (9-17) 15 16% Unsatisfactory (0-8) 79 86% OBSERVATIONS: Overall, we find these results to be weak. There is a need to increase the level of exposure of our MBA students to international business issues; this can be accomplished through in-class discussions and “Real Life Real Business” experiential activities such as MBA study trips abroad. Hopefully, an increased level of exposure to international business issues will make our MBA students more globally minded and likely to think about business from an international perspective. 2. Use of Results—What programmatic changes, if any, were made in response to the findings? Reference the SLO #. SLO1: Since spring 2009, 20 COB faculty members have attended the Harvard Case Workshop in order to become more consistent and effective in using the case methodology in class. The next step would be to further develop these skills by creating a “best practices in case teaching” seminar or workshop series for COB faculty. Review course description for MBA545 in order to determine how best to structure the course going forward. At present, there is a lot packed into this course (e.g. review of functional areas of business, strategy, integration, consulting project). We may want to focus our attention in order to build skill development in only a few key areas. SLO2: Continue to offer MBA international study trips. Specifically, offer 2-credit international study trip to Malaysia in Summer 2012 and AIB section of MBA545 focusing on India in Fall 2012. SLO3 and SLO5: Develop a 1-2 page memo which highlights the team takeaways from MBA505 and incorporates the additional team insights emphasized in MBA535. This memo would be given to students in MBA535 to reinforce and build upon the team concepts learned in MBA505. It would help us to explicitly coordinate team elements between these two courses. Based on the results of the syllabus mapping, this same memo may be incorporated in other courses as well. Incorporate same assessment instrument in MBA545 in order to evaluate student performance on SLO3 and SLO5 at the end of their course of study. Create a master database which tracks each student’s performance on this assessment instrument at beginning (MBA505), middle (MBA535), and end (MBA545) of curriculum in order to measure degree of skill development throughout MBA program. 3. What support services or resources for faculty would help your department assess its SLOs better? 4. What revisions, if any, to current SLOs did you make or are under consideration? SLO5: Change wording of SLO 5 (e.g. Identify strategies for managing or implementing organizational change) is currently under consideration. 5. Map each of your program’s SLOs to the University Learning Outcomes. Make annual updates only if your SLOs changed. For example: Butler University students will: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. Explore various ways of knowing in the humanities, social and natural sciences, quantitative and analytic reasoning, and creative arts. (Know) Articulate and apply required content knowledge within their area(s) of study. (Know) Find, understand, analyze, synthesize, evaluate and use information, employing technology as appropriate. (Know) Explore a variety of cultures. (Know) Recognize the relationship between the natural world and broader societal issues. (Know) Communicate clearly and effectively. (Do) Demonstrate collaborative behavior with others. (Do) Practice ways and means of physical well-being. (Do) Acquire the skills to make informed, rational and ethical choices. (Do) Experience diverse cultures, ethnicities, religions and sexual orientations. (Value) Share their talents with Butler and the greater community at large. (Value) Be exposed to the value of lifelong learning. (Value) MBA Program Student Learning Outcomes: Butler University Learning Outcomes 1 Identify the fundamental drivers of business, and show how integrated business knowledge can be used to solve business problems. 2. Identify the dynamics affecting businesses in the international economy. 3. Display the ability to work effectively in teams. 1. 3 4 5 X X X 6 7 X X X X X X X 8 9 X X X X X X X X 10 X X X 4. Practice high ethical standards in their decision-making. 5. Accept and support that organizational change is a natural business process. 2 X 11 12 6a. List all courses in the program and map each of your SLOs to the curriculum in which the learning occurs, indicating the extent to which the outcome is introduced (I) or refined (R). [Make annual updates only if your SLOs or curriculum changed.] MBA505 MBA510 MBA515 MBA520 MBA525 MBA530 MBA535 MBA540 MBA545 MBA551 MBA552 MBA553 MBA556 MBA557 MBA561 MBA562 MBA563 MBA564 MBA565 MBA567 MBA571 MBA573 MBA581 MBA582 MBA583 MBA587 SLO 1 I I I I I I I I R R R R R R R R R R R R R R R R R SLO 2 SLO 3 I I I I I I R R R R R R R R R R R R SLO 4 I I SLO 5 I I I I I I R R R R R R R I I I R R R I R R R R R R R R R R R R R R R R 6b. Learning/developmental opportunities for students outside the classroom—If any SLO was addressed outside the classroom, explain where and how the learning/developmental opportunities were provided to students in your program? (i.e., internships, field experiences, visiting lectures, collaborative projects, and other creative ideas you may have employed.) Appendix A Case Report Assignment Guidelines October 27, 2010 To: MBA545 Colleagues From: Dick Fetter Re: Ducati Case I have a strong desire to demonstrate to the MBA Committee that your outstanding analyses are surpassed only by your exceptional writing… Greetings to all. Recall that one aspect of this class is to analyze/submit a written analysis of a case that is provided by the MBA Committee. While this analysis comprises a portion of your grade, the MBA Committee also uses your written case analysis as part of its assessment of student learning, not for MBA545 but for your entire MBA learning experience. The case for this class is: Ducati: In Pursuit of Magic (A) This case is a bit dated (i.e., old), with actual financials running through 2004 and estimates for 2005. While this is a bit of a limitation, it is nevertheless a good case in that it allows us to cross over many of the functional areas of business covered in the MBA core curriculum. Why Write the Ducati Case: My main intents in having you write your case analysis are to: 1. Fulfill the MBA Committee’s evidence of “student learning outcomes” (SLOs) requirement. This SLO assessment is not for MBA545, but for your whole MBA learning experience. The MBA Committee just uses MBA545 as the platform for this assessment process due to (1) final core MBA class and (2) I don’t mind yielding part of the class to this important exercise (assess SLOs). This is important for accreditation purposes, but it also important for our (COB) continual learning initiatives. I want you to demonstrate your ability to cross over at least four, preferably 5-6 of the functional areas of business (leadership, law/ethics, etc.). In addition, I want you to use the business strategy material that we discussed early in the semester, integrating it with the functional areas of business, as well. 2. Demonstrate your ability to address the functional areas of business “somewhat” integratively. I include the business strategy material in this integrative approach, as well. That is, simultaneously intertwine 2-4 functional areas of business. Contrast this (integrated) approach with addressing functional 3. 4. 5. 6. areas serially – i.e., first talk about leadership…then address law…then operations management, etc. My point here is that I DO NOT want you to address the functional areas of business individually, or in isolation by themselves. Perform some financial analyses…and back-up your recommendations with financial evidence. If you are lacking “the numbers” that you need, then still build the model, tell me what numbers you are missing, tell me how you would use them, etc. I will consider it a lame excuse to use a phrase such as, “…I would have analyzed this financially, but I did not have the right numbers in the case…” I place special emphasis on “model building,” as this shows how you are thinking and how you would use your financial skills to assess a situation and/or support a recommendation. Provide evidence of your ability to address global issues. Some of you are highly engaged in global business; others of you might not have ever dealt with global matters. So, I am not looking for exact consistency across all of your papers; but, I want you to weave in something(s) related to global business. Write very well – this matters to me. No doubt, the content of your paper is important, but the structure of your paper and the overall quality of your writing also matters…a lot. I will work with you on this over the next few weeks, but keep in mind my comments earlier in the semester about writing briefly/succinctly, making papers “skimmable,” avoid one double-spaced page after another, just droning on and on and on…Make your papers easy-to-read and interesting. The key points on each page should be easy to find…literally “jumps off the page” at the reader. The executive summary matters!! I prefer that it not exceed one page, but if it spills over to two pages, I’ll not lose any sleep… Finally, try writing in the active voice – sometimes this is a bit awkward with case analyses, but if you are one of the many…many…who naturally write in passive voice, I want you to get some experience/practice at active-voice writing in this class. You will submit individual papers, but it is okay to talk with your colleagues. Remember, we share the responsibility not only to help one another cross the line (successfully complete MBA545), but to help maximize everyone’s take away. So, it’s okay to share…but you must write your own paper. Topics to Address: You have much latitude in this paper, with a few firm guidelines (as specified in the prior section) to follow. Here are some other topics on which I would encourage you to write: 1. Explain how Ducati created and captured value during the life of the case (i.e., not obliged to address this beyond 2005). 2. Identify the key issues that Ducati/Minoli had to address in creating the value that you identify in #1; what issues remain to be addressed? 3. Identify some “plausible alternatives” that would allow Ducati to build on your value creation discussion and the issues that need to be addressed. 4. Give me some guidance on how you would go about choosing amongst the plausible alternatives that you identify in #3. Harley-Davidson Comparison: Additionally, I want you to write an additional section in your case analysis comparing Ducati to Harley Davidson. The Ducati case provides us some financial info on the company; Harley is a publicly traded company, and I am pretty sure that we can get data back to the late 1990s/early 2000s on Harley that will match up with the Ducati financials. Ducati was sold in about 2007, so there is probably only scant financial info available on the company over the past few years. My key intent on the Ducati-vs.-Harley is to give you the chance to demonstrate your knowledge of the functional areas of business (integratively). I do not know much about motor cycles. I have never ridden one and have not plans to break my 58-year habit of only riding vehicles with four wheels and a windshield. I am an automobile racing fan, but I don’t follow motor cycle racing, nor do I particularly enjoy it. Nevertheless, I find the “business side” of the Ducati vs. Harley case very interesting. They (Ducati vs. Harley) seem to come from different worlds and target dramatically different audiences; but, the reality is that, at some point, motorcycles are motorcycles… Length of the Paper: In the past I have assigned an 8-page maximum (12-point font with one-inch margins and page numbers), plus appendices. Given I asked you to also write briefly about Harley, your paper may comprise 10 pages, plus appendices and executive summary. If your paper is longer than 10-pages, reread it and be a “tough reviewer” of your text. Whittle it down…try switching from passive to active voice (almost always saves text). Don’t “playback” the facts of the case to me…I already read the case. One key point here – I actually think that you could write an excellent paper in 5-pages, plus appendices and executive summary, so do not feel obliged to “fill space.” If I think you are just using up space, I will lower my assessment of your paper. Concluding Comments: I ask that you submit your written case analysis by December 1. We will talk about it in class each of the next few weeks to help bring everybody up to speed. As I indicated in my opening comment to this memo, I have a strong desire to demonstrate to the MBA Committee that your outstanding analyses are surpassed only by your outstanding writing… Let me know if you have questions about this assignment. Thanks--df Appendix B Rubrics Rubric: Revised by: Kathy Gjerde Program MBA SLO #1: Identify the fundamental drivers of business, and integrate such business knowledge to solve business problems. Instrument/Method: MBA545 Case Study Evaluators: Kathy Gjerde Semester: Fall 2010 Criteria for Success: 1 Characteristic Identifies and considers appropriate data, contexts, and assumptions relevant to the situation. Standards Used to Evaluate Student Performance Unsatisfactory Satisfactory (0 to 2 points) (3 to 5 points) Does not identify both internal and Identifies both internal and external external environmental factors and environmental factors and data discussed data discussed in case. in case, but merely repeats this information with no additional insight or data manipulation. 2 Assesses and analyzes appropriate data/evidence using appropriate set of tools and methods. Critically interprets data by correctly referencing theory/framework/methodological tool from 3 or fewer functional areas. Critically interprets data by correctly referencing theory/framework/methodological tool from at least four functional areas. 3 Integrates across functional areas of business Does not integrate analysis across more than one functional area of business. Integrates analysis across two functional areas of business, specifically taking into account the inherent tradeoffs. 4 Identifies, assesses and evaluates alternate options. Identifies only one realistic option or doesn’t provide both pros and cons for each option Identifies at least two realistic options and provides both pros and cons for each option. 5 Identifies and assesses conclusions, implications, and consequences. Does not state a conclusion or stated conclusion is inconsistent with analysis. Clearly states a conclusion or recommendation that follows from the analysis. 15 Excellent (6 to 8 points) Identifies both internal and external environmental factors and data discussed in case and builds on these facts by stating additional insights, manipulating the data, or incorporating data/information from outside sources. Critically interprets data by correctly referencing theory/framework/methodologic al tool from at least four functional areas, with at least one of these tools being explored or applied in depth. Integrates analysis across more than two functional areas of business, specifically taking into account the inherent tradeoffs. Identifies at least two realistic options and provides both pros and cons for each option. Pros/cons are clearly linked to the case data, assumptions, inferences, and formal issues of case. Clearly states a conclusion or recommendation that follows from the analysis and explains how this Score recommendation was chosen from the list of realistic options. Total Score Performance Level (0 to 14 is unsatisfactory, 15 to 29 is satisfactory, 30 to 40 is excellent)* 16 Rubric: Created by: Roberto Curci Program MBA SLO #2: Identify the dynamics affecting businesses in the international economy. Instrument/Method: MBA545 Case Study Evaluators: Kathy Gjerde Semester: Fall 2010 Criteria for Success: 1 Characteristic Visualizes the world as the potential market for his/her products or services Standards Used to Evaluate Student Performance Unsatisfactory Satisfactory (0 to 2 points) (3 to 5 points) Doesn't identify any single foreign Identifies at least two foreign market market opportunities for a opportunities for a company's products or company's products or services or services within a particular region identifies only one. Excellent (6 to 8 points) Identifies multiple foreign market opportunities for a company's products or services across global regions 2 Recognizes global business and cross-cultural factors applicable in a given context. Doesn't recognize any business practices or cross-cultural factors that may positively or negatively affect a company's operations in a given context Recognizes several business practices or cross-cultural factors that may positively or negatively affect a company's operations in a given context Clearly identifies relevant business practices and crosscultural factors that may positively or negatively affect a company's operations in a given context 3 Applies understanding of global business and cross-cultural influences while generating options/decisions/strategies. The proposed options, decisions, strategies do not reflect any international business practices or cross-cultural considerations The proposed options, decisions, strategies partially reflect international business practices or cross-cultural considerations The proposed options, decisions, strategies clearly reflect an equal level of consideration of domestic and global business practices and cross-cultural factors Total Score Performance Level (0 to 8 is unsatisfactory, 9 to 17 satisfactory, 18 to 24 is excellent)* 17 Score Appendix C Assessment Instrument Peer Evaluation and Team Assessment Please answer each of the following questions for each team member; use a scale of 1 - 100. SELF TEAM MEMBER TEAM MEMBER TEAM MEMBER TEAM MEMBER TEAM MEMBER ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ yes sometimes no yes sometimes no yes sometimes no yes sometimes no yes sometimes no yes sometimes no ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ Please fill in the names of your team members; include yourself! 1. Please rate this person's quality of work. (where "1" means "extremely low quality" and "100" means "excellent quality") 2. Please rate this person's quantity of work. (where "1" means "almost no work" and "100" means "a huge amount of work") 3. Please rate this person's reliability. (where "1" means "not at all reliable" and "100" means "extremely reliable") 4. How well does this person build on what others say ? (where "1" means "never builds on what others say" and "100" means "always builds on what others say") 5. Please rate this person's adaptability. (where "1" means "not at all adaptable" and "100" means "extremely adaptable") 6. When the status quo fails, how willing is this person to try something new ? (where "1" means "never likes to try new things" and "100" means "extremely willing to try new things") 7. Please rate how well this person contributes to the team's decision-making . (where "1" means "no contribution" and "100" means "excellent contribution") 8a. Does this person challenge ideas in the group? Please circle "yes," "sometimes" or "no." 8b. How well does this person challenge ideas respectfully ? (where "1" means "never challenges respecfully" and "100" means "always challenges respectfully") 9. Overall, how effective a team member is this person? (where "1" means "not at all effective" and "100" means "extremely effective") 18 Appendix D 19 Appendix E 20