January 29, 2014 Quality Assurance of Engineering Education by Outcome Based Evaluation Mitsunori Makino, Dr. Eng. Committee Chair, Criteria, JABEE Professor Program Leader, Rubric-based Competency Development, Faculty of Science and Engineering, Chuo University Copyright © 2014 Mitsunori Makino. All rights reserved. 1 Self Introduction of Mitsunori Makino • Graduated from Waseda University – B.Eng. in 1987, M.Eng. in 1989, Dr. Eng. in 1992 • Faculty member of Chuo University since 1992 – 2004- : Professor – 2008- : Leader of Rubric-based Competency Development Project, Faculty of Science and Engineering, Chuo University, (www.facebook.com/ChuoCompetency) – 2009-2013: Vice Dean, Faculty of Science and Engineering • JABEE – Chairperson of Criteria Committee, JABEE (www.jabee.org) Copyright © 2014 Mitsunori Makino. All rights reserved. 2 Outline • What is “outcomes-based” ? Quality assurance of higher education – Input-based vs. outcomes-based • Washington Accord --- World-wide accreditation alliance of engineering education • How to assess graduates’ knowledge and performance – Rubric – Rubric for achievement in a course – Rubric for achievement for …. • Discussion • Conclusion Copyright © 2014 Mitsunori Makino. All rights reserved. 3 What is quality assurance, especially in higher education? • The maintenance of a desired level of quality in a service or product, especially by means of attention to every stage of the process of delivery or production. from Oxford Dictionary (http://www.oxforddictionaries.com/definition/english/quality-assurance?q=quality+assurance) • In the higher education… The maintenance of a desired level or higher of quality in graduates by means of attention to every stage of the education system Copyright © 2014 Mitsunori Makino. All rights reserved. 4 Why do we need the quality assurance in engineering education by the third party? (1/2) • The 1st step to professional Practice Training and Experience Graduate Attributes: Indicate that program objectives are satisfied Accredited Program Observe Code of Conduct and Maintain Competence Meet Standard for Professional Competency Meet Standard for Engineering Education Copyright © 2014 Mitsunori Makino. All rights reserved. 5 Why do we need the quality assurance in engineering education by the third party? (2/2) • The substantial equivalence in the world: Applied to educational programs means that two programs, while not meeting a single set of criteria, are both acceptable as preparing their respective graduates to enter formative development toward registration. Signatory A Accredited Program Practice Training and Experience Substantial Equivalence Signatory B Accredited Program Copyright © 2014 Mitsunori Makino. All rights reserved. 6 How can we show the quality of engineering education? • Contents, environment – Book of Knowledge Benchmarks for what? • Environment – Budget – Facilities • Outcomes Necessity of “educational objective” & “performance criteria” – Skills & competencies of graduates based on professional knowledge • • • • Project/Research output (accepted papers, patents...) Enrollment numbers in professional qualification Employment as engineer … Copyright © 2014 Mitsunori Makino. All rights reserved. 7 Educational outcomes must inevitably given, not happen • Educational program is planned in advance, connected with its goal and/or objectives, about graduates’ performance and knowledge (not “what is taught” but “what is equipped”) • If so, the educational program can be trusted as assured human resource development. (Graduates are expected a certain level of their knowledge and competency) Copyright © 2014 Mitsunori Makino. All rights reserved. 8 Graduate Attributes & Disclosure of Three Policies of Admission, Curriculum and Diploma by MEXT Exemplified graduate attributes: 1. Knowledge and understanding i. ii. 2. Versatile skill i. ii. iii. iv. v. 3. Communication skill Quantitative skill Information literacy Logical thinking Problem solving Each university in Japan has and discloses its diploma policy regarding the attributes Behaviors and intentionality i. ii. iii. iv. v. 4. Understanding of knowledge of multiple/different cultures Understanding of knowledge of human culture, society and nature Self-control Teamwork and leadership Ethics Social responsibility as civil Ability of lifelong learning Integrated learning experiences and creative thinking Copyright © 2014 Mitsunori Makino. All rights reserved. 9 IEA Graduate Attributes • The 2nd version was released in June 2009, modified in 2013 • Graduates of engineering schools should equip both of “Engineering Knowledge” and “Graduate Attributes”. • These profiles are defined for – Washington Accord (WA): Engineering education – Sydney Accord (SA): Technologist education – Dublin Accord (DA): Technician education Copyright © 2014 Mitsunori Makino. All rights reserved. 10 Washington Accord Membership (as of 2013) Accreditation bodies Provisional status 6 Founding Members Signatory 1989 HKIE (HK) No system at that time 1995 ECSA (South Africa) 1994 1999 JABEE (Japan) 2001 2005 IES (Singapore) 2003 2006 BEM (Malaysia) 2003 2009 ASIIN (Germany) 2003 but was removed in 2013 ABEEK (RP Korea) 2005 2007 IEET (Chinese Taipei) 2005 2007 AEER (Russia) 2007 2012 AICTE (India) 2007 IESL (Sri Lank) 2007 MUDEK (Turkey) 2010 PEC (Pakistan) 2010 COE (Thailand) Submitted in 2010 but was differed BAETE (Bangladesh) 2011 CAST (PR China) 2013 PTC (The Philippines) 2013 Indonesia Interest Peru Interest 2011 Copyright © 2014 Mitsunori Makino. All rights reserved. 11 IEA (Engineering) Knowledge Profile 1. Natural sciences 2. Mathematics, numerical analysis, statistics, formal aspects of computer and information science 3. Engineering fundamentals 4. Engineering specialist knowledge 5. Knowledge supporting engineering design 6. Knowledge of engineering practice 7. Comprehension of the role of engineering in society and identified issues in engineering practice in the discipline 8. Engagement with selected knowledge in the research literature of the discipline Copyright © 2014 Mitsunori Makino. All rights reserved. 12 IEA Graduate Attributes Profile 1. Engineering Knowledge 2. Problem Analysis 3. Design/development of solutions 4. Investigation 5. Modern Tool Usage 6. The Engineer and Society 7. Environment and Sustainability 8. Ethics 9. Individual and Team work 10. Communication 11. Project Management and Finance 12. Life long learning Copyright © 2014 Mitsunori Makino. All rights reserved. 13 Quality Assurance by JABEE Criteria Criterion 1 : Learning Outcomes A set of clear students’ performance, planned by program, enough to achieve the outcomes Criterion 4:Educational Improvement Performance Criteria Implementation Criterion 2:Educational Methods What & how high? Assurance through review and evaluation Criterion 3:Achievement of Learning Outcomes Copyright © 2014 Mitsunori Makino. All rights reserved. 14 Learning Outcomes in JABEE 2012 Criteria (a) An ability of multidimensional thinking with knowledge from global perspective (b) An ability of understanding of effects and impact of engineering on society and nature, and of engineers’ social responsibilities (engineering ethics) (c) Knowledge of and an ability to apply mathematics, natural sciences and information technology (d) Knowledge of the related engineering disciplines, and an ability to apply it to solve problems (e) An ability to design to respond to requirements of the society by using various sciences, technologies and information (f) Communication skills including logical writing, presentation and debating in Japanese and fundamental communication skills on the international scene. (g) An ability of independent and life-long learning (h) An ability to manage and accomplish tasks systematically under given constraints. (i) An ability to work as a team member Copyright © 2014 Mitsunori Makino. All rights reserved. 15 JABEE’s Outcomes-based Assessment • Instead of structural assessment, the contents of education are assessed as outcomes. • JABEE does not question the approach of the program as long as the outcomes are assured. • The program shall set up and demonstrate with benchmarks the outcomes, which the program expects. • The outcomes of the program are evaluated by the society in the era of information. Copyright © 2014 Mitsunori Makino. All rights reserved. 16 How to make students learn by themselves and evaluate appropriately graduates attributes? • “Knowledge” profile may be assured by suitable curriculum with a certain assessment (paper exam, report, …)(almost input-based). • “Ability” may be implicit and expressed as his/her action. How do we “teach” action profile? Copyright © 2014 Mitsunori Makino. All rights reserved. 17 Rubric-based Competency Development (including assessment) in Chuo University Copyright © 2014 Mitsunori Makino. All rights reserved. 18 Rubric: Evaluation of Performance & Level • Nobody can evaluate implicit (inside). • Rubrics check performance and level which he/she presented as some actions • For assurance of learning outcomes, behavior characteristics is important. (happened actions is not important) Therefore, rubrics of “competencies” should be considered. Copyright © 2014 Mitsunori Makino. All rights reserved. 19 What is rubric? • Rubric: a tool to evaluate level of performance • Three elements: – Dimensions …performance criteria – Scales …levels of performance – Descriptors • Evaluation of performance of study items • Through self-check by students and evaluation by teachers, the rubric help students’ awakening. http://www.abet.org/Linked%20Documents-UPDATE/webinars/webinar-developing-rubrics-2010.pdf Copyright © 2014 Mitsunori Makino. All rights reserved. 20 Rubric: Trends in Higher Education • Especially effective for PBL • Objectivity of learning outcomes and mutual understanding between students and professors • Effective of course design and FD Copyright © 2014 Mitsunori Makino. All rights reserved. 21 Graduate attributes also use the rubric, showing differences between engineering, technologist and technician education. Range of Problem Solving (IEA) WA requires “complex problems” Copyright © 2014 Mitsunori Makino. All rights reserved. 22 Rubric: Examples in U.S. Engineering Education • ABET Webinar: Developing Rubrics http://www.abet.org/webinar-developing-rubrics.shtml • WSU-EERC (Engineering Education Research Center, Washington State University) IDEALS http://ideals.tidee.org/ Copyright © 2014 Mitsunori Makino. All rights reserved. 23 Rubric-based Competency Development Project in Faculty of Science and Engineering, Chuo University • Kick-off: Summer, 2008 • National Project “Good Practice in Higher Education” from autumn 2009 to March 2012 • Goal: – Clarification of graduate attributes (competencies) of Faculty of Science and Engineering, Chuo University – Development of systematic curriculum and teaching method of the competencies • Project leader: Mitsunori Makino • Leading department: Information and System Engineering (Computer Engineering) Copyright © 2014 Mitsunori Makino. All rights reserved. 24 Procedure of Competency Development in Chuo Univ. 1. 2. 3. 4. 5. Definition of Educational Goal Definition of competencies as graduate attributes Detailed competencies with levels Application to courses Sharing information on good practices Copyright © 2014 Mitsunori Makino. All rights reserved. 25 Seven Competencies as Graduate Attributes in ISE Department Competency Keyword Defi ni tion Level 0 1 2 Communi ca tion a bi l i ty After ha vi ng correctly unders tood other peopl e's opi ni ons or wri tings , cl ea rl y expres s one's own opi ni ons a bout them. Us e effective expl a na tion methods or techni ques to pers ua de i nvol ved pa rties . Probl em-s ol vi ng a bi l i ty Correctly unders ta nd the i s s ues . Devi s e a nd Correctly unders ta nds i mpl ement s ol utions . Correctly unders ta nds Is una bl e to correctly s el f-di s covered or gi ven Veri fy the res ul ts a nd gi ven i s s ues a nd tri es to unders ta nd gi ven i s s ues . i s s ues . Devi s es a nd refl ect them i n pl a n s ol ve them. i mpl ements s ol utions . revi ews or s ubs equent pl a nni ng. Knowl edge a cqui s i tion a bi l i ty Ha vi ng ma de efforts to continuous l y ga ther a nd s i ft through a deep a nd broa d ra nge of i nforma tion, a cqui re, a ppl y a nd util i ze knowl edge a nd expertis e. Is una bl e to unders ta nd other peopl e a nd expres s hi s /her own opi ni ons to them. Is una bl e to ga ther i nforma tion by hi ms el f/hers el f a nd a cqui re new knowl edge a nd expertis e. Is on the whol e a bl e to unders ta nd other peopl e's opi ni ons a nd on the whol e expres s hi s /her own opi ni on to them. Is on the whol e a bl e to ga ther i nforma tion by hi ms el f/hers el f a nd a cqui re new knowl edge a nd expertis e. Li s tens to other peopl e a nd communi ca tes s moothl y by expres s i ng hi s /her own opi ni ons to them. 3 Ha vi ng unders tood other peopl e, expres s es hi s /her own opi ni ons i n a wa y tha t i s ea s y to unders ta nd by devi s i ng methods for gi vi ng expl a na tions , a nd thus ga i ns a s uffi ci ent degree of unders ta ndi ng. 4 Us es a ra nge of methods or techni ques for gi vi ng expl a na tions a nd thus ga i ns a s uffi ci ent degree of mutua l unders ta ndi ng wi th peopl e. Di s covers i s s ues by hi ms el f/hers el f, s el ects the bes t s ol ution a nd i mpl ements thi s s ys tema tica l l y. Veri fi es the res ul ts from va ri ous a s pects a nd refl ects them i n s ubs equent pl a nni ng. Ha vi ng ma de efforts to continuous l y ga ther a nd Ha vi ng ma de efforts to Underta kes a ca reful s i ft through a deep a nd ga ther a nd s i ft through a exa mi na tion of ga thered broa d ra nge of deep a nd broa d ra nge of i nforma tion, a nd i nforma tion, a cqui res , i nforma tion, a cqui res , a cqui res , a ppl i es a nd a ppl i es a nd util i zes a ppl i es a nd util i zes util i zes knowl edge a nd knowl edge a nd expertis e knowl edge a nd expertis e. i n wa ys tha t other expertis e. peopl e ha ve not thought of. Di s covers i s s ues by hi ms el f/hers el f, a nd devi s es a nd i mpl ements s ol utions . Veri fi es the res ul ts a nd refl ects them i n pl a n revi ews or s ubs equent pl a nni ng. Copyright © 2014 Mitsunori Makino. All rights reserved. 26 Seven Competencies as Graduate Attributes in ISE Department Competency Keyword Orga ni zed beha vi or ca pa bi l i ti es Crea ti ve a bi l i ty Defi ni ti on Level 3 As certa i ns i n a As certa i ns i n a mul ti fa ceted a nd mul ti fa ceted a nd objecti ve wa y from objecti ve wa y from mul ti pl e pers pecti ves mul ti pl e pers pecti ves wha t s houl d be done to wha t s houl d be done to a chi eve tea m or a chi eve tea m or orga ni za ti ona l goa l s , orga ni za ti ona l goa l s , Unders ta nds common Is a bl e to work when rea ches a ppropri a te rea ch a ppropri a te Is una bl e to work i n a goa l s i n tea mwork a nd i ns tructed, but does not judgments a nd a cts wi th judgments a nd a ct wi th a tea m a nd beha ves i n a beha vi or a nd a cts wi th a ma ke a n effort to a chi eve a s ens e of owners hi p. s ens e of owners hi p. wa y tha t i s s el f-centered. s ens e of owners hi p i n goa l s by hi ms el f/hers el f When doi ng s o, When doi ng s o, order to a chi eve thes e. demons tra tes wi th demons tra te wi th others others mutua l res pect for mutua l res pect for ea ch ea ch other's opi ni ons other's i dea s a nd a ct a nd a cts whi l e whi l e devel opi ng a nd devel opi ng a nd ma i nta i ni ng ma i nta i ni ng rel a ti ons hi ps of trus t. rel a ti ons hi ps of trus t. Demons tra te i ntel l ectua l curi os i ty, be i nteres ted i n thi ngs wi thi n a nd beyond one's a rea of s peci a l i za ti on, obta i n i dea s from thos e thi ngs a nd concei ve unprecedented i dea s . When doi ng s o, obs erve rel eva nt l a ws a nd regul a ti ons , ha ve a s ens e of ethi cs a nd ful fi l the res pons i bi l i ty engi neers ha ve to s oci ety. 0 Is not i nteres ted i n new i dea s a nd technol ogi es , even i f he or s he knows a bout them. 1 Ha bi tua l l y col l ects i nforma ti on rega rdi ng hi s /her fi el d of i nteres t a nd pa ys a ttenti on to new i dea s a nd technol ogi es . 2 4 Ha vi ng cons i dered thoroughl y wha t s houl d be done to a chi eve tea m or orga ni za ti ona l goa l s a nd the benefi ts for a l l concerned, rea ches a ppropri a te judgments a nd does not a ct by hi ms el f/hers el f but gi ves di recti ons a nd gui des peopl e a s to wha t they s houl d a i m a t . Demons tra tes i ntel l ectua l curi os i ty, i s Demons tra tes i nteres ted i n thi ngs i ntel l ectua l curi os i ty, i s wi thi n a nd beyond Demons tra tes wi dei nteres ted i n thi ngs hi s /her a rea of ra ngi ng i ntel l ectua l wi thi n a nd beyond s peci a l i za ti on, obta i ns curi os i ty, rega rdl es s of hi s /her a rea of i dea s from thos e thi ngs whether thi ngs a re s peci a l i za ti on, obta i ns a nd concei ves uni que wi thi n or beyond hi s /her i dea s from thos e thi ngs i dea s of the ki nd tha t a rea of s peci a l i za ti on, a nd concei ves contri bute to the a nd s tri ves to a pproa ch unprecedented i dea s . devel opment of s ci ence thi ngs by a cti vel y When doi ng s o, obs erves a nd technol ogy. When i ncorpora ti ng new rel eva nt l a ws a nd doi ng s o, obs erves knowl edge. regul a ti ons , ha s a s ens e rel eva nt l a ws a nd of ethi cs a nd ful fi l s the regul a ti ons , ha s a s ens e res pons i bi l i ty engi neers of ethi cs a nd ful fi l s the ha ve to s oci ety. res pons i bi l i ty engi neers ha ve to s oci ety. Copyright © 2014 Mitsunori Makino. All rights reserved. 27 Seven Competencies as Graduate Attributes in ISE Department Competency Keyword Defi ni ti on Sel f-rea l i za ti on a bi l i ty To el eva te ones el f, cons ta ntl y purs ue new goa l s , cons i der pa ths towa rds thes e goa l s a nd s tri ve to a chi eve them. When doi ng s o, a l wa ys a ct to i mprove s el fcontrol . Speci a l i za ti on (Depa rtment of Informa ti on & Sys tem Engi neeri ng) On the ba s i s of broa d a nd deep knowl edge a nd experi ence, ma ke a ppropri a te us e of progra mmi ng a nd ICT tool s , enga ge i n a ccura te work a nd uti l i ze i nforma ti on engi neeri ng. Level 0 1 2 3 4 To el eva te hi ms el f/hers el f, To el eva te cons ta ntl y s ea rches for Determi nes cl ea r goa l s hi ms el f/hers el f, new goa l s a nd, when If he/s he ha s a goa l , by hi ms el f/hers el f, cons ta ntl y purs ues new found, thi nks up the Does not try to fi nd goa l s expres s es a des i re to cons i ders pa ths towa rds goa l s , cons i ders pa ths s hortes t route to a nd does not a chi eve a chi eve thi s goa l a nd thes e goa l s a nd s tri ves towa rds thes e goa l s a nd a chi evi ng thes e goa l s them, even i f gi ven. s tri ves towa rd i t. to a chi eve them. When s tri ves to a chi eve them. a nd s tri ves to a chi eve doi ng s o, demons tra tes When doi ng s o, a l wa ys them by fol l owi ng thi s s el f-control . a cts to i mprove s el fpa th. Even i f fa i l s , ma kes control . repea ted efforts to a chi eve goa l s . Pos s es s es a n extremel y a dva nced s peci a l i s t Pos s es s es a s ys tema ti c knowl edge, i s a bl e to La rgel y unders ta nds unders ta ndi ng of Pos s es s es a fra gmenta ry unders ta nd hi ghl y s peci a l i s t knowl edge, s peci a l i s t knowl edge, a ca demi c a bi l i ty a nd s peci a l i s t i nforma ti on a nd i s a bl e to a nd i s a bl e to Does not pos s es s a n knowl edge, i s a bl e to a nd judge i ts a ccura cy a t unders ta nd i nforma ti on unders ta nd hi ghl y a ca demi c a bi l i ty or unders ta nd s i mpl e the l evel of a s peci a l i s t rel a ted to thi s , judge i ts s peci a l i s t i nforma ti on, knowl edge to form the i nforma ti on, judge i ts a nd communi ca te a ccura cy a nd ma ke judge i ts a ccura cy a nd founda ti ons for s tudyi ng a ccura cy a nd ma ke hi s /her own a s s erti ons hi s /her own a s s erti ons . ma ke hi s /her own i nforma ti on a nd s ys tem s ma l l -s ca l e a s s erti ons , both domes ti ca l l y a nd Is a bl e to underta ke a s s erti ons . Is a bl e to engi neeri ng. but the preci s i on a nd overs ea s . Is a bl e to work tha t ha s a certa i n underta ke work tha t i s a ccura cy of hi s /her work underta ke work tha t i s s ta nda rd of preci s i on a bove a certa i n s ta nda rd i s i ns uffi ci ent. a bove a certa i n s ta nda rd a nd a ccura cy. of preci s i on a nd of preci s i on a nd a ccura cy a ccura cy. through s uperi or i nnova ti ons . Copyright © 2014 Mitsunori Makino. All rights reserved. 28 Two points for applying rubric to course assessment 1. Define the dimensions and scales which can be checked in the course. Then descriptor at each level can be described according to supposed students’ action. 2. If necessary, revise course which encourages students’ “awareness” and “innovation.” Copyright © 2014 Mitsunori Makino. All rights reserved. 29 Introduction of Competency Development to Course (Software Design Team Project) Clarification of competencies which can be observed at each class Copyright © 2014 Mitsunori Makino. All rights reserved. 30 Evidence of Competencies Development (1) Self-Assessment by Students 現在の段階(演習履修4年生) 2010年4月と2011年7月の 現在の段階 (演習履修4年生8名平均) 傾聴力 基盤となる学力 数学・自然科学 3.5 読解力 記述力 情報技術基礎 提案力 3 専門知識 精確性 議論力 課題発見 2.5 達成志向 課題分析 2 ストレスコントロール 論理的思考 1.5 自己管理 計画実行 1 スケジュール管理 検証 目標設定 学習 倫理 応用力 探究する意欲 情報収集力 2010年4月 感動する力 推論する力 発想する力 バランス力 役割認識 率先力 協働 主体性 2011年7月 5段階評価(0~4) Copyright © 2014 Mitsunori Makino. All rights reserved. 31 Evidence of Competencies Development (2) Assessments by Faculty Staff, Graduates and TA 個人評価(Max値)-平均(6名) チーム内個人評価ー平均(6名) 傾聴力 読解力 基盤となる学力 4 記述力 数学・自然科学 提案力 情報技術基礎 議論力 専門知識 3 課題発見 精確性 達成指向 ストレスコントロール 自己管理能力 傾聴力 読解力 基盤となる学力4.0 記述力 数学・自然科学 提案力 情報技術基礎 議論力 専門知識 3.0 2 1 0 スケジュール管理 目標設定 倫理 探求する意欲 感動する力 推論する力 発想する力 率先力 達成指向 論理的思考 ストレスコントロール 計画実行 自己管理能力 検証 学習 バランス力 役割認識 主体性 協働 2.0 1.0 目標設定 2010.02 2010.07 2011.07 課題分析 論理的思考 計画実行 0.0 スケジュール管理 応用力 情報収集力 課題発見 精確性 課題分析 倫理 探求する意欲 感動する力 推論する力 発想する力 率先力 検証 学習 応用力 情報収集力 バランス力 役割認識 協働 主体性 2010.02 2010.07 2011.07 Copyright © 2014 Mitsunori Makino. All rights reserved. 32 From Department to Faculty FD & SD through Planning of Rubric-based Competency Development • All departments in the Faculty of Science and Engineering, Chuo University have the rubrics, respectively, and apply them to some courses. • The faculty includes the competencies as the graduate attributes in its both of diploma and curriculum policies. Copyright © 2014 Mitsunori Makino. All rights reserved. 33 Conclusion and Future Work Copyright © 2014 Mitsunori Makino. All rights reserved. 34 For assurance of educational quality through achieving graduate attributes • Competency development is becoming mandatory in higher education in Japan as well as global engineering education. • Combination of the rubric-based competency development with engineering education seems positive result from our project. • How do we promote the rubric-based competency development to wide area of engineering education ? – Understanding of importance – Re-organization of courses, if necessary Copyright © 2014 Mitsunori Makino. All rights reserved. 35 Thank you for your attention! It’s my honor to be involved in establishment of accreditation system for engineering education in Indonesia. I hope I will be here for further discussion in near future. makino@m.ieice.org Copyright © 2014 Mitsunori Makino. All rights reserved. 36