北京师范大学
教育研究方法讲座系列 (2):
教育政策研究
第九讲
教育政策评估过程的研究
A. Conceptualization of Policy Evaluation
1. Conceptualization of policy a. Fitzpatrick, Sanders and Worthen define
“evaluation as the identification, clarification, and application of defensible criteria to determine an evaluation object ’s value (worth or merit) in relation to those criteria.
” (2004, p.5) b. Accordingly, the constituents of policy evaluation consist of i. The evaluation object: It is of course public policy or more specifically education policy. ii. The evaluation criteria: Define the criteria upon which the evaluations are based. They may include
- Policy objectives: To assess to what extent the identified policy objectives have been fulfilled.
- Policy measures and programs: To investigate to what extent the designed policy measures and programs have been implemented
- Policy outcomes: To measure to what extent the targeted outcomes have been realized
- Cost and benefit of the policy: To assess the cost of the implementation of policy means against the benefits entailed by the policy outcome iii. The judgment: The evaluation can be studies and judge by means of
- Empirical-positivist study: Effectiveness and efficiency of the policy design and implementation
- Interpretive study: The practicality of the policy
- Critical study: The reflexivity and capacities for betterment of the status quo of the policy phenomenon of the policy
3. Typology of policy evaluation a. In regards to the purpose of policy evaluation, policy evaluation can be categorized into i. Formative evaluations aim to provide information for personnel implementing the policy or carrying out the program. Hence, their primary purposes are for the development and improvement of the policy and/or program ii. Summative evaluations aim to pass judgment on the performance of the policy and/or program. They are primarily carried out as accountability mechanism or auditing measures. b. In relation to the policy stages, policy evaluation can be categorized into i. Need assessments: They the studies conducted at the formative stages of the policy process. They are conducted either to
W.K. Tsang
Policy Studies in Education
1

evaluate the urgencies of the policy problems or the needs that are required if the identified desirable states of affairs are to be realized. ii. Process-monitoring studies: They are studies carried out to monitor to what extent are the designed policy measures .and programs are implemented in real world situations. They also aim to provide information for the improvement of the effectiveness of the policy instruments in use. iii. Outcome studies: They are carried out to assess to what extent have the preconceived policy outcomes been attained. They can also provide information to the revelations of unanticipated consequences (desirable, undesirable or both) of the policy c. In terms of the policy context, policy evaluation can be classified into i. Endogenous policy evaluation: They are studies carried out to assess the internal effectiveness and coherence among various parts of the policy constituents. The most common types of the endogenous evaluation studies are those to assess whether the policy outcomes are in congruence with the policy objectives. ii. Exogenous policy evaluation: They are studies assessing whether the policy substance are in congruence with the policy environment. These type of evaluation studies are especially informative in ever changing policy environment.
B. Frank Fischer
’s Conception of Four Levels of Policy Evaluation
1. The framework of practical logical of policy evaluation: a. Post-positivist methodological approach to policy evaluation: By post-positivism, it refers to "a contemporary school of social science that attempts to combine the discourse of social and political theory with the rigor of modern science. It calls for a marriage of scientific knowledge with interpretive and philosophical knowledge about norms and values." (Fischer, 1995, p. 243) According, Fischer advocates that policy evaluation research should extend beyond the dominance of empirical-positivism and incorporate practical discourse about rightness of normative claims into policy evaluation research. b. Fischer names such a post-positivist approach in policy evaluation the framework of the "Practical Logic of Policy Evaluation".
"It is design to illuminate the basic discursive components of a full or complete evaluation, one which incorporates the full range of both the empirical and normative concerns that can be brought to bear on an evaluation. Structured around four interrelated discourses, the approach extends from concrete empirical questions pertinent to a particular situation up to the abstract normative issues concerning a way of life." (Fischer, 1995, p. 18) c. Accordingly, the framework of practical logic of policy evaluation is made up of i. First-order evaluation level:
- Technical-analytical discourse
- Contextual discourse ii. Second-order evaluation level
W.K. Tsang
Policy Studies in Education
2

- Systems discourse
- Ideological discourse
2. Technical-analytical discourse: Program verification a. By program verification, it refers to concerns "with measurement of the efficiency of program outcome. …The basic questions of verification are i. Does the program empirically fulfill its stated objective(s)? ii. Does the empirical analysis uncover secondary or unanticipated effects that offset the program objectives? iii. Does the program fulfill the objectives more efficiently than alternative means available?" (p. 20) b. "Program verification employs such methodologies as experimental research and cost-benefit analysis. The goal is to produce a quantitative assessment of the degree to which a program fulfills a particular objective (standard or rule) and a determination (in terms of a comparison of input and output) of how efficiently the objective is fulfill (typically measured as a ratio of costs to benefits) compared to other possible means." (p. 20)
3. Contextual discourse: Situational validation a. Validation focuses on whether or not the particular program objectives are relevant to the situation.
…Instead of measuring program objectives per se, validation examines the conceptualizations and assumptions underlying the problem situation which the program is designed to influence. Validation centers around the following questions: i. Is the program objective(s) relevant to the problem situation? ii. Are there circumstances in the situation that require an exception to be made to the objective(s)? iii. Are two or more criteria equally relevant to the problem situation?" (Pp. 20-21) b. "Validation is an interpretive process of reasoning that takes place within the framework of the normative belief systems brought to bear on the problem situation. Validation draw in particular on qualitative methods, such as those developed for sociological anthropological research." (p.21)
4. Systems discourse: Societal vindication a. "In vindication … the basic task is to show that a policy goal (from which specific program objectives were drawn) addresses a valuable function for the existing societal arrangement. Vindication is organized around the following questions: i. Does the policy goal have instrumental or contributive value for the society as a whole? ii. Does the policy goal result in unanticipated problems with important societal consequences? iii. Does a commitment to the policy goal lead to consequence (e.g. benefits and costs) that are judged to be equitably distributed?"
(p. 21) b. As second-order vindication, this type of policy evaluation "steps outside of the situational action context in which program criteria are applied and implemented in order to assess empirically the
W.K. Tsang
Policy Studies in Education
3

instrumental consequences of a policy goal in terms of the system as a whole." (p. 21) Hence, the methodological approach adopted by societal vindication is macroscopic-institutional analysis which usually takes the forms of comparative and/or historical-sociological methods.
5. Ideological discourse: Social choice a. "Social choice seeks to establish and examine the selection of a critical basis for marking rationally informed choices about societal systems and their respective ways of life. …Social choice raises the following types of questions: i. Does the fundamental ideal (or ideology) that organize the accepted social order provide a basis for a legitimate resolution of conflicting judgments? ii. If the social order is unable to resolve basic values conflicts, do other social orders equitably accommodate the relevant interests and needs that the conflicts reflect?
” iii. Do normative reflection and empirical support the justification and adoption of an alternative ideology and the social order it prescribes?
” (p. 22) b. "Social choice involves the interpretive tasks of social and political critique, particularly as practiced in political theory and philosophy.
Most fundamental are the concepts of a 'rational way of life' and
'good society'. Based on the identification and organization of a specific configuration or values ― such as equality, freedom, or community ― models of the good society serve as a basis for the adoption of higher level evaluative criteria." (p. 22) The typical methodological approach adopted by this type of evaluative studies is critical approach to political, social and philosophical inquiry.
C. Evaluation Study of MOI Policy for Secondary Schools in HKSAR
1. Technical-analytical Discourse: Program verification a. A policy measure in search of an objective: i. Summative or even judgmental evaluation of the efficiency of
CMI in comparison with EMI ii. Informative evaluation of the efficiency of preparing secondary-school leavers to become trilingual and bilateral iii. Upholding mother-tongue instruction iv. Summative evaluation of the effectiveness of eliminating mixed-coded teaching in secondary schools in HKSAR b. To compare the academic and personal development outcomes of students in EMI and CMI schools: Summative, outcome and endogenous evaluation research i. Instructional paradox in MOI policy outcomes ii. Socio-psychological paradox in MOI policy outcomes iii. School-organizational paradox in MOI policy outcomes c. To prepare of secondary-school leavers to be trilingual and bilateral i. Typology of bilingual
- Balance bilingual
- Chinese dominant bilingual
- English dominant bilingual
W.K. Tsang
Policy Studies in Education
4

ii. Dimensions of bilingual proficiency
- Cognitive demanding or undemanding
- Context reduced or embedded iii. Levels of bilingual proficiency
- Conversational proficiency
- Context-reduced and cognitive undemanding proficiency
- Context-embedded and cognitive demanding proficiency
- Academic language proficiency d. Upholding mother-tongue instruction (MTI): Unanticipated consequences of a. Relegating MTI to the status of second-class MOI, forced choice in SSPA, MOI for the lower 60% of primary-school leavers, … b. Instituting EMI schools to be the sector that needs least compensation e. Elimination of mixed-coded teaching in secondary schools in
HKSAR …
2. Contextual discourse: Situational validation of the relevance of the MOI policy a. To identify the stakeholders (e.g. students, parents, teachers and/or administrators in secondary schools, employers, higher-education institutions, etc.) and their
“definition of the situations” and “systems of relevance ” in regards to the MOI policy b. Situational paradoxes facing stakeholders i. Students and Parents: Situational paradox between immediate learning efficacy and long-term prospects in educational or even socioeconomic advancement ii. School teachers and administrators:
- Situational paradox between instructional efficacy and graduates' prospects in educational advancements
- Situation paradox between current instructional efficacy and future efficacy of attracting "high-quality" primary-school leavers or even future prospect of survival of the schools. iii. Employers: The situational paradox is between the prospects of local and global markets. iv. Higher-education institution: The situation paradox is between short-term instructional efficacy and long-term development of universally applicable scholarship and/or professional competence c. To validate whether or to what extent each of the stakeholders ’
“systems of relevance” have been fulfilled by the MOI policy measure.
3. System Discourse: Societal vindication of the MOI policy a. To define the institutional features of the social, political and economic systems of HKSAR: i. Biglossic or even triglossic structures of a post-colonial society, one-country-two-system polity, and open and small economy in global-informational capitalism. ii. To evaluate whether or to what extent that MOI policy have instrumental or contributive value for these systemic features of
HKSAR
W.K. Tsang
Policy Studies in Education
5

b. To vindicate the intensification effect of MOI policy on the contradictory structure of HKSAR i. Biglossic or even triglossic structures of the open and small economy of HK within the global-informational capitalism. ii. Biglossic structure of the political system of HKSAR within the context of One Country Two System iii. Biglossic or even troglossic structure of the cultural system of a post-colonial society c. To vindicate the unequalizing effect of MOI policy on the structural disparity among the 18 school districts in the SSPA system in terms of distribution of EMI school places d. To vindicate the institutionalized effect of MOI policy on the inequality of mobility chances between CMI and EMI students in competing access to higher education.
4 Ideological reflection on social choice underlying the MOI policy a. To identify the ideological stance underlying the MOI policy discourse i. Orientations to bilingualism
- Bilingualism as problem
+ Linguistic imperialism
+ Linguistic nationalism
- Bilingualism as rights
- Bilingualism as resource ii. Power implications of bilingualism
- Bilingualism as coercive power
- Bilingualism as collaborative power iii. Structure-agent stance in bilingualism
- Structural imposition stance
- Agent resistance stance
- Agent appropriation stance b. To evaluate the ideological stances toward Cantonese, Poutonghau and English, which have been embedded in the MOI policy discourse in post-1997 HK.
W.K. Tsang
Policy Studies in Education
6