PAC(July06)3a_2 draft
June 2006
726885476

I
NTERNATIONAL AND
R
EGIONAL
T
EST
C
OMPONENTS
................................................. 2
T HE I NTERNATIONAL C ORE C OGNITIVE T EST ............................................................. 2
T HE REGIONAL COGNITIVE TEST ( S ) ............................................................................. 3
A
LLOCATION OF COGNITIVE TEST CONTENT TO MODULES
........................................... 3
726885476

Cognitive Test Development
The cognitive test comprises two parts: the international core and the regional component.
All participating countries will complete the core test. A regional test may be developed for regional modules with a sufficient number of participating countries 1 . It is envisaged that there will at least be a European regional module in ICCES.
A two-day workshop for NRCs will be conducted by the co-ordinating members of the consortium for the cognitive test item development. The first day of the workshop will be used to create and confirm common understandings of the content of the cognitive tests and establish conceptual and practical guidelines for item development including: stimulus selection and development; principles of multiple-choice item development; principles of short-constructed response item development; and the development of scoring guides for short-constructed response items.
On the second day, NRCs will draft and panel core and regional items with their international colleagues. Such a day will contribute to a smoother and more productive item development process and enable a greater sense of collaborative engagement in the test development process.
The international core cognitive test will consist of new items and trend items. The trend items will be selected from the pool of 22 non-released items from the CIVED study.
The core test will include approximately 50 items (42 - 44 multiple-choice items and
6 – 8 short-constructed response items). Between 22 and 27 of the multiple-choice items will be new to ICCES and 15 to 18 will be trend items. All the short-constructed response items will be new to ICCES. Students will have 60 minutes to complete the test.
2
Approximately 60 new multiple choice items and 16 short-answer items will be included in the field trial. The items with the strongest psychometric properties which together cover the breadth of the assessment framework will be selected for inclusion in the full 2008 item set for the main study.
As the ICCES items will be written to reflect the developing ICCES assessment framework, all 22 of the non-released CIVED trend items should be included in the field trial. This will enable both the psychometric properties of the trend items with
1 The issue of what constitutes a viable region and regional component will need careful negotiation between NRCs, the IEA and consortium partners.
2 It is possible to increase the number of core test items to be included in the test by using multiple rotated test booklets linked by common items. The current proposal is based on the use of only one test form in the final core test, however, the use of multiple rotated test forms will be considered as an option for the main data collection.
726885476 2

Cognitive Test Development reference to the new item set, and to the new cohort of participating countries to be used to ensure selection of the strongest items as trend items in the full ICCES item set for the main study.
Table 1 contains the approximate number of new and trend items to be included in the cognitive test for ICCES pilot, field trial and main studies.
Table 1: Approximate number of new and trend items to be included in the cognitive test for ICCES pilot, field trial and main studies
Number of test forms
Trend items (total)
New MQC items
New Constructed response items
Pilot study
4 – 5
80
25
Field trial
3 – 4
22
60
16
Main study
1 or more
15 – 18
22 – 27
6 – 8
The regional cognitive test(s) will consist of approximately 15 to 20 multiple choice items for each regional module. All the items in the regional modules will be new to
ICCES.
Interested countries may pilot a set of regional items, specific to their region, to inform the item selection for the field trial. The development of the European regional cognitive test will be coordinated by NFER and carried out in close cooperation with the ICC. The coordination of the development of regional cognitive tests for other regions modules (where appropriate) would be coordinated by ACER, NFER or LPS.
Approximately 30 to 40 items will be included in the regional component of the field trial. The items with the strongest psychometric properties will be selected for inclusion in the full ICCES item set for the main study.
The Assessment Framework articulates the mapping of cognitive test items to the four
Domains of the ICCES Civics and Citizenship Framework.
Domain 1: Identity and Connectedness
This domain is concerned with concepts which are central to civics and citizenship education. Measuring students’ knowledge and understanding of the mechanisms of the formation and content of identity and connectedness – as distinct from students’ perceptions of and feelings about identity and connectedness – can be regarded as context-independent and is likely to be assessed only by items in the core module.
Student perceptions of and feelings about their identities and connectedness will be accessed through the non-cognitive international and regional components of instrument (Parts 1c, 1d and 1e).
The distinction between the measurement of knowledge and understanding with respect to underlying principles, rather than in relation to personal (local and regional)
726885476 3

Cognitive Test Development contexts, is fundamental to the allocation of the cognitive test content in item development.
Domain 2: Civic Knowledge and Understandings
This domain is concerned with the concepts of civic knowledge and understandings.
These can be assessed either as concepts of principle, or with reference to specific content at the local, national, regional and supranational level. For the purposes of this study, the distinction made is between specific regional concepts and concepts of principle applied in the generic context.
The core cognitive assessment module is likely to include civic knowledge and understandings of principles and generic national or supranational institutions and instrumentalities. The regional cognitive assessment module(s) is likely to focus on civic knowledge and understandings of the specific concepts, institutions and instrumentalities appropriate to each region.
Domain 3: Citizenship Dispositions
This domain focuses on some dispositional concepts which underpin citizenship.
Measuring students’ knowledge and understanding of these concepts requires students to respond to stimulus from a foundation of common principles. Measuring students’ knowledge and understanding of these principles can be thus regarded as contextindependent and therefore is likely to be assessed primarily in items in the core module. Items measuring students’ understandings of the concepts that underpin
Citizenship can also be developed relevant to regional contexts should this be required.
Domain 4: Civics and Citizenship Competencies
This domain focuses on the essential cognitive and behavioural civics and citizenship competencies. Evidence of achievement of these competencies can be demonstrated across all contexts.
Higher order thinking and observation and analysis of change processes in civics and citizenship will be assessed in both the core and regional modules. Students’ capacity to analyse and evaluate the mechanisms of decision-making and active engagement is likely to be assessed in both the core and regional modules.
Practising decision making and active engagement in civics and citizenship has a participatory focus and this will not be assessed in the cognitive tests.
Table 2 summarises the proposed distribution of the content of the four Domains of the ICCES Civics and Citizenship Framework, across the core and regional cognitive assessment modules.
726885476 4

Cognitive Test Development
Table 2: Proposed distribution of the content of the 4 Domains of the Civics and Citizenship Framework across the core and regional cognitive assessment modules
Core
Domain 1:
Identity & Connectedness

Personhood

Belonging
Domain 2:
Civic Knowledge and Understandings

Fundamental principles

Supranational institutions and instrumentalities
Domain 3 :
Citizenship Dispositions

Equity

Social Cohesiveness

Taking Action
Domain 4:
Civic and Citizenship Competencies


Higher order thinking
Analysis and observation of change processes

Understanding of decision making and active engagement
Regional
Domain 1:
Identity & Connectedness

Perceptions of students to regional issues of identity
Domain 2:
Civic Knowledge and Understandings

Regional institutions and instrumentalities
Domain 3:
Citizenship Dispositions
(if required)
Domain 4:
Civic and Citizenship Competencies


Higher order thinking
Analysis and observation of change processes

Understanding of decision making and active engagement
726885476 5