MedBiquitous Accelerated Standards
Educational Achievement Standards
Research Report
March 19, 2012
Introduction
As part of the standards development process within the Educational Achievement project,
MedBiquitous has researched existing standards and specifications that could be assets in the
development of an educational achievement specification. As a matter of best practice, MedBiquitous
builds on existing standards when it is practical to do so. MedBiquitous staff researched credible
learning technology standards developers and relevant open tools. A summary of the findings for each
standard or specification researched is included along with information on relevant vocabularies, best
practices, and recommendations on the use of competencies. Comments and suggestions for additional
research are welcome.
Organizations Researched
The following organizations were researched:









Advanced Distributed Learning (ADL)
CEN (The European Committee for Standardization)
HR-XML (A consortium for interoperability in human resources)
IEEE Learning Technologies Standards Committee (IEEE LTSC)
IMS Global Learning
The UK Joint Information Systems Committee (JISC)
Mozilla
The Post Secondary Education Standards Council (PESC)
Schools Interoperability Framework (SIF)
Within each organization, a search for assessment and competency related standards, specifications,
and systems was conducted.
Findings
While several of the specifications are partially overlapping in scope with the vision set our stated
requirements, none of them integrate competencies, professional activities, assessments, and eportfolios. Even in areas that are well-specified, such as assessment results, the fit is often imperfect.
Depending on the IP policies of the specification and standards developing organizations, we may be
able to build on existing work.
In addition, several of the specifications referenced are parts of a large architecture and appear to be
difficult to untangle from the larger whole and use with other components. The PESC transcript
specification and the EuroLMAI specification may hold promise for communicating some transcript data,
but they lack the ability to provide integration with competency frameworks. At a recent meeting of the
co-chairs of MedBiquitous’ competency-related working groups, participants came back with a
recommendation to leverage the Curriculum Inventory specification in describing a learner’s academic
record. This would allow the record to describe how the learner’s activities relate to competency
frameworks and learning objectives in addition to providing data on the instructional and assessment
methods used.
2
Aside from transcript data, health professions education appears distinctive in a few ways.
1) There is an interest in knowing the methodologies used for assessment and the evidence used
to determine a result
2) There is an interest in having learner provided portfolio data integrated with more formal
assessment data
3) Having the capability to understand the full meaning of a mark is essential
4) The inclusion of professional activities and the concept of entrustment as an educational
achievement are unique.
To accurately communicate educational achievement data with these distinctions will require a new
specification.
Existing standards, specifications, and open systems
Europass XML Schema v 2.0
http://europass.cedefop.europa.eu/en/home
http://europass.cedefop.europa.eu/TechnicalResources/XML/xsd/europass_XML-schema-v2.0description.pdf
Europass is a European Union directive that seeks to make skills and qualifications easily understood
across Europe. It includes tools, templates, and specifications for cvs and language skill assessment
called a Language Passport. Europass includes identifying information about the individual, their
occupation, work experience, education, language, and skills.
European Learner Mobility Achievement Information (EuroLMAI)
http://www.cen-wslt.din.de/sixcms_upload/media/3378/CWA16132.pdf
“The EuroLM Achievement Information Model seeks to address directly the European requirement for
rapid implementation and dissemination of the institution-owned transparency information tools
throughout Europe. This EuroLM Achievement Information (EuroLMAI) Model will be developed as a
generic model for representing the institution attested achievements of a learner within a formal
learning setting, resulting from the learner's participation in one or more learning opportunity instances
and including any available associated results. The Europass institution-owned learner achievement
information structures, namely the Europass Diploma Supplement and the Europass Mobility, as well as
other official achievement reports (e.g. the learner's Transcript of records), will be specified as
application profiles of the EuroLMAI model.”
EuroLMAI is a CEN Workshop Agreement approved by those who participated in its development. The
specification build on CEN’s Metadata for Learning Opportunities (MLO) – Advertising (MLO-AD) and its
Educational Credit Information Model. It combines information about the learning opportunity taken
(MLO-AD), which can be a course or an entire program, and adds information about the learner, the
3
issuer, the results of the learning opportunity, the grading scheme used, language, date, and additional
information. The document may indicate achievement of a qualification or credit. The specification may
be used as a lightweight transcript. The data is at a fairly course level of granularity, much coarser than
the information discussed during the working group’s data analysis.
The Europass Diploma Supplement Application Profile of the EuroLMAI (EuroLMAI Europass DS AP) does
add some further details to the data model, including the main field of study, any title conferred, and
professional status. There are no detailed reports on competencies obtained.
HR-XML Assessment Report
http://ns.hr-xml.org/schemas/org_hr-xml/3_2/Documentation/ComponentDoc/AssessmentReportnoun.php
“A document carrying information about an assessment result. Provides scores and other details to fulfill
or respond to an assessment order. AssessmentStatusDetails is part of the data model so that tests,
whether completed or pending, can be reported upon.”
The HR-XML assessment result does have a way of indicating the competency assessed, the proficiency
level (expressed as a score, point scale, or mark), and additional details determined by the implementer
if desired. Scores include a code, interval, minimum, and maximum. Scores may be percentile, raw,
stanine, sten, T-score, Z-score, Intelligence Quotient, Scaled, normal curve, or GPA. Scores may be
represented as text as well. The specification also permits the indication of a “band,” a place on a
normal distribution of scores. Comments may be added, and attachments are supported.
HR-XML EPM
http://ns.hr-xml.org/schemas/org_hr-xml/3_2/Documentation/ComponentDoc/EPMResult-noun.php
“A result from an employee performance management (EPM) process or ‘appraisal’. An appraisal
typically consists of a number of sections, including an overall rating or score, competency ratings,
objective ratings, and other ratings (e.g., ‘core values’ is an example of a section within some appraisals
outside of the competency or objectives section). Also includes result metadata, such as appraisal dates,
‘prepared by,’ etc. Note that EPM is an HR-XML approved abbreviation for "Employee Performance
Management".”
A note about IP: “Licensee may not facilitate or assist any HR technology Standards Organization other
than Licensor in co-opting, copying, distributing, publishing or using the Work without prior written
approval of Licensor. Licensee may not create any Derivative Work for ownership by, presentation by,
distributing by, publishing by or attribution to any Standards Organization or similar entity other than
Licensor, unless specifically approved in writing by Licensor.” If we were interested in leveraging EPM,
we would clarify our ability to create derivative works, and provide appropriate attribution to all parties,
in writing with HR-XML.
4
HR-XML Position Competency Model
http://ns.hr-xml.org/schemas/org_hrxml/3_2/Documentation/ComponentDoc/PositionCompetencyModel-noun.php
“A collection of competency information associated with a position or job. A position competency model
includes references to individual competencies and references to groups of competencies that are
required or associated with a position or job. Within those groups and individual competency references
is information about associated proficiency levels (required or desired proficiency), and weightings of
the group or individual competencies among their sibilings.”
Includes desired proficiency, required proficiency, level of difficulty, measurable dimensions of the
competence (ie frequency), evidence requirements, and weight among sibling requirements, and
related competencies. This is fairly different than the MedBiquitous notion of a competency framework
in that many more details about the importance of competencies and their relation to a specific job are
detailed.
The IMS Learner Information Package
http://www.imsglobal.org/profiles/
IMS Learner Information Package (IMS LIP) provides a common format for the following types of learner
information: accessibilities; activities; affiliations; competencies; goals; identifications; interests;
qualifications, certifications and licences; relationship; security keys; and transcripts. IMS LIP is able to
indicate the level of a qualification attained, for example, journeyman pipe fitter. The Competencies
section is able to indicate skills a learner has acquired and their relationship to other activities or
qualifications captured in the LIP. It also includes transcript data. The structure is abstract and fairly
general. While IMS LIP does allow one to state attainment of a competence, it does not provide a way of
indicating the level of performance at a particular point in time.
IMS also has a Learning Information Services development project which allows for outcomes
management within an institution, but the details of that project are only visible to IMS members. (See:
http://www.imsglobal.org/lis.html)
JISC Higher Education Achievement Report
http://www.xcri.org/wiki/index.php/HEAR
“The HEAR is a means for recording student achievement in higher education (HE) and is intended to
provide more detailed information about a student’s learning and achievement than the traditional
degree classification system. It will be issued to students on graduation and will include and extend the
existing record of academic achievement - the academic transcript - and the European Diploma
Supplement.”
5
“The Centre for Recording Achievement, the Higher Education Academy (the Academy), and the Joint
Information Systems Committee are developing the HEAR. This is being overseen by the Burgess
Implementation Steering Group. A pilot group of 18 institutions, using data from students who
graduated in 2008 in four subject areas, have trialled the report, and a second trial took place in 200910.”
HEAR provides more information on the diploma, degree, or certificate (collectively known as
qualification) including the level, contents, results, and function of the qualification. The broad approach
to assessment used may be included in addition to the weighting, attempts, results, and grading
scheme. HEAR is related to CEN’s European Learner Mobility Achievement Information.
Leap 2A & MedBiquitous Educational Trajectory
http://www.leapspecs.org/2A/
http://www.medbiq.org/sites/default/files/EducationalTrajectoryProfile.pdf
Leap 2A is an e-portfolio specification supported by CETIS, funded by JISC, and developed by Simon
Grant and colleagues. Leap 2A e-portfolios may contain:



digital artifacts, including documents, audio, video, etc, made or jointly made by the portfolio
holder
expressions or text entries by the portfolio holder , including learner reflections
information about the portfolio holder, their abilities, achievements, experiences, activities,
goals, plans etc.
MedBiquitous has developed a profile of LEAP 2A that creates several vocabularies specific to epotfolios for health professions education. The result is a profilecapable of describing the path of
individual learners through one or more programs of study, including extracurricular learning, dates of
study, gaps in study, and academic withdrawal or dismissal. Integration with the Educational Trajectory
is a requirement for the Educational Achievement specification we are to build.
Mozilla Badges
https://wiki.mozilla.org/Badges
The Mozilla badges project is creating an open infrastructure that allows learners to collect and display
badges. Badges are designed to recognize skills and abilities demonstrated outside of a formal
educational environment. With funding from the MacArthur Foundation, Mozilla is developing an open
infrastructure for awarding and collecting badges. The Khan Academy (www.khanacademy.org), a nonprofit offering free online learning program, awards a variety of badges ranging from “Just Getting
Started” ( for listening to 30 minutes of videos) to “Copernicus” (proficiency in any 200 exercises) or the
more targeted “Master of Trigonometry.” The value of the badge is entirely subjective; schools and
employers may or may not place value on badges a learner has earned. In some communities, peers
6
may award badges. There is no way to connect badges to predefined competency frameworks and no
way to include assessment information.
PESC College Transcript
http://www.pesc.org/interior.php?page_id=164
PESC offers specifications for both high school and college level transcripts. The transcript includes
academic awards and honors, degree requirements, academic summary (ie GPA and program details),
course grades, additional student achievements, immunizations, and test scores. PESc participants are
based in the US and Canada; it’s unclear how well the specification would work for colleges outside of
North America. There are codes for describing the area of study for a course. It’s not clear whether
grades like “proficient” are able to be communicated using this specification. Notes may be included.
Peer benchmark data is not included in the specification. There is an Education Record User Group
which focuses on utilization and development of standardized, electronic transcripts.
PESC Education Test Score report
http://www.pesc.org/interior.php?page_id=132
“The Education Test Score Reporting standard can be used by any testing agency to fully communicate
test score information to high schools, colleges and universities, affiliated third parties (clearinghouses,
employers, research facilities, etc.), and local, state and federal agencies.”
PESC Education test Score Result provides a detailed specification for the communication of test scores.
It includes capability to provide subtest scores (ie verbal scores on SATs). It also includes different types
of scores: raw score, percent correct, scale score, labels describing a score (ie proficient), mastery scores
(ie pass fail), academic grade, gpa, norm referenced, and scores using local scoring systems (ie not
otherwise defined). Other student characteristics may be included. While there is no explicit reference
to competencies, the subtest code could be used to reference competencies.
PESC Academic ePortfolio Workgroup
http://www.pesc.org/interior.php?page_id=209
PESC also has a workgroup around the development of Academic ePortfolios. It’s not clear if these
groups have draft specifications that we could build on. A description of the vision behind the work
follows. Dana
“Sophisticated ePortfolio software is being utilized today for integrating course assignments with
learning outcomes and academic standards. The assignments are scored using well-designed rubrics,
and the data from these scores are then aggregated and disaggregated into reports showing how
individuals, courses, programs and departments are performing on a wide variety of student learning
outcomes. In addition to helping to manage accreditation systems, these reports help to drive positive
change in the way teachers teach and students learn.”
7
“Taking ePortfolios one step further is the notion of "Folio Thinking," a concept proposed by Helen Chen
at Stanford. It is a unique teaching approach that is based on supporting students who are creating their
own ePortfolios with opportunities to reflect on the experience through coaching and other techniques.
This process both enhances and authentically displays distinct levels of student achievement, knowledge
and skills that ultimately will help learner's transition into meaningful work and career development
after they graduate.”
Schools Interoperability Framework (SIF)
http://specification.sifassociation.org/Implementation/US/2.5/html/
“The Schools Interoperability Framework (SIF) is not a product, but a technical blueprint for enabling
diverse applications to interact and share data related to entities in the pK-12 instructional and
administrative environment. SIF is designed to:



Facilitate data sharing and reporting between applications without incurring expensive
customer development costs;
Enhance product functionality efficiently; and
Provide best-of-breed solutions to customers easily and seamlessly.”
SIF does have capability for communicating assessment data. They have vocabularies for defining the
assessment type (achievement test, attitudinal test, personality test, statewide, diagnostic, etc) and the
mechanism to associate the assessment with learning standards identified by a globally unique SIF
identifier. Assessments include performance levels defining score ranges indicating a certain level of
achievement. There are numerous elements related to the administration, assignment, psychometrics,
and instance details of the assessment. Assessments may have subtests. Measurement results for an
item may be included and may detail Mean Square Fit, Weighted Mean Square Fit, Revised Mean Square
Fit, Revised Point Biserial measure, Rasch Item Score, Response Correlation, Response Correlation
Squared, Z CHI Square, PValue, PointBiserial, Biserial, DiscriminationIndex, Reliability Coefficient,
Coefficient Alpha, ItemTestCorrelation, ItemVariance, and ScaleValue. SIF can encode an individual
student’s responses to an assessment and a set of scores. Scores are described using a value and a
diagnostic statement (ie shows ability to identify symmetry). Metadata can be associated with the
assessment including grade level, subject areas, and learning standards (ie competencies) among other
things. SIF also provides detailed standards for K-12 academic records.
SIF has announced a collaboration with IMS to “to develop state of the art online assessment
capabilities for students across the country.” This appears to mean they are finding ways for SIF and IMS
specifications to work together. SIF has worked with the National Center for Education Statistics (NCES),
on Common Education Data Standards (CEDS).
8
Tin Can API
http://www.adlnet.gov/wp-content/uploads/2012/02/TinCanAPI.pdf
“ADL, through a 2011 Broad Area Announcement (BAA) with Rustici Software, researched an
implementation approach to Next Generation SCORM as if there were no previous generation of
SCORM. That is, if we were to start from scratch with the web infrastructure that exists today, how
would we implement distributed learning that is interoperable and reusable?”
“The answer was simple but powerful – ‘I did this’. The idea of describing learning experiences as ‘actor
verb object’ can describe traditional learning content or may go out to other forms of content not
normally tracked. So whether ‘Jack completed the Chapter 5 Algebra exam’ or ‘Jill experienced Fighter
Pilot Training Simulation #17’, what happened can be tracked easily, but can also be re-aggregated with
other learning experiences to create something larger and possibly more meaningful.”
The MedBiquitous Technical Steering Committee and the Virtual Patients Working Group received an
update on this project from ADL on March 12. The specification does allow for the transmission of score
data and the collection of educational analytics by those with appropriate rights (ie an instructor can run
queries regarding their class). The beta version of the specification is due for release in a day or two.
There is currently no notion of competencies in the specification.
Controlled Vocabularies
Our approach should account for some important controlled vocabularies for healthcare and health
professions education.
Instructional methods, Assessment methods, and Resources
https://www.aamc.org/download/273966/data/cistandardizedvocablist.pdf
The MedBiquitous Curriculum Inventory working group has developed a set of vocabularies describing
instructional methods, assessment methods, and resources used in health professions education. The
vocabularies are being maintained by the AAMC and will be updated on a quarterly basis.
Medical Subject Headings (MeSH)
Developed by the National Library of Medicine for indexing PubMed content, MeSH is used by many
schools to index their course offerings.
Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT)
SNOMED-CT is a comprehensive, multilingual healthcare terminology designed for use in healthcare
information systems. The International Health Terminology Standards Development Organization
(IHTSDO) administers SNOMED CT licenses. IHTSDO members manage the release, adaptation,
distribution and sub-licensing of SNOMED CT and other products of the Association within their
Territory. In the US, the National Library of Medicine distributes SNOMED CT at no cost. IHTSDO includes
9
15 member countries, including the US and the UK. For full details, see:
http://www.ihtsdo.org/members/
Healthcare LOM
Developed by MedBiquitous, Healthcare LOM provides vocabularies and recommended lists of terms for
the following:


















Activity sponsorship(joint/direct)
Participation modality (conference/workshop, technology based, on the job, print)
Activity delivery (live/not live)
Accrediting bodies (recommended list for CE)
Activity certification (recommended list for CE)
Credit type (CME, CE, CNE, CPE, and CHES, CPD)
Credit unit (CECH, CEH, CEU, Cognate, Contact Hour, Credit, Hour, Unit, Credit Hour, and Point)
Pacing (learner paced/provider paced)
Audience category (general, patient, caregiver, professional)
Profession (recommended list)
Specialty (recommended list)
Reading level (recommended list)
Orientation (axial, coronal, horizontal, longitudinal, sagittal, transverse)
Medical Image Type (11 terms)
Specimen type (cell, organ, organ system, organelle, tissue)
Learning resource type (expanded on LOM vocabulary)
Educational context (9 terms)
Classification purpose (expanded on lom vocabulary)
Use of Competencies
Whatever approach adopted should accommodate references to competencies, learning objectives, etc.
The MedBiquitous Competency Object and Competency framework specifications provide a mechanism
for uniquely identifying a competency statement/outcome/learning objective/milestone and collections
of competency statements/outcomes/learning objectives/milestones. A conceptual model of this work
is available at: http://groups.medbiq.org/medbiq/display/CWG/Conceptual+Model
10