Monitoring methodologies
for web-accessibility
in the European Union
SMART 2014/0061
INITIAL COUNTRY SCREENING ON WEB ACCESSIBILITY
MONITORING METHODOLOGIES
Date: 15th July 2015
Internal identification:
Contract number: 30-CE-0679566/00-74
SMART 2014/0061
Disclaimer:
By the European Commission, Directorate-General of Communications Networks,
Content & Technology.
The information and views set out in this publication are those of the author(s)
and do not necessarily reflect the official opinion of the Commission. The
Commission does not guarantee the accuracy of the data included in this study.
Neither the Commission nor any person acting on the Commission’s behalf may be
held responsible for the use which may be made of the information contained
therein.
© European Union, 2015. All rights reserved. Certain parts are licensed under
conditions to the EU.
Reproduction is authorised provided the source is acknowledged.
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
Table of contents
INITIAL COUNTRY SCREENING ON WEB ACCESSIBILITY MONITORING
METHODOLOGIES ................................................................................................................................. 1
OVERALL INTRODUCTION.................................................................................................................................... 3
Type I - Initiatives initiated in order to verify compliance with national provisions in the
field of web-accessibility................................................................................................................................... 4
Type II – Initiatives taken under the initiative of public administrations even if there are
no legal obligations in the field ..................................................................................................................... 9
Type III - Initiatives taken under the initiative of non-governmental actors including
organisations of persons with disabilities and private sector's companies .......................... 14
PRELIMINARY CONCLUSIONS ............................................................................................................................18
2
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
Overall introduction
The workplan of the current study includes a dedicated strand of empirical work
focusing on the generation of supportive evidence (WP1). According to the terms
of reference, this is to enable the generation of an “average” monitoring
methodology for each of the three types of web accessibility monitoring measures
defined by the European Commission. To this end, a two-staged methodological
approach has been adopted. In a first step, an initial screening of 30 countries to be
covered by the current study was undertaken in relation to any web accessibility
monitoring initiatives which have been conducted during the last five years (Task
1.2). This will be followed by an in-depth investigation of up to 15 selected
initiatives (Task 1.3) during the next stage of the overall study.
In the following, outcomes of the initial country screening (step 1) are presented.
Overall, 43 initiatives of relevance to the current study were identified and
described by national correspondents according to a common research template.
Together, these cover three types of monitoring measures as defined by the
European Commission, namely:
1. Initiatives initiated in order to verify compliance with national provisions in
the field of web-accessibility, such as regulation or guidelines (Type I);
2. Initiatives taken under the initiative of public administrations even if there
are no legal obligations in the field (Type II);
3. Initiatives taken under the initiative of non-governmental actors including
organisations of persons with disabilities and private sector's companies
(Type III).
In the following subsections, outcomes are reported according to this typology
Error! Reference source not found. and it presents an overview of the number of
web accessibility monitoring initiatives identified in each of the countries covered
according to the three types. In line with this study’s overall objective, the analysis
presented in the reminder of this chapter does however put the focus on a
comparison of the identified initiatives by methodological approaches adopted for
the purposes of web accessibility monitoring, rather than on a country-by-country
comparison.
3
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
Table 1 – Overview of web accessibility monitoring initiatives identified
according to type
Country
AT
BE
BG
CY
CZ
DK
EE
FI
FR
DE
GR
HU
IE
IT
LV
LT
LU
MA
NL
NO
PL
PT
RO
SK
SI
ES
SE
UK
USA
HR
Type I
Type II
Type III
1
1
2
1
5
1
1
1
1
3
2
7
3
2
6
2
2
1
2
1
1
Type I - Initiatives initiated in order to verify compliance with national
provisions in the field of web-accessibility
In total, 25 web accessibility monitoring initiatives could be identified which have
been initiated during the last five years to verify compliance with existing web
accessibility provisions. As can be seen from Error! Reference source not found.
overleaf, the overwhelming majority of these initiatives were designed as a nonregular monitoring effort, i.e. either they represent a one-off monitoring initiative
(10) or repetition occurred in irregular intervals (8). Overall, annual (5) and biannual (1) web accessibility monitoring was less frequently applied. In one case
(other), a tool was developed that monitors public websites on a continuous basis
according to a confined number of automatically performed tests, thereby
displaying statistics on outcomes on a publicly available web site. Outcomes are
hoped to stimulate further in-depth evaluation in cases where any accessibility
barriers have been spotted by the automatic tool.
4
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
Figure 1 – Overview of Type I initiatives according to repetition intervals
other; 1
irregular
monitoring
intervals; 8
bi-annual
monitoring; 1
one-off
monitoring; 10
annual
monitoring; 5
n=251
When it comes to different types of web sites covered by the monitoring initiatives
identified under this category, public web sites across different governance levels
were most frequently addressed by the identified monitoring initiatives, when
compared with web sites operated by non-governmental public services/bodies or
commercial web sites (Error! Reference source not found. overleaf). This result
may not come as a surprise as existing legal or regulatory provisions concerning
web accessibility do in most countries concern web sites and online services
operated by public bodies rather than commercial parties.2 It is also worth to be
noted that in a given country different types of web sites may be addressed by
different web accessibility monitoring initiatives, which may again vary in terms of
methodological approaches adopted on a case-by-case basis.
1
The acronym “n” represents the number of Type I initiatives identified for the purposes of this study.
2
See empirica, WRC, Funka NU and Technosite (2013): Study for Assessing and Promoting E–
Accessibility. Final Report.
(http://ec.europa.eu/information_society/newsroom/cf/dae/document.cfm?doc_id=3163 , latest
access: 29th June 2015)
5
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
Figure 2 – Overview of Type I initiatives according to types of web sites
covered in relation to web site ownership
commercial
public, non-government
local government
regional government
national government
0
5
10
15
20
n=25
Figure 3 - Overview of Type I initiatives according to assessment technique
applied
other
combined automatic & manual expert &
end user
combined automatic & manual expert
manual user only
manual expert only
automatic only
0
2
4
6
8
10
n=23, missing =2
When it comes to particular assessment techniques applied for the purposes of
web accessibility monitoring (Error! Reference source not found.) about half of
the Type I initiatives relied upon a combination of automatic and manual
assessment procedures (13), whereby manual end users assessment can be less
frequently observed (4) when compared with manual expert assessment (9). A
minority of initiatives relied upon automatic assessment techniques exclusively
(4).
6
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
Beyond this, other approaches were pursued as well (6). These include, e.g.,
surveys of web site owners on whether or not a web accessibility monitoring has
been conducted for their web sites and if so what the outcome was. In such cases
details on particular assessment techniques applied or on other methodological
aspects were usually not available. Also, some initiatives relied upon qualitative
approaches which were not immediately directed towards quantifying the
proportion of web sites which do or do not comply with a given web accessibility
standard or relevant guidelines. These focused e.g. on assessing whether any
procedures and workflows had been put in place by the web site owner
organisations to ensure that the design of a given web site and/or content
provided electronically were indeed accessible to users with disabilities.
The majority (15) of Type I web accessibility monitoring initiatives relied upon
WCAG 2.0 – or ISO International Standard “ISO/IEC 40500” respectively - as a
benchmark against which selected web sites were assessed (Error! Reference
source not found.). In some cases (9) dedicated national standards/guidelines
were applied, whereby individual assessment criteria may more or less overlap
with WCAG 2.0. Other (1) benchmarks were applied for the purposes of the
assessment as well, e.g. procedural and workflow related guidelines/standards.
Figure 4 - Overview of Type I initiatives according to assessment criteria
applied
other; 1
mandatory
national
guidelines /
standard; 8
voluntary
national
guidelines /
standard ; 1
WCAG 2.0; 15
n=25
Only a minority (5) of the identified Type I monitoring initiatives relied on a selfassessment of compliance with relevant web accessibility standards/guidelines by
the web site owners themselves (Error! Reference source not found.). In
contrast, a clear majority of the initiatives (20) relied upon third-party assessment.
When it comes to the amount of web sites included in the assessment procedure
numbers vary a lot across the individual monitoring initiatives identified, ranging
from 11 to 3100 web sites assessed per imitative. On average, 488 web sites were
assessed across all monitoring initiatives. The majority of cases – about three
7
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
quarter – the number of web sites assessed ranged however below the statistical
average (Error! Reference source not found.).
Figure 5 - Overview of Type I initiatives according to parties performing the
assessment
selfassessment; 5
third party
assessment;
20
n=25
Figure 6 - Overview of Type I initiatives according to numbers of web sites
assessed3
3500
3000
2500
2000
1500
1000
500
0
n = 25
Some sort of information on outcomes was made publicly available by almost all
monitoring initiatives identified (Error! Reference source not found.), either
3
For illustrative purposes, each of the Type I monitoring initiatives analysed for the purposes of this
study is plotted against the number of web sites assessed in the framework of the initiative under
investigation. Each of the blue coloured rhombi represents one initiative.
8
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
online or in a printed format or both. Only in one case no information on outcomes
was published. Synthesis reports (22) and selected statistics (16) represent the
preferred content published by the monitoring initiatives. Raw data were made
publicly available only by few initiatives (3). In one case a hand book was
generated on the basis of monitoring outcomes to share experiences on how
identified failures in complying with relevant web accessibility
standards/guidelines can best be avoided.
Figure 7 - Overview of Type I initiatives according types of publicly available
feedback
other
raw data
selected statistics
synthesis report
0
5
10
15
20
25
N = 25
Earlier research points into the direction that web site owner organisations seem
to perceive a risk that country-wide or even EU-wide monitoring might turn out to
as a bureaucratic exercise rather than an operationally useful endeavour.4 Against
this background, it seems worth to be noted that few monitoring initiatives
provided specifically tailored feedback to the owner organisations of the web sites
assessed. In some cases, detailed metrics were provided for individual web sites as
part of the synthesis report, e.g. in terms of a dedicated data annex. In case of a
one-off monitoring initiative identified, a detailed outcome report was prepared
for each web site that had been tested. The expert who had conducted the
assessment was available for providing up to 30 hours counselling to the web site
owner organisation in order to support the implementation of potentially required
improvements. This was then followed by a second audit (expert audit only). In
another case a dedicated tool was made available to web site owner organisations
for reanalysing their web site on a continuous basis.
Type II – Initiatives taken under the initiative of public administrations even if
there are no legal obligations in the field
4
See Funka Nu, empirica and WRC (2014): Measures to Improve Accessibility of Public Websites in
Europe. (http://universaldesign.ie/Web-Content-/Measures-to-improve-accessibility-of-publicwebsites-in-Europe.pdf, latest access: 29th June 2015)
9
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
When it comes to web accessibility monitoring initiatives which have been
initiated by public administrations where no legal obligation exists (Type II), six
initiatives could be identified across the 30 countries covered by the current study.
As can be seen from Error! Reference source not found., most of them were
conducted more than once (5), albeit not always according to regular repetition
intervals (2).
Figure 8 – Overview of Type II initiatives according to repetition intervals
one-off
monitoring; 1
irregular
monitoring
intervalls; 3
annual
monitoring; 2
n=6
When it comes to different types of web sites covered by the monitoring initiatives
identified under this category, the majority (5) of initiatives addressed public web
sites across the national and/or regional governance levels (Error! Reference
source not found.). Web sites operated by commercial parties and by nongovernmental public bodies and/or local governments were addressed in one case
each.
Figure 9 – Overview of Type II initiatives according to types of web sites
covered in relation to web site ownership
10
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
commercial
public, non-government
local government
regional government
national government
0
0,5
1
1,5
2
2,5
3
3,5
n=7
Figure 10 - Overview of Type II initiatives according to assessment technique
applied
other
combined automatic & manual expert &
end user
combined automatic & manual expert
manual user only
manual expert only
automatic only
0
1
2
3
4
5
n5=6 , missing6=1
When it comes to particular assessment techniques applied most initiatives relied
upon a combination of automatic and manual assessment procedures (4), whereby
only one initiative included manual end users assessment (Error! Reference
5
The abbreviation “n” stands for the number of Type II monitoring initiatives identified for the purposes
of the current study.
6
The abbreviation “missing” stands for the number of initiatives for which no information was available
in relation to the particular feature illustrated by the figure. This reads as follows: For one of the six
Type II initiatives identified no information was available on the assessment technique applied.
11
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
source not found.). None of the identified initiatives relied upon automatic
assessment techniques exclusively. Another approach observed (1) includes
manual assessment by instructed students.
The majority of Type II web accessibility monitoring initiatives relied upon
voluntary national standards/guidelines (3) or WCAG 2.0 (1) as a benchmark
against which web sites were assessed, whereby the former may overlap with
WCAG requirements to greater or lesser extent (Error! Reference source not
found.). Dedicated assessment criteria were developed in two cases, whereby it is
not clear to what extent these referred to WCAG.
12
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
Figure 11 - Overview of Type II initiatives according to assessment criteria
applied
other
WCAG 2.0
voluntary national guidelines / standard
0
1
2
3
4
n=6
Only a minority (2) of the identified Type II web accessibility monitoring initiatives
relied on a self-assessment by the web site owner organisations (Error!
Reference source not found.). Most initiatives (4) relied upon third-party
assessment when it comes to checking compliance with relevant web accessibility
standards or guidelines. The amount of web sites included in the assessment
procedure varied less strongly when compared with the Type I initiative under
investigation. Actual numbers of web sites sampled per initiative range from 11 to
307. On average, 151 web sites were assessed across all Type II monitoring
initiatives. In three quarters of the Type II initiatives identified, however, volumes
of web sites assessed do not exceed 100 sites (Error! Reference source not
found.).
13
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
Figure 12 - Overview of Type II initiatives according to parties performing
the assessment
selfassessment; 2
third party
assessment; 4
n=6,
Figure 13 - Overview of Type II initiatives according to numbers of web sites
assessed7
350
300
250
200
150
100
50
0
N=6
As in the case of the Type I initiatives analysed earlier, some sort of information on
outcomes was made publicly available by almost all monitoring initiatives falling
under this category (Error! Reference source not found.). Again, synthesis
reports (4) and selected statistics (4) represented the preferred content published.
Raw data were made publicly available only by one initiative.
7
For illustrative purposes, each of the Type II monitoring initiatives analysed for the purposes of this
study is plotted against the number of web sites assessed in the framework of the initiative under
investigation. Each of the blue coloured rhombi represents one initiative.
14
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
Figure 14 - Overview of Type II initiatives according types of publicly
available feedback
raw data
selected statistics
synthesis report
0
1
2
3
4
5
n=6
Type III - Initiatives taken under the initiative of non-governmental actors
including organisations of persons with disabilities and private sector's
companies
In total, 15 web accessibility monitoring initiatives were identified that were
initiated by non-government organisations or commercial parties (Type III). As
shown by Error! Reference source not found., almost three quarters of these
were conducted more than once (11), albeit repetition frequently occurred at
irregular time intervals (7). One initiative (other) has been regularly conducted in
conjunction with the national presidential elections.
Figure 15 – Overview of Type III initiatives according to repetition intervals
other; 1
one-off
monitoring; 4
irregular
monitoring
intervalls; 7
annual
monitoring; 3
n=15
15
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
As shown by Error! Reference source not found., public web sites at the regional
and national governance levels (3 each) were slightly less frequently covered
when compared with commercial web sites (6), non-governmental public web
sites (5) and local government web sites (4). In relation to web sites or online
surveys operated by businesses it is worth to be noted that often particular
business sectors are covered – e.g. the largest banks or railway carriers – rather
than commercial web sites in general.
Figure 16 – Overview of Type III initiatives according to types of web sites
covered in relation to web site ownership
commercial
public, non-government
local government
regional government
national government
0
1
2
3
4
5
6
7
n=15
Figure 17 - Overview of Type III initiatives according to assessment
technique applied
other
combined automatic & manual expert &
end user
combined automatic & manual expert
manual user only
manual expert only
automatic only
0
1
2
3
4
5
6
N=15, missing value=1
16
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
Two in three monitoring initiatives (10) relied upon automatic testing, whereby
half of these augmented automatic testing procedures with manual expert testing
(5). In contrast, two initiatives relied upon manual testing only, either by experts
or by users (Error! Reference source not found.). Other approaches applied
include manual assessment by instructed students (1) and a manual assessment
jointly conducted by experts and users (1).
For the assessment procedure, more than half of the Type III initiatives that could
be identified relied upon WCAG 2.0 (8) as a benchmark against which web sites are
assessed (Error! Reference source not found.). National guidelines – either
mandatory ones (3) or voluntary ones (3) – were applied in most other cases. One
initiative applied a set of 15 selected “basic” criteria for the purposes of the
assessment.
Figure 18 - Overview of Type III initiatives according to assessment criteria
applied
other; 1
mandatory
national
guidelines /
standard; 3
voluntary
national
guidelines /
standard ; 2
WCAG 2.0; 8
n=15, missing value = 1
All Type III monitoring initiatives relied upon third party assessment. The
numbers of web sites included in the assessment procedure range from 10 to 1000
per initiative. On average, 205 web sites were assessed across all monitoring
initiatives. However, in almost two third of the initiatives the number of web sites
sampled was well below 200 (Figure 19 overleaf).
17
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
Figure 19 - Overview of Type III initiatives according to numbers of web sites
assessed8
1200
1000
800
600
400
200
0
n = 15, missing = 1
Information on outcomes was made publicly available by all monitoring initiatives
identified under this category (Type III). Synthesis reports (9) and selected
statistics (5) represent the preferred content published. Raw were not made
available by any of the Type III initiatives. In one case outcomes were published
only in an article, but no dedicated report was made available.
Figure 20 - Overview of Type III initiatives according types of publicly
available feedback
other
1
raw data
0
selected statistics
5
synthesis report
9
0
2
4
6
8
10
N = 15
8
For illustrative purposes, each of the Type III monitoring initiatives analysed for the purposes of this
study is plotted against the number of web sites assessed in the framework of the initiative under
investigation. Each of the blue coloured rhombi represents one initiative.
18
SMART 2014/0061. Country screening on web accessibility monitoring
methodologies
Again, few monitoring initiatives seem to have provided dedicated feedback to the
owner organisations of the web sites that were assessed, with a particular view to
helping these in addressing potentially identified shortcomings in that respect. In
one case, dedicated outcome reports were e.g. made available to web site owner
organisations on request. In another case, results were presented in dedicated
meetings with umbrella organisations representing a particular industry sector
covered by the monitoring effort in questions.
Preliminary conclusions
Based on the hitherto discussed evidence, Error! Reference source not found.
overleaf shows the most frequently observed methodological approaches
identified across the three types of web accessibility monitoring measures as
defined by the European Commission. It is however worth to be noted that
individual initiatives deviate considerably from the “average” derived from the
available data in relation to each particulate type of initiative.
Overall, the emerging “average” patterns show that web accessibility monitoring
which is repeated at regular intervals has remained an exception as of today. Also,
there is some variation when it comes to the types of web sites covered. While
Type I initiatives most frequently cover public web site at the national, regional
and local governance levels, most Type II and Type III initiatives tend to focus on
one type of web sites only. WCAG 2.0 represents the most commonly applied
standard, albeit national standards may to some extent deviate from WCAG 2.0 in
one or another regard.
19
Table 2 – Most frequently observed methodological approaches according to Type of imitative
Methodological dimension
Type I *
Type II**
Type III***
Frequency of monitoring
One-off monitoring
Irregular monitoring intervals
Irregular monitoring intervals
Types of web sites covered
National government web sites,
regional government web sites, local
government web sites
Regional government web sites
Commercial web sites
E-accessibility guidelines/standards applied
Mandatory standard / guidelines (partly
based on WCAG 2.0)
Voluntary national standards/
guidelines (partly based on WCAG 2.0)
WCAG 2.0
Average no. of web sites assessed
488
151
205
Assessment approach
Third party assessment
Third party assessment
Third party assessment
Assessment techniques applied
Combined automatic assessment and
manual expert assessment
Combined automatic assessment and
manual expert assessment
a) Combined automatic assessment
and manual expert assessment
b) Automatic assessment only
Means of public outcome reporting
Synthesis report and selected statistics
Synthesis report and selected statistics
Synthesis report and selected statistics
Dedicated feed back to web site owner organisations
beyond public reporting
No
No
No
*n=15, **n=6. ***n=15
The number of web site assessed per initiative varies quite a lot within each
individual type of monitoring initiatives, albeit the statistical average turns out as
somewhat higher for Type I initiatives when compared with Type II and Type III
initiatives. Across all three types combined automatic assessment and manual
expert assessment is most frequently applied. Also synthesis reports and selected
statistics are the most frequently published outcome format across all three types
of web accessibility monitoring initiatives. Likewise, most initiatives do not give
specifically tailored feedback to the owners of the web sites that have been
assessed.
During the next stage of the study, these methodological patterns will be further
detailed by means of an in-depth investigation of a selected number of initiatives