Data accessibility, performance and security for a hosted
Primo
•
Robert Watt (Service Development Manager,
Information Resources, Library Services, King’s College London)
•
email: robert.watt@kcl.ac.uk
1
Agenda
•
•
•
•
•
•
•
Brief introduction to King’s
Background to the Primo implementation project at King’s
Accessibility: requirements; test methodology; outcomes and resolution
Performance: requirements; test methodology; outcomes and
resolution
Security: requirements; test methodology; outcomes and resolution
Lessons learned
Questions
2
Brief introduction to King’s
•
•
•
•
•
5 campuses around London
21,500 FTEs
9 Schools
Collections: >855,000 print books; >30,000 e journals; >338,000
e books and >700 databases
Ex Libris products used: Aleph; bX; Metalib; Primo; SFX; Ustat
3
Background to the Primo implementation project at King’s
•
•
•
•
•
•
•
Tendered for a discovery tool in mid 2012
Key strength of Primo was its integration with Aleph
Primo implementation project kicked off October 2012
Go live date initially December 2012 – optimistic!
King’s rigorous testing identified problems around accessibility;
performance; security - reported to EXL in December/January
2012/2013 – go live postponed
King’s worked with EXL first half 2013 to resolve go live issues
Primo (branded “Library Search”) went live on 29/05/2013
4
King’s Primo
• librarysearch.kcl.ac.uk
5
Accessibility: King’s requirements for a cloud service
•
•
•
•
King’s places great importance on accessibility for its users: the
College’s Strategic Plan specifically mentions “addressing accessibility
issues”
King’s complies with the Equality Act 2010, including its provisions on
disability
King’s requires end-user and back-office web interfaces should
conform with current World Wide Web Consortium (W3C) Web Content
Accessibility Guidelines (WCAG) to Level A
Primo is required to work with King’s College London accessibility tools
including Dragon naturally speaking software and Supernova screen
reading software
6
Accessibility testing: methods and tools used
•
•
King’s tested the Primo web pages against WCAG guidelines with
Firefox’s HTML validator add on; Achecker software and Total Validator
Primo tested with the following accessibility tools:
Dragon Naturally Speaking 11.5 speech recognition software
SuperNova 11.54 screen reading software
7
Accessibility testing: outcomes for web page validators
•
Verification that the web interface complies with the WCAG standards
resulted in 1,500 errors i.e. malformed html and css
–
–
•
Achecker
Input validator (Total Validator – the same software that EXL use)
Tests conducted using King’s accessibility software by accessibility
expert
–
–
Full interface revealed some deficiencies
Mobile interface seemed to work better
8
How did King’s work with Ex Libris to resolve the issues
around accessibility? (1)
•
•
•
•
Issues were discussed in a number of conference calls between King’s
and EXL’s security experts
EXL service pack 4.4 to address many of the issues identified by King’s
note: King’s to verify how many have been resolved
EXL were validating the wrong pages i.e. the PDS log in page rather
than the actual Primo pages
EXL said that they did not identify the accessibility issues discovered by
King’s, chiefly because: “they resulted from a change in the HTML that
was not visible in the user interface, and was therefore not checked the
way that changes to the user interface are typically checked. We have
amended our accessibility procedures so that this will not be repeated”
9
How did King’s work with Ex Libris to resolve the issues
around accessibility? (2)
•
•
EXL provided King’s with a written commitment to meet the
accessibility requirements expressed in King’s tender document for
Primo
King’s have created an accessibility link in the FE to a mobile version of
the interface. Because the mobile version is less verbose it makes it
faster when used with a screen reader.
10
Performance: King’s requirements for a cloud service
•
•
•
•
King’s Primo tender stipulated that response times should be under 5
seconds to retrieve and display results from the single Primo index
King’s identified that the performance of Primo in testing prior to go live
vastly exceeded the 5 seconds threshold, and with only a handful of
users!
King’s didn’t specify the amount of load in Primo regarding the number
of searches per minute on the tender requirement
King’s expected the load to be higher than the Aleph load of 140 – 170
peak “load”
11
12
Performance testing: methods and tools used
•
•
•
King’s performed load testing of Primo with Apache Jmeter to:
 Execute a search and go to the results
 Access the user’s Library Account
The load testing was conducted with an incremental increase of the
number of users mimicking use of Primo
Methodology
–
–
Multiple locations
Search expressions extracted from Aleph Apache log - “160,000”
13
Performance testing outcomes (1)
•
On first test run system brought to a halt*
–
•
On second run PDS server brought to a halt
–
•
Script amended to login and logout to close session.
On third run and later runs
–
–
–
•
Due to not closing the sessions
Script amended to do searches in a loop without opening new sessions
Search loops paused for one minute before proceeding
Number of simultaneous users gradually increased until system brought a halt
Final run was successful up to 200 user searches per minute
* : more than a 30 second response time
14
A note on how server sizing is performed by Ex Libris
Number Users* = FTE/200
King’s sizing = 20,000 users/ 200 = 100 users (normal usage)
* A user is defined as doing one search per minute
15
How did King’s work with Ex Libris to resolve the issues
around performance?
•
•
•
•
EXL made configuration changes on the production server which
resolved the performance issues: the major initial cause being zombie
Apache processes that over loaded the system, then minor
adjustments to the application server’s environment
EXL and King’s negotiated a formal commitment from EXL to resize the
King’s Primo server after go live, should it be required i.e. if it exceeded
predicted usage of 200 searches per minute and caused performance
issues
King’s redrafted their SLA with Ex Libris to specify different categories
of performance requirements e.g. a P1 rated category (not available to
multiple users) has a resolution upper limit of 2 working days
Note: EXL said this was the first time they’d set specific response
targets for any customer in this way
16
How do King’s measure performance on an ongoing basis
•
King’s uses a centralised monitoring platform called Nagios
–
–
Verifies the system is up by calling the xml server and searching for a particular title
Uses Jmeter’s script to verify 5 seconds delay in search requests
•
•
•
Primo
BBC (external web page for comparison
Future improvement
–
Supply library with periodic performance report
17
Example of Nagios report
18
Security: King’s requirements for a cloud service
•
•
•
•
King’s Primo tender stipulated that Primo “must comply with all relevant
data protection and data security law and best practice, including the
UK Data Protection Act, and preferably be stored on servers in the
European Economic Area (EEA)”
King’s requires the encryption of the sign in pages for Primo FE; and
the back office for production and staging
King’s requires that personal or confidential information travel through
an encrypted (HTTPS) tunnel. This includes the personal data in the
FE and the OvP interface with ALEPH server
King’s requires permission from the vendor to run penetration (PEN)
tests on any cloud based system it subscribes to
19
Security: methods and tools used
•
King’s used several Penetration software applications to assess the
security compliance of Primo. All these tools were used with Vanilla
configuration and executed using black-box PEN testing. No Gray
PENetration methods were used. Some of the tools used were: Burp
Scanner and Nessus Vulnerability Scanner
20
Security testing outcomes
•
PENetration tests
–
•
Revealed serious XSS reflection* issues
King’s security policies
–
–
All user personal information needs to travel in secure encrypted channels (HTTPS)
All login screens need to travel through secure encrypted channels (HTTPS)
* XSS reflection – is an attack where a user can inject javascript on the
end-users browser
21
How did King’s work with Ex Libris to resolve the issues
around security? (1)
•
•
•
•
EXL escalated resolution of security issues to their senior staff and held
a number of conference calls with King’s and EXL security experts
Shlomo Sanders (EXL Chief Technology Officer) visited King’s in June.
EXL implemented service pack (4.3) to address security fix, which
resolved King’s original issues
King’s then re-ran the PEN tests and uncovered new security issues
22
How did King’s work with Ex Libris to resolve the issues
around security? (2)
•
•
•
•
•
EXL installed a local hot fix to address these latest security issues
King’s reported that the hot fix had not fixed these security issues
EXL installed another hot fix which did resolve the outstanding issues
EXL enabled HTTPS on all Primo web pages (note: King’s wanted just
the sign in page encrypted) - this caused problems on BO staging
server as King’s lost access to it for one month!
OVP encryption of data transfer – Ex Libris are developing this and
plan to release in a service pack in early 2014
23
Lessons learned: Communication
•
•
•
Be frank and open in exchanges with the vendor
Direct communication between experts on both the customer and
vendor side yields results
EXL demonstrated flexibility and a willingness to adapt their procedures
to address King’s requirements as a result of fruitful communication
between King’s and EXL
24
Lessons learned: Project and implementation
•
•
•
•
•
•
When drawing up project tender requirements, ensure that all
requirements are included. These include an up-to-date figure for FTEs
to ensure that the Primo server is sized appropriately
Be transparent at the tender stage about how you intend to test Primo.
Be explicit about how you intend to test the system
Ensure SLAs are clearly and unambiguously defined before the
contract is signed, particularly around timescales for support where
there are problems, and performance
Ensure all requirements for accessibility; performance; security are
defined at the tender stage and at contractual level (before anything is
signed)
Specify in the tender the specific (particularly third party) software that
Primo is required to work with e.g. Dragon
Consider the option of service credits to compensate for any failure in
SLAs
25
Lessons learned: Testing
•
•
•
•
Clarify the test parameters before testing i.e. exactly what is being
tested and which tools will be used by both customer and vendor to
ensure that results are comparable
Where tools differ, coordinate testing to ensure both vendor and
customer are testing exactly the same data with the same outcomes
Conduct rigorous testing internally on cloud based systems – don’t
assume that things are ok (just because the vendor says so)
Ensure that the service is monitored regularly for performance issues
with monitoring tools.
26
Questions?
?
27