DIE ÖSTERREICHISCHE BIBLIOTHEKENVERBUND
UND SERVICE GMBH
Primo in Austria – 4 years of operation
Ulrich Leodolter
System administrator
The Austrian Library Network and Services Ltd (OBVSG)
10. September 2013, IGeLU Berlin
Agenda
Primo in Austria today
● How it all began
● Evolution of Primo Versions
● Authentication
●
Monitoring Primo
● Sysadmin Tools
● Final Notes
●
2
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Primo in Austria today
The Austrian Library Network consists more than 80 academic libraries
11 Institutions already live with Primo, more will come.
Instiution
3
SPO
Number of PNX
Austrian Central Catalog
2010-04
11127007
Austrian National Library
2011-05
4744689
University of Vienna
2010-03
3785804
University of Graz
2011-02
2273484
University of Salzburg
( 2012-12 )
2159670
University of Innsbruck
2009-10
2076265
University of Klagenfurt
2012-12
875239
University of Economics and Business
2011-07
730792
Vienna University of Technology
2013-02
692458
Medical University of Vienna
2013-05
211123
University of Applied Arts Vienna
2011-11
169673
University of Veterinary Medicine Vienna
2010-07
113527
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Primo in Austria today – Institutions and Views
4
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
How it all began
Primo Pilot Project in started in 2009
• University of Vienna
• University of Innsbruck
First Primo 2.1 Staging System in April 2009
• Operating system RHEL 5.3
• Single hardware platform using KVM Virtualization
• Primo Topology FE,SE(2+1),BO,DB
Development of first Tools started
• PPS (Primo Publishing System)
• Plugins (Integration of local repository eDOC)
5
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
How it all began – first Production Instance
6
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Evolution of Primo – Version 2
Production Instance
• First version in October 2009
• Two institutions live respectively in beta phase
It was important to us because ...
• Investment and setup of hardware for the next 5 years
• Implementing all the workflows and data processing tools
• Get used to the to completely new system
7
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Evolution of Primo - Version 2 Hardware Diagram
8
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Evolution of Primo - Version 3
Migration from Version 2 to 3 started in Dec 2010
• Primo software upgrade and service pack installation was done by Ex Libris staff
• Migration took about 2 weeks including post upgrade checks
• More than one downtime during migration
Important new features of Version 3
• Improved user interface usability and customizability
• Opac via Primo
The system constantly grows
• New institutions, features and projects
• Primo began to replace Aleph OPAC
• Shutdown end of February 2013
9
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Evolution of Primo – Version 4
Primo Version 4 Early Adopter
• Invitation to become an early adopter of Primo 4 in December 2011
Motivation
• Test new multi-tenancy back office features
• See how the Upgrade Express Kit works
• Report bugs and possible enhancements in an early phase
• Production ready?
Implementation
• Staging system was duplicated (new virtual machines)
We are pleased to inform you that the early release of Primo version 4.0 will be available for you
on April 4th, 2012. We are looking forward to working closely with you during the Primo v.4.0
early adopters’ phase.
• Oracle RAC used to have production like environment
10
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Evolution of Primo – Version 4 Upgrade
Requirements for Production Upgrade
• Minimize planned downtime
• Minimize risk if upgrade fails
Infrastructure changes needed to fulfill requirements
• Hardware upgrade (Memory, NAS) for virtualization. In Summer 2012 Hardware was ready.
• P2V - Physical to Virtual finished in November 2012 (Primo 3 virtualized except Database)
• Everything was done without shutting down Primo, topology allowed us to upgrade one by one
How long will the Upgrade run? - Internal Migration needed
• Reuse Oracle RAC from EA project, export and import Primo 3 database
• Install 7 new virtual machines for Primo 4 (2xFE, 4xSE, 1xBO)
• “copy” Primo 3 to new machines and adjust hostnames.
• Software and DB Upgrade successfully finished end of December 2012
11
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Evolution of Primo – Version 4
12
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Evolution of Primo – Version 4
Production upgrade schedule
• 14. Jan 2013:
Primo V3 in read-only mode
• 14. Jan 2013:
Primo V3 clone including index files
• 15. Jan 2013:
Primo V3 database export finished
• 16. Jan 2013:
Primo V3/V4 database import finished
• 16. Jan 2013:
Primo V4 Upgrade Express Kit started (DB upgrade)
Wed Jan 16 12:46:47 CET 2013 Upgrade db - started
Sun Jan 27 20:39:43 CET 2013 Upgrade db - completed
• 03. Feb 2013:
Primo V4 Live
From: Ulrich Leodolter <ulrich.leodolter@obvsg.at>
To: primo-info@obvsg.at
Subject: [Primo-info] Primo Version 4 Live
Date: Sun, 03 Feb 2013 23:25:55 +0100
13
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Evolution of Primo – Version 4
Why did the DB Upgrade on 27M pnx run for 11 days ?
14
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Authentication
PDS and Single Sign On using Shibboleth
• In 2009 Shibboleth Authentication was implemented by Ex Libris for first 3 Institutions
• Precondition: ACOnet (Austrian Academic Computer Network) Identity Federation since 2008
• Implementation was very time consuming (Communication between OBVSG, Institutions, IT-Support
and Ex Libris)
• Authentication was not important at that time
• “Cross Login Problem” from the beginning (possibility of automatic login when switching views)
Primo 3 and Opac via Primo
• Authentication became important because of new functionality
• Link from Primo user to the ILS system (Aleph) required
• Synchronization of Users between Shibboleth(Institution) and Aleph(Library) was/is a real challenge
• “Cross Login Problem” still not resolved
15
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Authentication
PDS Today
• Central PDS on Primo Frontend used for Primo, Metalib and Aleph
• “Cross Login Problem” almost solved, PDS SaaS mode (one Cookie per institution)
• Application Cookie Problem can be solved by using different domains e.g.
http://catalogplus.tuwien.ac.at (catalogplus.tuwien.ac.at is an alias for search.obvsg.at)
Central PDS vs. PDS at each Institution
• Primo allows to configure PDS at institution level
• It would be possible to use PDS at Aleph but what if ILS moves to the Cloud?
• To configure Shibboleth and PDS is not an easy task, central PDS allows reuse of configuration
• Institutions need no to care about, it simply works ;-)
16
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Monitoring Primo
Hardware Health and Operating System
• Icinga (Nagios) Open Source Monitoring (addons NagVis,NagTrap,pnp4nagios)
• Basic Operating System checks via SNMP (Disk,CPU,Memory and Network usage)
• Hardware Health via SNMP and SNMP traps
Primo Application Monitoring
• Became more and more important as the system grows
• Java Server Health (Heap Memory, Garbage Collection, Sessions, Threads) by Jmx4Perl
• Functional Frontend Checks (Deep Search)
• Backoffice Process State using SQL queries (Indexing or Pipe Stopped on error)
• Authentication PDS / Shibboleth (webisoget)
• Remote Monitoring (Opac Via Primo, RTA, Primo Central ?, bX)
• Frontend Performance Monitoring (JavaMelody)
17
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Monitoring Primo - Icinga
18
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Monitoring Primo - Icinga
19
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Sysadmin Tools – Web Statistics
Web Statistics
• Analyze Frontend access_log using Open Source tool AWStats
• Problem 1: Single access_log for all Views / Institutions
• Problem 2: MFE (multiple frontend)
• Solution:
–
–
–
–
20
Modify JBOSS server.xml to use apache combined log format
Merge daily access_log from all FE servers (logresolvemerge.pl)
Split merged file by Views (detect sessions based on IP and referrer)
Update awstats for each View
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Sysadmin Tools – Web Statistics
21
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Sysadmin Tools – Primo Frontend Sync
Problem
• Single fe_web directory for all Views / Institutions
• Multiple Frontend Servers (MFE topology)
• Automatic Sync from Staging System not possible (different SP versions)
Solution & Workflow
• Organize everything into subdirectores named by view code
• Modify (update,create,delete) using FTP on working copy of production fe_web
• Send Email (Subject: SYNC static_htmls css)
• Run Sync Tool every 5 Minutes
–
–
–
–
–
22
Check Email (Ticket System) and identify users institution by email
Identify changes in subdirectories given in Subject using git (modified,new,deleted)
Commit changes and push to git server
Pull git updates from server on each frontend
Send report using Ticket System and close ticket.
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Sysadmin Tools – Primo Frontend Sync
23
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
Final Notes
2009
24
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin
2013
Thank you – Questions ?
Thank you
ulrich.leodolter@obvsg.at
25
Primo in Austria – 4 years of operation
Ulrich Leodolter, IGeLU 2013 Berlin