Criteria for evaluating portal framework software
(based on San Diego State University's Rubric for Rating Commercial Portal Vendors and Paul Browning's adaption of same)
S1 = score for Oracle Portal
S2 = score for u-portal
S3 = score for Luminis
S4 = score for SAP EP
1.1
Look and feel (of
front-end)
Aesthetics
1.2
Look and feel (of
front-end)
Ease of use
2.1
Security
Insufficient
(0 points)
Adequate
(1 point)
Excellent
(2 points)
Static background with
few or no graphic
elements. No ability
for variation in layout
or typography.
There are a few graphic
elements and there is
limited ability for variation
in type size, colour, and
layout.
2
OP: uses wizard for PL/SQL, not
for Java
uP: no wizard; XSLT or CSS
Counter-intuitive
interface, requiring
greater than two hours
of user training.
Somewhat intuitive
interface, requiring two
hours or less of user
training.
UOL has full control of look
2
and feel and changes can be
made quickly. Appealing
graphic elements are included
appropriately. Differences in
type size and/or university
colours and logos are used
well.
Intuitive interface, requiring
1
little or no user training.
2
OP: needs training, navigation
difficult
uP: layout easy to understand
No authentication.
Requires multiple log-ins
in order to access
different databases.
1.5
Both retain and pass on
credentials
No authentication or
authentication only
against a local
database.
One authentication
method that can use an
existing database.
Single sign-on for multiple
1.5
functions from one central
database (assuming
appropriate infrastructure
exists).
Multiple authentication
2
methods that can use existing
databases (for example, can
authenticate using an existing
LDAP server or using Active
Directory via Kerberos).
Authentication
2.2
Security
Authentication
method
S1
S2
2
S3
S4
Comments
2.3
Security
No access via secure
connection.
Secure connection can be Secure connection can be
specified but only in non- specified using standard
standard way.
method.
2
2
Information access is
all or nothing.
User is allowed to access
certain information based
on their user type.
2
There is a limit to the
number of roles a user
can have in the system.
User is allowed to access and 2
change certain information
based on who they are and
their user type.
There is no limit to the
2
number of roles a user can
have in the system.
Information access is
all or nothing.
Server(s) under
control of vendor at
location undetermined
by UOL.
User receives targeted
information relevant to
their constituency but
no dynamic data.
1 points
Server(s) located locally.
Server(s) located locally or at
a location approved by UOL.
2
2
User receives information
relevant to the individual,
but limited dynamically
updated data available.
User receives specific
information relevant to the
individual and available in
real-time. For instance,
student-specific courseschedule, enrolment details,
and degree checklist.
2
2
No editing tool to
customise the portal
environment.
Editing tool for
customising tabs, panel
buttons colours, and
fonts.
Personalised view of all
the information relevant to
user-specific needs and
preferences.
2
2
No link.
Partial access to webenabled course
information/classes.
Editing tool for full
customisation as well as the
ability to subscribe to new
channels (discussion boards,
chat etc). User has ability to
add/edit/remove information
from a list of internal and
external resources that the
university approves.
Full interoperability with VLE.
Encryption
2.4
Security
Access
2.5
Security
Roles
2.6
Security
Hosting
3.1
Personalisation
Information push
3.2
Personalisation
Information pull
(portal editor)
3.3
Personalisation
Link to exisiting
2
Both use SSL
VLE
4.1
Interaction
No email.
Portal only
accommodates
proprietary email system.
Supports multiple email
standards and protocols.
No chat or message
board functionality.
Only chat or only
message board.
Supports real time chat and
message boards.
No electronic balloting
and polling
functionality.
Criteria for balloting and
polling are only partially
supported.
Electronic balloting and
polling are fully supported.
No streaming audio or
video.
Options for streaming
audion and video are
limited.
Support for plug- ins that
allow for streaming audio
and/or video.
No search engine.
Limited search engine for
UOL intranet and/or
internet only.
No calendar.
Shared calendar(s)
available.
Does not support a
campus-wide meeting
scheduler
Limited campus-wide
meeting scheduler
available.
No to-do-list.
To-do-list available but
Natural language search
engine for both intranet and
internet (eg internal Ask
Jeeves).
Personalised calendar is
available.
Allows others (with user
approval) to populate their
calendar.
Synchronisation with Palm
OS is available.
Campus-wide meeting
scheduler for specific users
with ability to select and
reserve specific rooms and
equipment.
To-do-list available, but with
Email
4.2
Interaction
Chat & message
boards
4.3
Interaction
Electronic
balloting and
polling
4.4
Interaction
Multimedia
5.1
Productivity tools
Search engine
5.2
Productivity tools
Calendar
5.3
Productivity tools
Meeting
scheduler
5.4
Productivity tools
with limited features.
To-do-list
5.5
Productivity tools
No address book.
Address book is limited
and proprietary.
Address book
6.1
E-commerce
Advertising
control
6.2
E-commerce
Banner advertising on Banner advertising is
every page of portal or optional.
not able to control per
user group.
No advertising
revenue possible.
Advertising revenue is
limited and shared with
the vendor.
Cannot be integrated
with campus systems
offering web-based
transactions.
Can be integrated with
campus systems offering
web-based transactions.
No forms routing.
Forms routing available paper documents can be
replaced by web-based
forms.
No fit with existing
relational databases,
for example Oracle
RDBMS
Vendor relies solely on
UOL staff.
Some fit with existing
relational databases, for
example Oracle RDBMS
Advertising
revenue
6.3
E-commerce
Transactions
7.1
Workflow
Forms routing
8.1
Support
Integration
8.2
Support
many features, eg items can
be placed in categories and
ranked in priority order.
Address book can interface
with other contact lists and
databases.
Banner advertising is
optional, controllable and can
be targeted to specific user
groups based on their role
within the university.
Advertising revenue goes
directly to the university
Can be integrated with
campus systems offering
web-based transactions with
option of one "shopping cart"
enabling the user to credit the
appropriate entity.
Forms routing available paper documents can be
replaced by web-based
forms. In addition, forms
tracking software built in.
Great fit with existing
2
relational databases, for
example Oracle RDBMS
Vendor provides
Vendor provides
implementation training to implementation training to
UOL staff and consultants UOL staff and on-site
2
2
Both can use ODBC and JDBC
1
OP: training more readily
available
Implmentation
8.3
Support
Scalability
8.4
Support
Not currently in use at
a large-scale
reference site.
Maintenance
8.7
Support
Vendor support
8.8
Support
Dependencies
8.9
Support
Long-term
viability - vendor
experience
8.10
Support
Long-term
viability -
2?
Reference site may not be
Solaris
1
Need to contact reference sites
Some evidence from
reference sites that
service is reliable.
No vendor-provided
load-balancing.
Vendor provides some
assistance with enabling
load-balancing.
Vendor provides loadbalancing as part of solution
(or via alliance with
commercial partner).
2
1
No plan for ongoing
support or
maintenance.
Weak plan for ongoing
support or maintenance.
Strong plan for ongoing
support or maintenance.
2
1
Vendor requires toll
call during business
hours. No email or
Web-based help.
Requires installation of
several packages
(other than product)
that are not already in
use locally.
Vendor is in pilot
phase and has no
experience or
references.
Vendor provides email
and Web-based help. No
phone or fax help (24/7).
Vendor provides email, Web- 1.5
based and toll-free help (24/7)
for free.
1
Requires installation of
one or two packages
(other than product) that
is not already in use
locally.
Vendor has experience in
higher education portal
development but has
limited references.
Does not require any software 1
that is not already in use
locally.
2
Vendor has significant higher 1
education portal development
experience and can provide
numerous references.
1.5
Vendor company is
small with limited
funding to develop
product.
Vendor company is
investing some funding in
product.
Vendor company has ample
financial backing that is
committed to developing the
product.
1
Load balancing
8.6
Support
consultants for free.
Existing large-scale reference 2?
site demonstrating that
application server scales on
Solaris.
Clear evidence from
1
reference sites that
production service is reliable.
No reference site.
Reliability
8.5
Support
for a fee.
Currently being
implemented at a largescale reference site.
1
OP: needs mod_jserv
commitment to
product
9.1
Standards
API (Application
Programmer
Interface)
9.2
Standards
LDAP
(Lightwieght
Directory Access
Protocol)
9.3
Standards
Portal API cannot
pass information to
other applications, or
is not available to
campus.
Portal API can pass some
information to other
applications and is
available on a limited
basis.
Portal API can pass
2
information to other
applications, seamlessly
integrating multiple sources of
information and campus can
write their own interface, for
example passing user
preferences between
applications
Portal is not LDAP
Portal is LDAP compliant, Portal is LDAP compliant and
compliant (it will not
but only allows limited
allows user to actively
allow a user to query a online querying.
manage and customise a
database via the
personal database.
internet).
2
No tools for outputting
in different formats
Vendor provides tools for
a limited number of
standard formats
Vendor provides tools giving
full flexibility of output format
1
2
No tools for ensuring
that site meets WAI
accessibility
guidelines.
Some help provided to
ensure that site meets
WAI accessibility
guidelines.
Good set of tools provided to
ensure that site meets WAI
accessibility guidelines.
0
0
Seven or more fulltime UOL staff
required for managing
and maintaining
system software.
Vendor relies solely on
UOL staff.
Between four and six full
time UOL staff required
for managing and
maintaining system
software.
Vendor provides training
to UOL staff and
consultants for a fee.
Three or fewer full time UOL
staff required for managing
and maintaining system
software.
2
2
Vendor provides training to
UOL staff and data
migration/integration
1
1
Accessibility output in different
formats
9.4
Standards
Accessibility administrative
tools
10.1
Administration
UOL staffing
10.2
Administration
Vendor staffing
Both: 1 person to maintain
10.3
Administration
Fit with existing
staff skill set management of
software
10.4
Administration
Fit with existing
skill set development
10.5
Administration
User definition
10.6
Administration
Information
channels
10.7
Administration
Time to market
10.8
Administration
Hardware
resources
10.9
Administration
No existing staff
currently have skills
required for
management of
system software.
consultants for free.
Only one or two members Three or more members of
of staff currently have
staff currently have skills
skills required for
required for management of
management of system
system software.
software.
1
1
OP: Oracle, Portal interface,
Apache
uP: Oracle, Apache, Tomcat
OP: PL/SQL, Java
uP: Java, XML/XSLT
No existing staff
currently have skills
required for
developing
applications using the
software.
Only one or two members
of staff currently have
skills required for
developing applications
using the software.
Three or more members of
staff currently have skills
required for developing
applications using the
software.
2
1
System will NOT allow
UOL administrator to
define custom user
types.
System will NOT allow
UOL administrator to
define custom
information channels.
System will allow UOL
administrator to define
limited user types.
System will allow UOL
administrator to define
custom user types.
2
2
System will allow UOL
administrator to define
limited information
channels.
2
2
System will take
greater than:
- 8 weeks to define
- 12 weeks to design
- 8 weeks to prototype
- 16 weeks to rollout
Hardware
requirements do not
coincide with
university standards.
System will take:
- 8 weeks to define
- 12 weeks to design
- 8 weeks to prototype
- 16 weeks to rollout
System will allow UOL
administrator to define
custom information channels.
No limit to number of
information channels.
System will take less than:
- 8 weeks to define
- 12 weeks to design
- 8 weeks to prototype
- 16 weeks to rollout
Hardware requirements
loosely coincide with
university standards.
Hardware requirements
coincide with university
standards.
2
2
Annual licence fee and
Service Level Agreement
2
2
Annual licence fee and Annual licence fee and
Service Level
Service Level Agreement
Pricing - software
10.10
Administration
Pricing production
service hardware
10.11
Administration
Pricing development
hardware
10.12
Administration
Online help,
documentation
and training
11.1
General
Essential
channels
11.2
General
Exit strategy
Agreement costs not
based on fixed price
schedule or exceeds
budget.
Large increase in
hardware required for
production service
costs based on fixed price costs based on fixed price
schedule.
schedule and are under
budget.
Modest increase in
hardware required for
production service
Will be able to run production
service on current hardware
0?
1?
Large increase in
hardware required to
develop service
Modest increase in
hardware required to
develop service
Will be able to develop
service on current hardware
1?
2?
No help putting portal
applications into
production.
Plan for integration, but
Clear integration, with plenty
little useful documentation of supporting documentation
and training.
and face-to-face train-thetrainer training.
Several online help features,
for example tutorials, job aids
and FAQ?s.
1.5
1
OP: lots of documentation!
No indication of offthe-shelf channels for
HE.
Some off-the-shelf
Many off-the-shelf channels
channels for HE available for HE available or being
or being developed.
developed.
0
1
OP: can use in-line frames
uP: Aleph and Blackboard
Channels are nonstandard and would
require a lot of work to
transfer to another
framework.
Channels are written
using standards but
would require some work
to transfer them to
another framework.
0.5
1
OP: less work needed if only
Java used (thought to be
unlikely)
Channels are written using
standards and can be very
easily transferred to another
framework using those
standards.
Total scores
Weighting
1.1 Look and feel (of front-end)
Aesthetics
1.2 Look and feel (of front-end)
Ease of use
2.1 Security
Authentication
2.2 Security
Authentication method
2.3 Security
Encryption
2.4 Security
Access
2.5 Security
Roles
2.6 Security
Hosting
3.1 Personalisation
Information push
3.2 Personalisation
Information pull (portal editor)
3.3 Personalisation
Link to exisiting VLE
4.1 Interaction
Email
4.2 Interaction
Chat & message boards
4.3 Interaction
Electronic balloting and polling
4.4 Interaction
Multimedia
5.1 Productivity tools
Search engine
5.2 Productivity tools
Calendar
WS1
WS2
WS3
WS4
5.3 Productivity tools
Meeting scheduler
5.4 Productivity tools
To-do-list
5.5 Productivity tools
Address book
6.1 E-commerce
Advertising control
6.2 E-commerce
Advertising revenue
6.3 E-commerce
Transactions
7.1 Workflow
Forms routing
8.1 Support
Integration
8.2 Support
Implementation
8.3 Support
Scalability
8.4 Support
Reliability
8.5 Support
Load balancing
8.6 Support
Maintenance
8.7 Support
Vendor support
8.8 Support
Dependencies
8.9 Support
Long-term viability - vendor experience
8.10 Support
Long-term viability - commitment to product
9.1 Standards
API (Application Programmer Interface)
9.2 Standards
LDAP (Lightwieght Directory Access Protocol)
9.3 Standards
Accessibility - output in different formats
9.4 Standards
Accessibility - administrative tools
10.1 Administration
UOL staffing
10.2 Administration
Vendor staffing
10.3 Administration
Fit with existing staff skill set - management of software
10.4 Administration
Fit with existing skill set - development
10.5 Administration
User definition
10.6 Administration
Information channels
10.7 Administration
Time to market
10.8 Administration
Hardware resources
10.9 Administration
Pricing - software
10.10 Administration
Pricing - production service hardware
10.11 Administration
Pricing - development hardware
10.12 Administration
Online help, documentation and training
11.1 General
Essential channels
11.2 General
Exit strategy