Design Specification
Visual Analytics Visualization Plug-In Framework
Authored by Arun Katkere
History
Updates
Initial version
Incorporated feedback from May 10
meeting
Incorporated results of client arch
discussion
Date
May 6, 2013
May 16, 2013
Author
Arun Katkere
Arun Katkere
May 28, 2013
Arun Katkere
Reviewers
Contents
References .......................................................................................................................................3
Problem Statement........................................................................................................................... 3
Design Goals................................................................................................................................ 4
Requirements................................................................................................................................4
Approvals..................................................................................................................................... 5
Planned Solution.............................................................................................................................. 5
Concepts....................................................................................................................................... 5
Proposed User Interface............................................................................................................... 6
Affected Software Modules......................................................................................................... 6
Expected Performance Impact..................................................................................................... 7
Required Testing...........................................................................................................................7
High Level Design of Planned Approach........................................................................................ 7
Built-in Extension Points............................................................................................................. 7
Visualization Extension Point...................................................................................................7
Toolbar extension point............................................................................................................ 7
Layout manager extension point...............................................................................................8
Context menu extension point.................................................................................................. 8
Built-in Extensions.......................................................................................................................8
Plug-In Location...........................................................................................................................8
Unpacked Plug-In Resources....................................................................................................... 8
Plug-In Management UI...............................................................................................................9
Plug-In Versioning........................................................................................................................9
Plug-In File Layout...................................................................................................................... 9
Plug-In Configuration.................................................................................................................. 9
Example Plug-In Configuration..............................................................................................10
Plugin.json.............................................................................................................................. 11
Plugin Configuration processing............................................................................................ 11
Plug-In Data Model....................................................................................................................12
Plug-In REST API...................................................................................................................... 12
Components................................................................................................................................13
Packaging And Deployment.......................................................................................................14
Source Control............................................................................................................................14
Versioning and Upgrade............................................................................................................. 15
Plugin Schema............................................................................................................................15
References
List all relevant documents and their location, i.e. User Model, Use cases and other design
documents.
Problem Statement
Describe problem or requirement and refer to what use cases are addressed.
Currently Oracle VA team implements all the visualization types supported in an BI analysis/
report. It would be beneficial if other teams within Oracle and end users can add their own
visualization types. Specific custom visualization use cases include:
•
Used internally by BITech to implement all OOTB visualization types as “plugins”
•
Used internally by BITech to re-use BIPS implementation of a visualization type not
implemented natively in VA
•
Used internally by BITech to ship visualization types that are not generic enough to ship
as OOTB viz types
•
Used internally by BIApps to implement a visualization type for a particular vertical
•
Used internally by BI field to implement a visualization type for a particular customer (?)
•
Used by tenant administrator to upload new visualization types specific to a tenant
•
Used by site administrator to upload new visualization types for entire site
Deploying any tenant developed code into a multi-tenant environment involves security issues.
To avoid these we will only support client-side visualization implemented in JavaScript. Most of
the out of the box visualizations are based on ADF, ADF JET, or BIPS, and will require some
amount of server-side code. These are also implemented as “plugins”. However, special handling
on the server-side is required for these components.
For initial release, it is very likely that the plugin functionality is not exposed to third party
developers. However, it is very difficult to retrofit a plugin framework at a later stage, and hence
all visualizations that are shipped out of the box are implemented on top of a plugin framework
in the initial version itself.
Design Goals
•
Out of the box visualizations are implemented exactly the same way as custom plugins as
much as possible. This is done to force us to think through framework/plugin contracts,
and to make sure we exercise the custom plug-in functionality.
•
Leverage existing BI infrastructure. For instance, webcat and ACLs.
•
Pay close attention to amount of data on the client while designing APIs for plugins.
Requirements
Requirement
Define a viz plugin packaging format
Provide a mechanism for a per-tenant
administrator or system administrator to
upload a viz plugin package
Provide REST API to lists all viz
plugins available for a specific user
Provide REST API to list latest version
compatible version of viz plugin given
minimum version. Viz plugin version
follows major.minor.patch version
scheme where backwards compatibility
can only be broken with major version
change, features can be added with
minor version change, and only patch
version change is for bug fixes only
Versions of plugins for a particular user
session should be stable even if a plugin
is updated to a newer version number
during user session
Client-side: On initial report load, all
JavaScript files referenced by all PlugIns used in the report should be loaded
Client-side: Provide a dynamic
mechanism to load JavaScript files
referenced by PlugIns not currently
loaded into the browser (say, on a
Source
Priority
High
Med
High
High
High
High
High
Comments
visualization switch)
Client-side: Provide a set of APIs for
Plug-In author to get report metadata,
event Pub/Sub, report editing
Provide ability to add visualization
types
Provide ability to add new toolbar items,
new context menu items, etc.
Provide ACL mechanism for plugins
and custom viz
Upgrade support: Handle scenario
where a report is edited to add new viz
from a plugin when it already contains
viz from the same plugin. Options:
support a single report using multiple
versions of same plugin or provide
upgrade hooks (see below)
Update support: Provide hooks for
extensions to upgrade data when Plug-In
version changes
User should be able to upload Plugin
and use it in reports without restarting
server
Provide mechanism to ship out of the
box viz. Options: ship in an ear file and
have PlugIn manager look at this ear
and webcat OR provide a mechanism to
ship webcat content with OOTB viz
plugins
High
High
Med
Low
High
Med
High
High
Approvals
Approver
Role
Date
Comments
Planned Solution
Give a high level overview of solution.
Planned solution involves a generic framework for plugin packaging, definition of extension
points and extensions that implement extension points. For “built-in” extension points, details of
configuration of extensions and runtime contract between extension point and extensions will be
provided.
Concepts
•
Plug-In: A package of “resources” (JavaScript files, HTML files, CSS files, and images).
Plug-In has an unique “id”, a “name”, and a “version”. Plug-In can depend on other PlugIns. A Plug-In can expose a set of “extension point”s and implement a set of
“extension”s.
•
Plug-In Configuration (plugin.xml or plugin.json): A well-known file inside the PlugIn package (/META-INF/plugin.xml or /META-INF/plugin.json) is used to specify PlugIn configuration. Plug-In configuration contains meta data about the Plug-In as well as
configuration of its resources, extension points, and extensions (see below).
•
Plug-In Resource: A file (JavaScript, HTML, image, CSS, …) that is accessible from the
client. Resources can depend on other resources or plug-ins.
•
Extension Point: A “socket” that can host extensions. Extension point is identified by an
id. Extension Points are versioned (??). Extension Points can be built-in or exposed by
Plug-Ins. Extension point specifies the schema used by extensions to configure the
extension.
•
Plug-In Schema: Plug-Ins that define extension points also define schemas that are used
by extension points for configuration. Schemas can be XSD file resources or reference to
JAXB generated package that is already built into the server (i.e., for out of the box
extension points).
•
Extension: A “plug” that plugs into extension point “socket”. Extensions implement a
specific extension point referenced by id. (Specify version??). Extensions can be built-in
or defined by Plug-Ins.
•
Extension Configuration: Extension configuration is specified in plugin.xml. Schema
for extension configuration is defined by the extension point that a particular extension
implements.
•
Extension Point/Extension Runtime Contract: Runtime contract between extension
point and extension is defined by the extension point. This can include APIs that
extension point exposes to extension (passed to extension in a well defined way), APIs
that an extension point calls on the extension, and eventing.
Proposed User Interface
If applicable – insert any relevant user interface sketches.
Affected Software Modules
List all source code projects that will be modified and expected complexity of changes or new
code (low, medium, high). Also describe risks for impact on existing functionality.
Expected Performance Impact
Describe the performance characteristics that is expected, i.e. effect on query time, expected UI
refresh time with small, medium and large datasets (1k, 100k, 100M), etc.
Required Testing
What automated tests needs to be developed and what must be tested manually
High Level Design of Planned Approach
Give a fairly detailed description of the design, list major risks and estimated cost (in days).
Built-in Extension Points
Visualization Extension Point
• oracle.bi.tech.plugin.visualization
•
Extension configuration defines:
◦ “Host” type: “JavaScript”, “ADF”, “JET”, “BIPS”
◦ Visualization Type and SubType
◦ Factory method to create Visualization
◦ Methods on Visualization for initialization, rendering, initialization from suggestion
data etc.
◦ viz extension point can contribute to the auto suggest engine configuration
◦ viz extension point can contribute panels to property editor
◦ Plug-In viz runtime
▪ Shape of data (0 or more cubes)
▪ Configuration blob
•
Managed by the viz plugin
•
Format only known to the plugin
▪ Events (0 or more)
▪ Metadata query
▪ marking type (filter | highlight)
Toolbar extension point
• oracle.bi.tech.plugin.toolbar
Layout manager extension point
• oracle.bi.tech.plugin.layout.manager
Context menu extension point
• oracle.bi.tech.plugin.context.menuitem
Built-in Extensions
•
Visualization Extensions
◦ Pivot, KPI’s, etc
◦ ADF DVT components
◦ BIPS views (trellis, etc)
◦ Oracle JET visualizations
◦ External visualizations (D3, etc)
Plug-In Location
Plug-Ins are stored in BI webcatalog using the following layout:
•
/shared/obitech/plugins/oracle: Factory plugins
•
/shared/obitech/plugins/custom: Custom Plugins
•
/tenants/<tenantID>/<serviceName>/obitech/plugins: Tenant specific plugins
All these locations are readable and writeable by BI system user out of the box.
Unpacked Plug-In Resources
Unpacked Plug-In resources will be stored in webcat as well in the same location as the original
plugin in a folder with the same name as plugin file name without extension. Advantage of this
approach is that standard webcat ACL schemes can be applied to unpacked resources. It makes
development of plugins easier since a developer does not have to bump up version, package
Plug-In plugin, and upload to webcat to test each change.
One potential issue with this approach is additional latency in loading a report since potentially
multiple JavaScript files and other resources will have to loaded from webcat, and this involves 2
hops (browser to WLS and WLS to sawserver), potentially 3 if JBIPS is involved. During initial
development this approach will be tested for performance and latency, and if the results are not
satisfactory, alternate approach is to unpack to a local directory on each node, and server content
via virtual directory mapping.
Plug-Ins will be unpacked to “hidden” folders under:
•
/shared/obitech/plugins-unpacked
•
/tenants/<tenantID>/<serviceName>/obitech/plugins-unpacked
ACLs on these folders will allow read permission for all authenticated users in case of shared
plugins, and all tenant specific users in case of tenant specific plugins.
Plug-Ins will be exposed to the browser via a Jersey Resource accesed as follows:
•
/resource/{plugin-id}/{version}/{path/to/resource.ext}
Logged in user's session id will be used to access these resources from webcat.
Plug-In Management UI
A UI (and corresponding CLI) is provided to:
•
Upload new version of a plugin
•
Delete existing plugin (all versions or selected version)
Catalog management tools can be used to upload plugins as well.
Plug-In Versioning
All Plug-Ins must use three part version number <major>.<minor>.<patch>. Multiple versions of
the same Plug-In can exist in the catalog.
Plug-In File Layout
•
File name format <plugin_id>_<pluginversion>.jar
•
Contents:
◦ /META-INF/plugin.xml or /META-INF/plugin.json: Plug-In configuration
◦ JavaScript, image, and HTML resources in nested directory structure
Plug-In Configuration
Plug-In Configuration contains the following information:
•
Plug-In meta data:
◦ Plug-In id (unique id, using same rules as Java package name)
◦ Plug-In name (name shown in UI)
◦ Plug-In version
•
Required Plug-Ins:
◦ List of Plug-Ins that are required to be present in the system and loaded before this
plugin. Minimum required version number is specified for each required plug-in.
Note that this Plug-In does not need to require Plug-Ins that provide extensions points
implemented by this Plug-In. Extension points and extensions are loosely bound, and
do not require a direct dependency.
•
Provided Resources:
◦ List of all script files in the Plug-In.
▪ Optionally these files can depend on another resource in either this Plug-In or
another Plug-In. For instance a JavaScript file that requires JQuery can depend on
jquery.js resource. Plug-In container will make sure dependent resources are
loaded before this resource is loaded.
▪ Scripts by default are not loaded until they are required. Optionally they can be
specified to always load when plug-in container is loaded.
◦ List of all images in the Plug-In
◦ CSS files? Other resources?
•
Provided extension points
•
Implemented extensions and configuration for each extension.
Example Plug-In Configuration
<plugin id=”org.d3js.d3” name=”D3.js” version=”3.1.6”>
<resources>
<script id=”min” type=”JavaScript” file=”d3.v3min.js”/>
<script id=”full” type=”JavaScript” file=”d3.v3.js”/>
</resources>
</plugin>
<plugin id=”com.example.oracle.bi.tech.customvizplugin” name=”Custom
Visualization Plug-In by Example.com” version=”1.2.3”>
<resources>
<resource id=”customviz” type=”JavaScript” file=”customviz.js”
load=”always”>
<requiredresource id=”org.d3js.d3.min” version=”3.0.0”/>
</resource>
</resources>
<extensions>
<extension point=”oracle.bi.tech.plugin.visualization”>
<viz:configuration type=”chart” subtype=”customchart”>
<viz:host type=”javascript”/>
<viz:script id=”customviz”/>
<viz:factory-method name=”createViz”/>
<viz:init-method name=”init”/>
<viz:render-method name=”render”/>
</viz:configuration>
</extension>
</extensions>
</plugin>
Plugin.json
Plugin.xml is the preferred format for specifying a plugin since it is backed by schemas (one for
base plugin schema and one for each for extension point). Plugin.json format is converted to
XML before processing using Jackson library. TODO: document JSON schema.
Plugin Configuration processing
Internally base PlugIn configuration is mapped to JAXB generated classes. Each of the out of the
box extension point schemas are also mapped to JAXB generated classes. While we do not
anticipate user provided extension points at this stage, if there are any, their configuration is
stored as a blob.
Plug-In Data Model
Plug-In REST API
URI: /api/v1/plugins
Method: GET
Matrix Arguments:
•
id: optional: by default all plugins will be returned. If id is specified, plugins
matching id will be returned. It is legal to specify multiple ids. Each id is a plugin
id optionally followed by ':' and version. If version is specified, latest compatible
version of the plugin is returned. For example, if id = “
com.example.oracle.bi.visualanalyzer.customvizplugin:1.2.1” and available
versions of com.example.oracle.bi.visualanalyzer.customvizplugin are 1.1.5,
1.2.0, 1.2.8, and 2.0.1, only
com.example.oracle.bi.visualanalyzer.customvizplugin:1.2.8 is returned. If same
plugin is refenced multiple times with different versions, then plugin version
compatible with both is returned. Error is thrown if no compatible version is
found.
Response: JSONArray of
•
JSON Object (PlugIn):
◦ “id”: plugin id
◦ “version”
◦ “self”: URI
◦ “resources”: JSON Array of
▪ JSON Object (PlugInResource)
◦ “extensionPoints”: JSON Array of
▪ JSON Object (PlugInExtensionPoint)
◦ “extensions” JSON Array of
▪ JSON Object (PlugInExtension)
URI: /api/v1/plugins/{plugin_id}
Method: GET
Response: JSON Object (PlugIn)
Components
•
Server Components:
◦ Plug-In Registry
▪ In memory registry of all plugins, extension points, extensions and resources. It
exposes API to list/query/access specific plugins.
▪ API:
•
List<PlugIn> getAllPlugins()
•
PlugIn getMatchingPlugIn(String version)
•
PlugIn getPlugInWithExtensionPoint(String extensionPoint, String version)
•
List<PlugIn> getPlugInsImplementingExtensionPoint(String extensionPoint,
String version)
◦ Plug-In Manager
▪ Provides an API to upload plugins to webcat. Plug-In is unpacked into webcat and
a marker is written to indicate to Plug-In scanner that plugin is fully unpacked and
ready.
▪ Provides an API to delete plugins from webcat. Both packaged version as well as
unpacked version are deleted from webcat.
◦ Plug-In Scanner
▪ Scans webcat (unpacked locations for plugins) for new plugins and deletion of
existing plugins. Notifies Plug-In registry on new plugin events. Plug-In scanner
uses folder last modified time to determine changes to plugin folders.
◦ Plug-In REST API
▪ Jersey-based REST API to expose Plug-In registry API to remote clients.
◦ Report Defn Processor
▪ This component returns a list of plugin and extension references from a particular
report definition. This is run either on report save (if we introduce a preamble
section to report.xml that lists all plugin and extension references in the
document) or during report open operation.
•
Client Components:
◦ Plug-In Manager
◦ EventHub, Report
Packaging And Deployment
Plug-In management infrastructure is packaged as a web application. Whether it is independently
deployed or combined with va-ui.ear file is TBD. Plug-In Registry and Plug-In Scanner are
initialized via a context listener registered via web.xml. Plug-In REST API is implemented using
Jersey, and Jersey servlet is registered via web.xml and mapped to “/api” URI under context
path. In weblogic deployment, a system policy is defined to allow the web application to access
credentials to access Plug-In store in the webcat. For development, these credentials are provided
in web.xml and/or system properties, allowing deployment of the web application independently
into a Jetty or Tomcat container.
Source Control
Plug-In management Java source will be checked into gradle jar project “bi-viz-pluginframework” under multi-project GIT repo “bi-tech-components”. Jar file generated by this
project can be consumed in 2 different ways (one of which will be chosen):
•
Gradle ear project “bi-viz-plugin-framework-slib” to package this in a WebLogic shared
library. va-ui.ear file will add a dependency to this shared library and include necessary
context listener and Jersey entries in web.xml.
•
Gradle war project “bi-viz-plugin-framework-webapp” to package this in a war file that
can be embedded in va-ui.ear file.
Versioning and Upgrade
Plug-Ins provide support for upgrade from any previous version to the latest version. When a
report is created, latest versions of all available plugins are used. Plug-Ins and versions used are
recorded in report.xml.
If the report is edited at a later stage, and some of the plugins used have newer versions,
updateConfiguration is called on each of the Plug-Ins to update the configuration to latest
version. When report is saved, updated versions of these Plug-Ins is stored in report.xml. If
update can be no longer supported since implementation and/or configuration of a particular
plugin has changed drastically, then it is recommended that the a brand new plugin id be used.
Previous plugin can be marked as “hidden” and it will not be available for use in new reports.
When a report is viewed at a later stage, and some of the plugins used have newer versions,
updateConfiguration is called on each of the Plug-Ins to update the configuration to latest
version. This is done in memory on the client and persisted report XML is not touched.
Updating Report XML format
Updating report XML when a plugin configuration is changed is the responsibility of the
extension point. Framework will provide a generic API based on jQuery to parse and manipulate
report XML on the client.
Plugin Schema
<?xml version="1.0" encoding="UTF-8"?>
<schema xmlns="http://www.w3.org/2001/XMLSchema" targetNamespace="http://plugin.frameworks.tech.bi.oracle"
xmlns:tns="http://plugin.frameworks.tech.bi.oracle" elementFormDefault="qualified">
<complexType name="BIPlugInType">
<sequence>
<element name="resources" maxOccurs="1" minOccurs="0">
<complexType>
<sequence>
<element name="resource" type="tns:BIPlugInResource"
minOccurs="0" maxOccurs="unbounded" />
</sequence>
</complexType>
</element>
<element name="schemas" maxOccurs="1" minOccurs="0">
<complexType>
<sequence>
<element name="schema" type="tns:BIPlugInSchema"
minOccurs="0" maxOccurs="unbounded" />
</sequence>
</complexType>
</element>
<element name="extensions" maxOccurs="1" minOccurs="0">
<complexType>
<sequence>
<element name="resource" type="tns:BIPlugInExtension"
minOccurs="0" maxOccurs="unbounded" />
</sequence>
</complexType>
</element>
<element name="extension-points" maxOccurs="1" minOccurs="0">
<complexType>
<sequence>
<element name="extension-point"
type="tns:BIPlugInExtensionPoint"
minOccurs="0" maxOccurs="unbounded" />
</sequence>
</complexType>
</element>
</sequence>
<attribute name="id" type="string" />
<attribute name="name" type="string" />
<attribute name="version" type="string" />
<attribute name="hidden" type="boolean" default="true"/>
</complexType>
<complexType name="BIPlugInResource">
<sequence>
<element name="required-resource">
<complexType>
<attribute name="id" type="string"/>
<attribute name="version" type="string"/>
</complexType>
</element>
</sequence>
<attribute name="id" type="string" />
<attribute name="path" type="string" />
<attribute name="mime-type" type="string" use="optional"/>
<attribute name="load" default="whenneeded">
<simpleType>
<restriction base="string">
<enumeration value="always"/>
<enumeration value="whenneeded"/>
</restriction>
</simpleType>
</attribute>
<attribute name="type" use="optional">
<simpleType>
<restriction base="string">
<enumeration value="script"/>
<enumeration value="schema"/>
<enumeration value="binary"/>
</restriction>
</simpleType>
</attribute>
</complexType>
<complexType name="BIPlugInSchema">
<sequence>
<choice>
<element name="resource-id" type="string"/>
<element name="jaxb-package" type="string"/>
</choice>
<element name="target-namespace" type="string"/>
</sequence>
</complexType>
<complexType name="BIPlugInExtensionPoint">
<sequence>
<element name="configuration-element" minOccurs="1">
<complexType>
<sequence>
<element name="namespace" type="string"/>
<element name="local-name" type="string"/>
</sequence>
</complexType>
</element>
<element name="upgrade-script">
<complexType>
<sequence>
<element name="script-id" type="string"/>
<element name="upgrade-method" type="string"/>
</sequence>
</complexType>
</element>
</sequence>
<attribute name="id" type="string"/>
<attribute name="version" type="string"/>
</complexType>
<complexType name="BIPlugInExtension">
<sequence>
<any minOccurs="0"></any>
</sequence>
<attribute name="point-id" type="string"/>
<attribute name="version" type="string"/>
</complexType>
<element name="obiplugin" type="tns:BIPlugInType"/>
</schema>