RESULTS/FTA
Electronic Submission Framework
Technical Design Document
Prepared for:
Ministry of Forests
Information Management Branch
Document Revision 1.5
Prepared by:
Ministry of Forests
Electronic Submission Framework
Technical Design
Document Change Control
REVISION NUMBER
DATE OF ISSUE
AUTHOR(S)
BRIEF DESCRIPTION OF CHANGE
1.0
1.1
Jan 30, 2003
Feb 19, 2003
Vivid Solutions Inc.
Vivid Solutions Inc.
1.2
Aug 05, 2003
Vivid Solutions Inc.
1.3
Sep 14, 2004
Vivid Solutions Inc.
1.4
Feb 10, 2005
Vivid Solutions Inc.
1.5
April 10, 2006
Vivid Solutions Inc.
Document Creation
Added a fourth role (Queue
Administrator) to the security section of
the document
Update to reflect final delivery of code.
Changes to code samples and update to
“Transaction” section.
Added section for WebService.
Added new Audit types and new
Notification type.
Added section for Agent Management.
Added section for ESF Map Visualization
Tools (TMS Integration).
Added Support role information.
Updated ESF Message class diagram.
Added section for Attachments
Updated Web Service specification for
attachments.
Updated ESF Messages specification.
Updated
ESF
Agent
to
support
attachments.
Updated Security section to specify use of
GUIDS.
Document Sign-Off
The undersigned have read and agree with the content of this document.
Project Manager
VIVID Solutions Inc.
Project Manager - MOF
Ministry of Forests, Information Management
Branch
Table of Contents
Page 2
Ministry of Forests
Client Company
1.
SCOPE ................................................................................................................. 6
1.1
2.
REFERENCES ................................................................................................... 6
OVERVIEW ............................................................................................................. 7
2.1
2.2
2.3
3.
ENSURED DELIVERY ............................................................................................ 7
HIDDEN DELIVERY METHOD .................................................................................... 7
AUTOMATED ACKNOWLEDGEMENT/TRACKING.................................................................. 8
STORAGE DESIGN ...................................................................................................... 9
3.1
3.2
3.3
APPLICATION DATABASE ....................................................................................... 9
MESSAGING QUEUES ........................................................................................... 9
FILE STORE .................................................................................................... 9
3.3.1File Store ................................................................................................... 9
3.3.2
4.
SCRATCH DIRECTORY ......................................................................................... 10
QUEUING ............................................................................................................. 11
4.1
5.
Electronic Submission Framework
Infrastructure Design
QUEUE MANAGEMENT......................................................................................... 11
MESSAGES ............................................................................................................ 14
5.1
5.2
5.3
5.4
TEXT MESSAGE ............................................................................................... 15
PING MESSAGE ............................................................................................... 15
STATUS MESSAGE ............................................................................................. 16
FILE MESSAGE ................................................................................................ 17
5.4.1Submission Message ...................................................................................... 17
5.4.2Attachment Message..................................................................................... 18
5.5
5.6
6.
OBJECT MESSAGE............................................................................................. 18
AUDIT MESSAGE .............................................................................................. 18
QUEUE AGENTS....................................................................................................... 20
6.1
6.2
PAYLOAD ..................................................................................................... 20
TRANSPORT IMPLEMENTATION ................................................................................ 20
6.2.1Queue Polling ............................................................................................. 20
6.2.2Exception Queues ........................................................................................ 21
6.3
AUTOMATED AUDITING ....................................................................................... 21
6.3.1Audit Events ............................................................................................... 21
6.4
6.5
6.6
AGENT INITIALIZATION ........................................................................................ 22
TRANSACTIONS ............................................................................................... 22
AGENT MANAGEMENT ......................................................................................... 24
Page 3
Ministry of Forests
Client Company
Electronic Submission Framework
Infrastructure Design
6.6.1JMX as a Management Framework .................................................................... 25
6.6.2Remote Management .................................................................................... 25
6.6.3Scheduling of Tasks ...................................................................................... 26
6.6.4Authentication and Authorization ..................................................................... 26
6.6.5Custom Business Agent Integration ................................................................... 27
7.
SUBMISSION BROKER .................................................................................................. 28
8.
STATUS AGENT ....................................................................................................... 30
9.
SIMPLE BUSINESS AGENT.............................................................................................. 31
10.
ESF AGENT........................................................................................................... 32
10.1
10.2
10.3
11.
SECURITY ............................................................................................................. 34
11.1
11.2
11.3
11.4
11.5
11.6
11.7
11.8
12.
RECEIVING A STATUS MESSAGE ............................................................................... 32
RECEIVING AN AUDIT MESSAGE ............................................................................... 33
RECEIVING A PING MESSAGE .................................................................................. 33
AUTHENTICATION ............................................................................................. 34
AUTHORIZATION .............................................................................................. 34
DEFINED ROLES ............................................................................................... 34
SUBMISSION QUERY RESTRICTIONS ............................................................................ 35
SUBMISSION DETAIL RESTRICTIONS ............................................................................ 35
ADMINISTRATION RESTRICTIONS............................................................................... 35
ENCRYPTION .................................................................................................. 35
TRACKING USER AND ORGANIZATION OWNERSHIP............................................................. 35
SUBMISSIONS WEB SERVICE ........................................................................................... 36
12.1
INTERFACE DESIGN ........................................................................................... 36
12.1.1addAttachment ......................................................................................... 37
12.1.2getSubmissionStatus ................................................................................... 38
12.1.3getSupportedAttachmentTypes ...................................................................... 38
12.1.4uploadSubmission ....................................................................................... 39
12.1.5getSupportedSubmissionTypes ....................................................................... 39
12.1.6isSupportedSubmissionType .......................................................................... 39
12.1.7makeSubmission ........................................................................................ 39
12.1.8searchSubmission ....................................................................................... 40
12.2
EXCEPTION HANDLING ........................................................................................ 41
Page 4
Ministry of Forests
Client Company
13.
ESF MAP VISUALIZATION TOOLS ...................................................................................... 43
13.1
13.2
13.3
13.4
14.
Electronic Submission Framework
Infrastructure Design
ESF SUBMISSION ITEM DOCUMENT ............................................................................ 43
IMF HANDLER ................................................................................................ 44
ATTRBIBUTE REPORT HANDLER ............................................................................... 44
SOE HANDLER ................................................................................................ 44
ATTACHMENT SUPPORT ............................................................................................... 45
14.1
ATTACHMENT TYPE DEFINITION............................................................................... 45
14.2
UPLOADING AN ATTACHMENT ................................................................................ 46
14.2.1Attachments Using Web Service ..................................................................... 46
14.2.2Virus Scanning File Uploads ........................................................................... 46
14.2.3Archiving Attachments ................................................................................. 46
14.2.4Business Agent Handling ............................................................................... 47
15.
SECURITY MATRIX .................................................................................................... 48
Page 5
Ministry of Forests
Electronic Submission Framework
Detailed Design
1. SCOPE
The scope of this document is to define the design of the base services of the Electronic
Submission Framework and specifically the components required to fulfill those services.
1.1
REFERENCES
This document is technical in nature and assumes the reader is familiar Java Messaging
Service (JMS), Oracle PL/SQL, XML, XSLT, GML and Java J2EE. Specific documents
referenced are:


Message Queuing Investigation
Requirements Document – Electronic Submission Framework
Version 1.4
Page 6
Ministry of Forests
Electronic Submission Framework
Detailed Design
2. OVERVIEW
The word “agent” is defined as:
\A"gent\, n. 1. One who acts for, or in the place of, another, by authority from him; one
entrusted with the business of another; a substitute; a deputy; a factor.
In the realm of ESF, “Agents” are components that will provide communication between
loosely coupled business processes throughout the life cycle of an electronic submission. In
simpler terms, the Agent acts like a FedEx courier. Consider the following scenario:
“Sally wishes to send a letter to John and needs to ensure that John
receives the package. She calls FedEx and promptly a courier is
standing at Sally’s door to accept the letter and deliver it to John on
Sally’s behalf. The courier takes the letter from Sally, drops it into a
FedEx envelope, places a FedEx tracking number on it and hands the
number to Sally. The courier puts the FedEx envelope into his pack
and leaves.
Some time later, John’s doorbell rings and there is a FedEx courier
standing there with a letter from Sally. John accepts the letter and
acknowledges the receipt by signing the waybill and decides to send a
return letter to Sally. He follows the same steps that Sally went
through.”
While this is a simple example, it does highlight some common attributes and services we
rely on in a courier service that should be present in an ESF Agent and the services they
provide.
2.1
ENSURED DELIVERY
As in the letter scenario above, once Sally handed the letter to the courier, she no
longer had to worry about how to get the letter to John; the courier took
responsibility for the delivery of the letter into John’s hands. Within the ESF, a
Queue Agent will accept a payload for delivery to a destination, whether it is a simple
text message or a 100 MB FTA electronic submission and ensure the delivery to the
responsible Agent.
2.2
HIDDEN DELIVERY METH OD
The scenario above also highlights the fact that Sally does not know how the letter is
going to be transferred to John. As long as the actual letter she sent arrives safe
and sound in John’s hands, she is a happy customer. The letter could travel by
truck, bus, plane or any combination of the three. These logistics are hidden from
the client and is left to the discretion of FedEx. The same goes for a Queue Agent.
The Queue Agent will choose the best delivery method for the payload and will
ensure the payload is delivered in the exact form it was sent in.
Version 1.4
Page 7
Ministry of Forests
Electronic Submission Framework
Detailed Design
Also note from the scenario that the courier put the letter into a FedEx envelope.
This will allow FedEx to transfer/track the letter through the FedEx delivery system
from start to finish, as well as place the letter in a form that is familiar and
compatible with the FedEx infrastructure. A Queue Agent will perform the exact
same operation for the client, packaging up a payload within its own ESF delivery
package and will deliver the payload to the receiving process in its original form.
2.3
AUTOMATED ACKNOWLEDG EMENT/ TRACKING
Sally could have contacted FedEx to check on status of the delivery of her letter.
She would present FedEx with the tracking number that was given to her by the
courier that accepted the package, and they could tell her the exact day John signed
for the letter. Similarly the Queue Agent needs to be able to file status reports
automatically back into the tracking system, so that all transactions for a submission
can be fully audited and traced.
This document discusses the components necessary to provide a service to accept and
reliably deliver electronic submissions for processing by another loosely coupled business
process.
Version 1.4
Page 8
Ministry of Forests
Electronic Submission Framework
Detailed Design
3. STORAGE DESIGN
The ESF requirements for storage can be broken down into three main areas: Application
Database, Messaging Queues and disk file store. It is estimated that we can expect
approximately 30,000 transactions in a year for both the RESULTS and FTA projects. From
this we will define an estimate of the storage requirements in each of the areas indicated.
3.1
APPLICATION DATABASE
The application database will be a standard WebADE application style configuration
with the one caveat that a submission may be stored in the database in a BLOB
format. Whether a submission is defined to be stored or not is defined within the
XML_SCHEMA entity. If the business area chooses to keep the original submission
on record, they also must supply how long they would like to keep it online in the
database. While there is no plan to do archiving at this time, the framework is in
place to achieve this functionality.
3.2
MESSAGING QUEUES
The queue storage is basically a temporary storage for the submissions during
delivery of the messages. If the system were operating optimally, there would
normally be one or two submissions waiting in the queue.
However, should
something go wrong and an Agent stopped processing, the queue infrastructure
should be able to store many requests for an amount of time. This is one of the
benefits of using asynchronous messaging; the infrastructure should accommodate
this by providing ample space for the worst-case scenario.
3.3
FILE STORE
The ESF website has a requirement to store files on a file system both permanently
for schema related information as well as temporarily for submission processing.
These parameters are defined in the web application’s WEB.XML file.
3.3.1
F ILE S TORE
The ESF requires a location on the web server in order to publish the schemas
and stylesheets used in the operation of ESF. Not only will ESF use this directly
when processing submissions but the ESF also publishes the available schemas to
the ESF users for their own use in client side applications for generating the data.
There are two attributes that affect this operation: esf-file-store-location and esffile-store-url. The location is used for ESF internally to manage files in that
location and the URL tells ESF how the files need to be accessed by outside users
and applications.
This URL will be a fully formed URL such as
http://www.for.gov.bc.ca/schema.
Version 1.4
Page 9
Ministry of Forests
Electronic Submission Framework
Detailed Design
A schema configured for the ESF will be stored below this mount point, in a
directory named after the schema name. The schema naming will adhere to the
following format:
http://<host.name>/schema/<schema name>/<schema
version>/<schema>.xsd
Example:
http://www.for.gov.bc.ca/schema/fta/2/fta.xsd
http://www.for.gov.bc.ca/schema/fta/2/fta_summary.xslt
3.3.2
SCRATCH DIRECTORY
The ESF system requires temporary file space for the processing of incoming
submissions. The ESF stores the file temporarily while initially verifying the
correctness of the submission as well as possibly creating transformations of the
original submission for placing on the queues. The location of the Scratch
Directory is configured in the web application’s WEB.XML via the esf-scratchdirectory attribute.
Version 1.4
Page 10
Ministry of Forests
Electronic Submission Framework
Detailed Design
4. QUEUING
The core of the ESF infrastructure will be implemented using asynchronous message
queuing. In particular, the implementation will leverage Oracle Advanced Queuing (OAQ) as
delivered in the Enterprise Edition of the Oracle database. This section of the document will
define the design of the queues for use by ESF.
4.1
QUEUE MANAGEMENT
OAQ has the ability to be managed via Oracle Enterprise Manager as well as
programmatically from various environments such as PL/SQL and Java. The initial
release of ESF will employ the PL/SQL management scheme, leaving the complete
queue management integration to be handled via Oracle Enterprise Manager or
possibly extending the ESF Admin interface in the future.
The queues to be used for the ESF will have the following properties:
Setting
Value
Description
PAYLOAD TYPE
JMS_BYTES_MSG
A stream of un-interpreted bytes.
This message type is for literally
encoding a body to match an existing
message format.
This format will
also enable us to seamlessly break
apart a single submission across
multiple physical queue messages.
MULTI-CONSUMER
TRUE
Allow multiple consumers to register
and de-queue messages from a
single queue.
SORT LIST
PRIORITY/ENQ_TIME
Will allow the ESF to prioritize
messages that should be processed
ASAP such as a Ping Message. Also
allows for future enhancements to
API
for
application-defined
prioritization.
MESSAGE
GROUPING
TRANSACTIONAL
Allows messages in a single queue to
be grouped together in a single
transaction. This ensures the same
consumer in a single transaction
consumes a group of messages. This
feature will be used to enable a
submission to exist across multiple
physical messages within the queue.
QUEUE VERSION
8.1
Queue mode to use.
Version 1.4
Page 11
Ministry of Forests
Electronic Submission Framework
Detailed Design
While some of the configuration for accepting submissions can be customized,
generally there will be the following queue configuration:
ESF Status Queue
FTA Queue
RESULTS Queue
This queue will belong to the ESF and all Agents will have
access to submit messages to the queue. Only the ESF
Agent will have the ability to de-queue.
Queue used for FTA submissions. De-queue ability to the
FTA Agent service ID.
Queue used for RESULTS submissions. De-queue ability to
the RESULTS Agent service ID
Below are some PL/SQL examples of some of the basic queue administration tasks.
Please refer to the Oracle documentation for a more in-depth description of these
commands and other queue management specifics.
Version 1.4
Page 12
Ministry of Forests
Electronic Submission Framework
Detailed Design
Creating Users
# Create the user to administer the queues
CREATE USER aq IDENTIFIED BY password PROFILE SYSTEM_MANAGER;
GRANT CONNECT, RESOURCE TO aq;
# Grant the AQ_ADMIN Role to the user.
GRANT AQ_ADMINISTRATOR_ROLE TO aq;
# Grant access to AQ admin functions.
GRANT EXECUTE ON DBMS_AQADM TO aq;
# Grant access to enqueue and de-queue.
GRANT EXECUTE ON DBMS_AQ TO aq;
# Grant access to Java use.
GRANT EXECUTE ON DBMS_AQIN TO aq;
GRANT EXECUTE ON DBMS_AQJMS TO aq;
# Create a generic user of queues
CREATE USER aq_user IDENTIFIED BY password PROFILE DEFAULT;
GRANT CONNECT, RESOURCE TO aq_user;
# AQ_USER role is no longer used. Simply grant access dbms_aq
GRANT EXECUTE ON dbms_aq TO aq_user;
# Grant access to Java use.
GRANT EXECUTE ON DBMS_AQIN TO aq_user;
GRANT EXECUTE ON DBMS_AQJMS TO aq_user;
Creating a Queue
Declare
Begin
DBMS_AQADM.CREATE_QUEUE_TABLE (
queue_table
=> 'aq.JMSBytes_q_inbox_table',
multiple_consumers
=> TRUE,
queue_payload_type
=> 'SYS.AQ$_JMS_BYTES_MSG',
sort_list
=> ’PRIORITY,ENQ_TIME’,
message_grouping
=> DBMS_AQADM.TRANSACTIONAL,
compatible
=> '8.1',
storage_clause
=> 'tablespace aq');
DBMS_AQADM.CREATE_QUEUE (
queue_name
=> 'JMSBytes_q_Inbox',
queue_table
=> 'aq.JMSBytes_q_inbox_table');
DBMS_AQADM.START_QUEUE (
queue_name
=> 'JMSBytes_q_Inbox');
# Grant system wide de-queue to user
DBMS_AQADM.GRANT_SYSTEM_PRIVILEGE(
privilege
=> ’ENQUEUE_ANY’,
grantee
=> ’aq_user’,
admin_option
=> FALSE);
# Grant specific rights for a user to a queue
DBMS_AQADM.GRANT_QUEUE_PRIVILEGE (
privilege => 'DEQUEUE',
queue_name => ' JMSBytes_q_Inbox ',
grantee => 'aq_user',
grant_option => FALSE);
End;
Version 1.4
Page 13
Ministry of Forests
Electronic Submission Framework
Detailed Design
5. MESSAGES
In order to be able to track the flow of messages through ESF, a certain amount of meta
data must be included with each message. As per the Electronic Submission Requirements
document, the ESF requires the following types of messages:



Payload
Status
Ping
The ESF will provide various different message types for use in sending and receiving
messages. While there are messages that have a special meaning such as “Status
Message”, most are simply convenience classes for sending and receiving familiar objects
such as a file or even a native Java object.
All message used within the ESF will be based on a JMS_BYTES_MSG. The extended
information about a message such as the submission id, will be encoded into the message
using the message property methods provided in the JMS API. An example of this follows:
bytes_msg = q_sess.createBytesMessage();
bytes_msg.setLongProperty(“submissionID”, in_msg.getSubmissionID() );
bytes_msg.setLongProperty(“sentTimeStamp”, new Date().getTime() );
bytes_msg.setStringProperty(“originatorAgentID”, in_msg.getOriginatorID() );
bytes_msg.setStringProperty(“submittedBy”, in_msg.getSubmittedBy() );
bytes_msg.setStringProperty(“messageType”, “StatusMessage”);
Refer the Class diagram below for the currently supported message types:
Version 1.4
Page 14
Ministry of Forests
5.1
Electronic Submission Framework
Detailed Design
TEXT MESSAGE
This is a generic message for simply sending and receiving a text message. The text
of the message will be converted to bytes and save as part of the payload.
bytes_msg = q_sess.createBytesMessage();
bytes_msg.setLongProperty(“submissionID”, in_msg.getSubmissionID() );
bytes_msg.setLongProperty(“sentTimeStamp”, new Date().getTime() );
bytes_msg.setStringProperty(“originatorAgentID”, in_msg.getOriginatorID() );
bytes_msg.setStringProperty(“submittedBy”, in_msg.getSubmittedBy() );
// Now set the message type specific properties.
bytes_msg.setStringProperty(“messageType”, “TextMessage”);
// Now add the text as the payload
bytes_msg.writeBytes(msg.getBytes());
5.2
PING MESSAGE
This message is used internally by the ESF system in order to test infrastructure
availability. The message contains the destination Agent ID to enable directed Agent
pinging should the queue in question be monitored by multiple Agents.
The
Version 1.4
Page 15
Ministry of Forests
Electronic Submission Framework
Detailed Design
QueueAgent will automatically receive a Ping Message and reply with its own Ping
Message back to the standard ESF Status Queue. This message will be implemented
with a simple message with some attributes set on the header of the message to
indicate its functionality. The Ping Message will be sent with a priority of nine, to
ensure timely response to system queries.
bytes_msg = q_sess.createBytesMessage();
bytes_msg.setLongProperty(“submissionID”, in_msg.getSubmissionID() );
bytes_msg.setLongProperty(“sentTimeStamp”, new Date().getTime() );
bytes_msg.setStringProperty(“originatorAgentID”, in_msg.getOriginatorID() );
bytes_msg.setStringProperty(“submittedBy”, in_msg.getSubmittedBy() );
// Now set the message type specific properties.
bytes_msg.setStringProperty(“messageType”, “PingMessage”);
5.3
STATUS MESSAGE
This message is used in order to relay messages back to the ESF system. All status
messages sent back to the ESF must reference the original Submission ID that was
assigned to the submission when it was first submitted by the user.
These status messages will be viewable by the submitter using a web browser, so
the content of the messages within the status may contain HTML tags such as
formatting and hyperlinks to take the user to another web application.
Any
formatting should be kept simple so the style does not conflict with the one used
within the ESF web application.
A status message may also include an attribute to update the submission state. If
this is included as part of the status message, the ESF Agent will also update the
submission state as part of recording the rest of the status message.
The notification indicator is used by the Agent responsible for processing the Status
Messages to decide whether or not this message will be sent to the submitter via email. This indicator can be set to the following values:
Indicator
Value
Description
NOTIFYNOW
1
REGULAR
2
NONE
3
NOTIFYADMIN
4
This message and any other that may be
waiting in the normal mail schedule queue
will be sent immediately.
This message will be delivered on the normal
notification schedule defined for the
submission schema.
This message will not be emailed to the
submitter and will only be viewable via the
ESF website.
This message will be emailed to the
administrator responsible for the schema in
question. The message is not recorded in
ESF, only delivered to an administrator.
Version 1.4
Page 16
Ministry of Forests
Electronic Submission Framework
Detailed Design
bytes_msg = q_sess.createBytesMessage();
bytes_msg.setLongProperty(“submissionID”, in_msg.getSubmissionID() );
bytes_msg.setLongProperty(“sentTimeStamp”, new Date().getTime() );
bytes_msg.setStringProperty(“originatorAgentID”, in_msg.getOriginatorID() );
bytes_msg.setStringProperty(“submittedBy”, in_msg.getSubmittedBy() );
// Now set the message type specific properties.
bytes_msg.setStringProperty(“messageType”, “StatusMessage”);
bytes_msg.setIntProperty(“notificationIndidcator”,in_msg.getNotificationIndicator() );
bytes_msg.setStringProperty(“statusCode”, in_msg.getStatusCode() );
bytes_msg.setStringProperty(“submissionState”, in_msg.getSubmissionState() );
// Now add the text of the message as the payload
bytes_msg.writeBytes(in_msg.getText().getBytes(),
0,
in_msg.getText().length());
5.4
FILE MESSAGE
This message is used to send and receive files through the ESF system. The files will
be read as un-interpreted bytes, so formats other than ASCII are acceptable.
Theoretically, there are no maximum sizes to the file, however there are limits set by
some of the infrastructure components used in the implementation. The ESF was
designed and tested with a maximum 100-megabyte file size specification.
bytes_msg = q_sess.createBytesMessage();
bytes_msg.setLongProperty(“submissionID”, in_msg.getSubmissionID() );
bytes_msg.setLongProperty(“sentTimeStamp”, new Date().getTime() );
bytes_msg.setStringProperty(“originatorAgentID”, in_msg.getOriginatorID() );
bytes_msg.setStringProperty(“submittedBy”, in_msg.getSubmittedBy() );
// Now set the message type specific properties.
bytes_msg.setStringProperty(“messageType”, “FileMessage”);
// Now add the file as a payload of the message.
5.4.1
S UBMISSION M ESSAGE
This message is used to send and receive submissions to the Business Agents.
Initially a File Message was used for this purpose but this was introduced as a
counterpart to the Attachment Message. This will make the difference between
them clear for a developer of a Custom Business Agent.
bytes_msg = q_sess.createBytesMessage();
bytes_msg.setLongProperty(“submissionID”, in_msg.getSubmissionID() );
bytes_msg.setLongProperty(“sentTimeStamp”, new Date().getTime() );
bytes_msg.setStringProperty(“originatorAgentID”, in_msg.getOriginatorID() );
bytes_msg.setStringProperty(“submittedBy”, in_msg.getSubmittedBy() );
// Now set the message type specific properties.
bytes_msg.setStringProperty(“messageType”, “SubmissionMessage”);
Version 1.4
Page 17
Ministry of Forests
Electronic Submission Framework
Detailed Design
// Now add the file as a payload of the message.
5.4.2
A TTACHMENT M ESSAGE
The Attachment Message was introduced to allow a submitter provide additional
information (binary data such as PDF’s or images) to augment their original XML
based submission.
bytes_msg = q_sess.createBytesMessage();
bytes_msg.setLongProperty(“submissionID”, in_msg.getSubmissionID() );
bytes_msg.setLongProperty(“sentTimeStamp”, new Date().getTime() );
bytes_msg.setStringProperty(“originatorAgentID”, in_msg.getOriginatorID() );
bytes_msg.setStringProperty(“submittedBy”, in_msg.getSubmittedBy() );
// Now set the message type specific properties.
bytes_msg.setStringProperty(“messageType”, “AttachmentMessage”);
bytes_msg.setStringProperty(“attachmentType”, “OpeningInfo”);
bytes_msg.setStringProperty(“attachmentNote”, “Test from user”);
// Now add the file as a payload of the message.
5.5
OBJECT MESSAGE
This message is used to send and receive native Java objects through the ESF
system. While the Object Message is constrained by the same size limits as the File
Message, it can send any type of Java object as long as the object is serializable.
The ESF QueueAgent will simply serialize the object to a file and send it via the
queue in a byte format. The receiving side will deserialize the object and package it
back up into an object message before handing it to the intended receiver of the
message.
bytes_msg = q_sess.createBytesMessage();
bytes_msg.setLongProperty(“submissionID”, in_msg.getSubmissionID() );
bytes_msg.setLongProperty(“sentTimeStamp”, new Date().getTime() );
bytes_msg.setStringProperty(“originatorAgentID”, in_msg.getOriginatorID() );
bytes_msg.setStringProperty(“submittedBy”, in_msg.getSubmittedBy() );
// Now set the message type specific properties.
bytes_msg.setStringProperty(“messageType”, “ObjectMessage”);
// Now add the file containing the serialized object as a payload of the message.
5.6
AUDIT MESSAGE
This message is used to send informational messages back to the ESF for use in
monitoring/troubleshooting the health of the distributed processing. Audit messages
are not viewable by the submitter, so technical information regarding the processing
of the information can be placed in these messages.
bytes_msg = q_sess.createBytesMessage();
Version 1.4
Page 18
Ministry of Forests
Electronic Submission Framework
Detailed Design
bytes_msg.setLongProperty(“submissionID”, in_msg.getSubmissionID() );
bytes_msg.setLongProperty(“sentTimeStamp”, new Date().getTime() );
bytes_msg.setStringProperty(“originatorAgentID”, in_msg.getOriginatorID() );
bytes_msg.setStringProperty(“submittedBy”, in_msg.getSubmittedBy() );
// Now set the message type specific properties.
bytes_msg.setStringProperty(“messageType”, “AuditMessage”);
// Now add the text of the message as the payload
bytes_msg.writeBytes(in_msg.getText().getBytes(),
0,
in_msg.getText().length());
Version 1.4
Page 19
Ministry of Forests
Electronic Submission Framework
Detailed Design
6. QUEUE AGENTS
A Queue Agent is responsible for delivering and receiving all messages to and from the
queues. The Queue Agent will be developed using Java and JMS. A Queue Agent will
provide the following base services for communication within the ESF system: receive
message, send message and send status. Along with the public services a Queue Agent will
offer, we must also define the infrastructure and protocol that must be in place in order to
ensure reliable and convenient operation.
6.1
PAYLOAD
Transfer of payloads in an un-interpreted byte format will allow us to transfer the
payload to the destination in the exact format it was sent in. This feature will also
allow us to perform other operations such as splitting a payload across multiple
messages, should the size of the payload deem this necessary. The byte format will
also give us the option to send a non-text payload, should the need arise.
Should the need arise to split the message payload across multiple queue messages,
the message grouping feature will be used in order to ensure the same Agent dequeues all messages belonging to the single submission. The underlying queuing
system has some limitations on the size of message that can be put onto a queue.
An optimal size will be defined and any submissions that exceed that threshold, will
be split across multiple queue messages and reconstructed by the receiving agent.
6.2
TRANSPORT IMPLEMEN TA TION
The underlying transport will be implemented using Oracle Advanced Queuing and
the Java JMS API; however, none of this implementation is to be unveiled to the
clients of a Queue Agent. The JMS API and asynchronous messaging can be
complicated and only needs to be implemented once. The API needs to be as simple
as copying a file so there is no chance of misuse of the queues by a developer. The
Queue Agent will be used by applications and possibly sub-classed in order to create
Business Agents to process submissions made to the ESF.
6.2.1
Q UEUE P OLLING
As many issues have arisen due to the instable nature of the callback
implementation in the 8i OAQ API, ESF has been implemented using polling to
retrieve messages. Essentially, each agent will routinely check its queues to look
for messages waiting. Although this means that message retrieval is not as
immediate as notification, it is much more reliable with the current technology.
This can be reviewed at a later time, should the underlying database be upgraded
to 9i.
Version 1.4
Page 20
Ministry of Forests
6.2.2
Electronic Submission Framework
Detailed Design
E XCEPTION Q UEUES
By default, the Oracle Advanced Queues supplies one exception queue per queue
table. All messages for queues defined in this queue table will end up on this one
queue when the retry count has expired. ESF and the ESF Base Agent now will
supply the ability for the Agents to send messages, specifying a different
exception queue.
Generally it is good practice to create an exception queue for each queue so that
it is easily identifiable as to the source queue of the exception message. The
specification of the exception queue to be used is done at the time of sending the
message. The base agent will allow the specification of this queue via an optional
attribute in the JMSConfig.xml file in order to ensure backwards compatability.
6.3
AUTOMATED AUDITING
The Queue Agent will provide some automated auditing to track certain Audit Events
during the normal operation of dequeueing a message from the queue. These Audit
Messages will be returned automatically to the ESF Status Queue without any
developer interaction. Depending on some of the physical implementation details of
the Agents, it may make sense to supply these notifications via HTTP, should there
be any firewall implementations
6.3.1
A UDIT E VENTS
Audit Events are events that occur within the ESF system in the normal operation
of the asynchronous queuing system.
Because of the complexity of
asynchronous messaging, it is important to keep very verbose log information
about the flow of messages through the system. This logging helps in the
troubleshooting of possible errors in the processing as well as providing valuable
information when it comes to tuning and maintenance.
Below is a list of the possible Audit Messages that are used to describe events
within the ESF system:
ID
Message
1
Submission XML verified.
2
Submission GML verified.
3
Submission queued.
4
Agent has attempted to de-queue submission.
5
Agent has successfully de-queued submission.
6
Agent has sent message.
7
Received submission status update.
8
Received submission state change request.
9
Agent has successfully processed the submission.
10
Agent has failed while processing the submission.
Version 1.4
Page 21
Ministry of Forests
6.4
Electronic Submission Framework
Detailed Design
AGENT INITIALIZATION
The queue configuration for the Agents will be done via an XML configuration file in
the first release. A client of a queue will be able reference a queue simply by name if
it has been configured in the XML file. A future release may add the ability to use a
networked configuration service for this connection information.
An example of this functionality is as follows:
// Create the agent
SimpleAgent sAgent = new SimpleAgent(“FTA Agent”);
// Send a message to a queue already defined in the XML config file
sAgent.sendMessage(“FTA Spatial Queue”, new TextMessage(“This is a test.”) );
Each Agent will have a requirement for a temporary disk space location in order to
stream message off the queues and reassemble the submission for delivery to the
Agent logic. This location will be configured in the XML configuration file. A file is
needed in order to be able to support large objects that may not be able to fit into
memory, such as a 100MB submission file.
6.5
TRANSACTIONS
The QueueAgent supplies the method (commitMessage(submissionID)) for finally
removing a message from the queue once the message processing is complete. This
is to be called after the Agent has successfully processed the message contents.
Calling this method removes the submission from the queue so that it will not be
processed again. As you can imagine, it may make sense to leave it on the queue
for processing at a later time if there was a failure in the post-processing.
The trick is to decide when to leave it on the queuing system. A good rule of thumb
for this decision is to determine if the error encountered during the processing is an
error that may correct itself (i.e. poor network connectivity or database down) if the
processing was retried at a later time. If the message has a basic error in the
composition of it and it will never be processed correctly, it is better to commit the
message and handle the error appropriately.
Below is some sample code of how it might be used in a typical scenario:
import ca.gov.bc.mof.esf.agent.*;
import ca.gov.bc.mof.esf.message.*;
public class FTAAgent extends QueueAgent {
public FTAAgent() {
}
public void receiveMessage(ESFMessage msg) {
if( msg instanceof TextMessage ) {
TextMessage tmsg = (TextMessage)msg;
Version 1.4
Page 22
Ministry of Forests
Electronic Submission Framework
Detailed Design
System.out.println( tmsg.getMessageText() );
// Do some more good stuff with the message…
// Now commit the message in order to remove it from the queues
if( SUCCESS ) {
this.commitMessage( msg.getSubmissionID() );
//Send status message
this.sendStatus(msg.getSubmissionID(),
StatusMessage.ACCEPTED,
"Cut Permit Accepted",
"The submission has been accepted.");
}
}
}
}
Behind the scenes, the QueueAgent code will perform the following logic when dequeuing and committing the message.
1. The QueueAgent receives a message on behalf of the client. The QueueAgent
locks the message on the queue.
2. The QueueAgent de-queues the message and recreates the message as the
original ESFMessage type.
3. The QueueAgent passes the new ESFMessage to the client code by calling the
receiveMessage(ESFMessage) method, which was implemented by the client.
4. The QueueAgent keeps the lock on the message until the client calls the
commitMessage() method or the receiveMesssage() call returns.
The
QueueAgent deletes the message from the queue.
Version 1.4
Page 23
Ministry of Forests
Electronic Submission Framework
Detailed Design
Below is the definition of a Queue Agent as well as the implementing components:
6.6
AGENT MANAGEMENT
As of ESF 2.0, agents will be manageable in the ESF web application through a remote
interface. This interface will enable ESF to discover all agents hosted at a particular
location, and then expose a set of tools to the ESF administrator. These tools will allow
them to view the current status of each agent, as well as issue start and stop commands to
these agents.
The architecture is detailed below using FTA Services as an example for remote agent
management:
Version 1.4
Page 24
Ministry of Forests
Electronic Submission Framework
Detailed Design
Web Interface
Web Interface
FTA Agent
ESF JMX
Agent
ESF Agent
Render
Agent
JMX Server
JMS/AQ
JMX/RMI
JMX Server
JMX/RMI
ESF
Map Report
Agent
Spatial
Agent
JMS/AQ
FTA Services
Syncronous Monitor/Manage
Asyncronous Submissions
6.6.1
Oracle AQ
Asyncronous Notifications
JMX AS A M ANAGEMENT F RAMEWORK
The key to allowing ESF to access the agents assigned to a particular submission schema is
to use a general management API that exposes access to an application’s internal objects in
a controlled manner. The framework to be used for ESF is the Java Management Extensions
(JMX).
JMX is a standard framework that allows Java objects to be accessed locally and remotely
with little or no change to the existing code base. This API also abstracts the method of
communication between the managing code and the objects to be managed. This allows
virtually any underlying communications protocol to be used, including RMI, HTTP, or Web
Services.
ESF’s agent management implementation will use RMI as the underlying
transport protocol.
6.6.2
R EMOTE M ANAGEMENT
The remote connectivity within JMX has been defined by a separate standard: Java
Management Extensions (JMX) Remote API 1.0 (JSR-160). This defines a standard for
remote connectivity but does not actually specify a single protocol and in-fact recommends
a couple alternatives for underlying transports such as RMI and JMXRMP. For the initial
configuration, an RMI solution will be used due to the fact that this is the only mandatory
transport of the specification. For this reason, this will be the most well-supported of the
choices among other JMX products.
The other aspect of Remote Management is the ability lookup a JMX service in order to
obtain the bindings to the service at runtime rather than hard-coded locations. For this
problem space, various standard technologies exist for perform these types of service
Version 1.4
Page 25
Ministry of Forests
Electronic Submission Framework
Detailed Design
lookups: JNDI, JINI and SLP are among some of the choices.
chosen JNDI.
6.6.3
For this release, we have
S CHEDULING OF T ASKS
In the previous release of the ESF Base Agent code, the Base Agent contained its own
scheduling logic for polling the queue. With this release, the scheduling component will be
factored out of the Agent and moved into a general purpose Scheduling system based on
JMX. JMX supplies much of the functionality of a scheduled task in the form of Timers. This
is standard JMX functionality and easily applied as a generic Task Scheduler.
The configuration of the tasks and schedules will be driven from an XML file. This will be
read at initialization and all schedules will be created automatically by the scheduler. At this
time, there is no requirement for runtime scheduling but this may be required in future
releases. The Scheduler will be realised as a WebADE extension. Below is a diagram
illustrating how the Agent scheduling will be accomplished.
Admin
Schedule
(JMX Timer)
Notification
Scheduled Task
(ESF Agent)
Initialize
JMX Server
Scheduler Configuration
JMX Management
Connector
JMX Control
WebADE Scheduler Extension
6.6.4
A UTHENTICATION AND A UTHORIZATION
Authentication of JMX MBean requests sent to agents will conform to the JMX standard,
allowing username/password credentials to be passed in the request.
Agents will
authenticate the credentials against the appropriate Active Directory, refusing the user’s
request if they fail authentication.
An optional authentication is also proposed to allow for a signed user request. If the
request to the agent originates from a trusted web application, instead of the user directly,
the web application may not have access to the user’s password. Prompting the user for
their password can be tedious for the user and also brings up various security issues.
Alternatively, the application can create a signed request on behalf of the user. The agents
will store the public key to decrypt the signed message, trusting that the user has been
authenticated by the referring application that signed the request.
Version 1.4
Page 26
Ministry of Forests
Electronic Submission Framework
Detailed Design
In addition to authentication of the user’s credentials, agents can also perform a validation
against the user’s roles in WebADE. If the user is not a member of the expected role, the
agent can also refuse to perform the action.
6.6.5
C USTOM B USINESS A GENT I NTEGRATION
Minimal changes will be required by Agent developers. A framework will be provided to
Agent developers to initialize the Agents, provide queue polling automation, and initialize
the Agent in a JMX server. This will provide further enable Agent developers to focus on
business requirements and not underlying infrastructure.
The Agent Developer Best
Practices document will outline the use of this framework.
The ESF will have the ability to register for Notification coming from the Business Agents
regarding events such as “Attempt to Dequeue”. This means that the Agent may be having
troubles processing messages. The ESF will indicate this on the System Overview page by
rendering the Agent in a yellow colour. From the Agent Status screen, the last events
received from the Agent will be visible.
Version 1.4
Page 27
Ministry of Forests
Electronic Submission Framework
Detailed Design
7. SUBMISSION BROKER
The Submission Broker is responsible for delivering the submissions made via the ESF
website to the queuing infrastructure. This section describes the processing required in
order to deliver a submission to the proper queue.
1.The form parameters from the submission form will provide the following
information:
frm_XML_SCHEMA_VERSION: Which version of the schema to use.
frm_XML_SCHEMA_ID: Which schema to validate against
frm_SUBMISSION_FILE: The actual file they submitted.
frm_USER_REFERENCE: A ref code supplied by the user for their internal use.
2.The WebADE API will provide the following information:
ade_USER: The currently logged in user id (“idir\jsmith”)
ade_ORG: The Organisation the current user belongs to.
ade_ORG_SOURCE: The location the information was retrieved from (e.g. IDir or
BCeID).
ade_EMAIL: The email address of the current user.
3.Retrieve a sequence from the SubmissionID sequence: seq_SubmissionID
4.Save the submission to a temporary file using the retrieved sequence as part of
the name in order to avoid collisions (e.g. submission4321.xml):
fil_SUBMISSION.
5.Retrieve the following information from the ESF database using
frm_XML_SCHEMA_VERSION and frm_XML_SCHEMA_ID in the “where” clause:
XML_SCHEMA_VERSION.SCHEMA_URL
XML_SCHEMA_VERSION.SUBMISSION_SUMMARY_XSLT_URL
XML_SCHEMA_VERSION.VALIDATE_SPATIAL_IND
XML_SCHEMA_VERSION.COORDINATE_SYSTEM_CODE??
XML_SCHEMA.SUBMISSION_ARCHIVE_IND
XML_SCHEMA.SUBMISSION_ARCHIVE_DURATION
ESF_SCHEMA_QUEUE_XREF.XSLT_URL
ESF_DISTRIBUTION_QUEUE.QUEUE_NAME
ESF_DISTRIBUTION_QUEUE.QUEUE_JDBC_URL
ESF_DISTRIBUTION_QUEUE.DB_SCHEMA
6.Parse the XML in the submission, verifying its correctness against the schema
defined (XML_SCHEMA_VERSION.SCHEMA_URL). If it fails, display error to user
and delete temp file.
7.If the VALIDATE_SPATIAL_IND is defined as 'Y', then validate GML within
submission using the COORDINATE_SYSTEM_CODE. if
COORDINATE_SYSTEM_CODE is not defined for this submission type, continue
with the GML validation with no re-projection. If it fails, display error to user
and delete temp file.
8.Now the submission is considered correct. Start a transaction with the ESF
database. Insert submission information into the ESF database. If the
SUBMISSION_ARCHIVE_IND is 'Y', then store the temp submission file in with
the other submission information.
9.Insert an audit message into the ESF_SUBMISSION_AUDIT, recording this event.
10.For each queue defined to receive the submission, perform the following steps:
Version 1.4
Page 28
Ministry of Forests
Electronic Submission Framework
Detailed Design
1.If the XSLT_URL is NOT NULL, process the temp submission file against
the ESF_SCHEMA_QUEUE_XREF.XSLT_URL and save results to another
temp file: Submission_4321_Transformed.xml
2.Create a Submission Message containing the information about the
submission. If the queue defined the submission to be transformed use
the transformed file from the step above, otherwise use the original
submission document as the file in this command:
SubmissionMessage fm = new SubmissionMessage(
seq_SUBMISSSION_ID,
new File( submission_file ) );
fm.setOriginatorAgentID( “Submission Broker” );
fm.setSentTimeStamp( new Date() );
3.Send message to the queue defined by the following:
ESF_DISTRIBUTION_QUEUE.QUEUE_NAME
ESF_DISTRIBUTION_QUEUE.QUEUE_JDBC_URL
ESF_DISTRIBUTION_QUEUE.DB_SCHEMA
4.Insert an audit message into the ESF_SUBMISSION_AUDIT, recording
this event.
5.Upon success, delete the transformed file, if one was created, and repeat
the process on the next queue, if another is defined.
11.Set the ESF_STATUS_CODE of the submission to 'SUB' for submitted.
12.Commit the submission information insert to the ESF database.
13.If a summary XSLT is defined for this submission type, process the temp
submission file against the SUBMISSION_SUMMARY_XSLT and display result to
the user.
14.Delete the temp submission file.
Version 1.4
Page 29
Ministry of Forests
Electronic Submission Framework
Detailed Design
8. STATUS AGENT
This Agent implementation will simplify the task of sending back a status to the ESF system,
without having to have in-depth knowledge of the workings of the ESF system or queuing.
Other than the usual Java class plumbing (i.e. constructor and setters and getters), the only
public method will be sendStatus. Behind the scenes, the Agent will package up the status
in an ESF Status Message and send it on the ESF Status Queue. Refer to the class diagram
for more information. Below is a sample code of how the StatusAgent will be used in a
typical scenario:
StatusAgent sa = new StatusAgent(“FTA Business Agent”); // Pass in an Agent ID
sa.sendStatus(1234,
StatusMessage.ACCEPTED,
"Received Submission",
"FTA has received your submission.");
sa.close(); // Close the Agent in order to cleanly close any queue sessions.
Version 1.4
Page 30
Ministry of Forests
Electronic Submission Framework
Detailed Design
9. SIMPLE BUSINESS AGENT
The Simple Business Agent (SBA) is designed with primarily sending messages in mind,
however the ability to receive messages with this agent is also possible. As with the
StatusAgent, the SBA keeps most of the details of the underlying implementation hidden
from the developer.
This makes this Agent a very attractive solution should the
requirements of the business be simple.
The SBA will provide the same functionality in regards to updating the status of an ESF
submission but also adds other functionality such as sending and receiving messages. This
implementation requires a more in-depth knowledge of the messaging system however it is
still kept as simple as possible to the developer.
Sending messages with the Simple Business Agent is accomplished by simply creating an
ESFMessage and calling the sendMessage() method on the Simple Business Agent.
Receiving a message is accomplished by registering as an event listener on the
SimpleBusinessAgent. The developer will provide a MessageReceived event handler to
process the new message, which will be passed in as part of the event object.
Below is some sample code of how it might be used in a typical scenario:
public class FTAAgent implements ESFMessageListener {
public FTAAgent() {
}
public void init() {
// Create the agent
SimpleAgent sAgent = new SimpleAgent(“FTA Agent”);
// Register this class as a ESF Message Listener
sAgent.addMessageListener(this);
sAgent.listenQueue(qCtx);
public void messageReceived(NewMessageEvent e) {
// Do good business stuff here…
// Send Status message back to ESF
sAgent.sendStatus(e.getNewMessage().getSubmissionID(),
StatusMessage.ACCEPTED,
"Cut Permit Accepted",
"The submission has been accepted.");
}
public void sendFTAMessage() {
// Create message to send
TextMessage tmsg = new TextMessage(“This is a test.”);
// Send message to Queue
sAgent.sendMessage(qCtx, tmsg);
}
}
}
Version 1.4
Page 31
Ministry of Forests
10.
Electronic Submission Framework
Detailed Design
ESF AGENT
The ESF Agent is responsible for handling all messages that arrive in the ESF Status Queue.
The queue can contain any of the following message types:



Status Message: An Agent wishes to update the status or state of a submission
that is “In Progress”. This message also gives the Agent control over the client
notification scheme. Refer to the Status Message section for more information on
the notification indicator usage.
Audit Message: An internal ESF message sent automatically by an Agent indicating
a submission audit event has fired.
Ping Message: An Agent has responded to a “Ping” message sent by the ESF.
Each of these messages must be applied to the ESF database in order to track progress of a
submission as it travels through the ESF infrastructure.
Below is a more in-depth
explanation of the processing required when each of these messages arrive in the “ESF
Status Queue”.
The ESF Agent should respond to ALL message types as this Agent provides the ESF with
the first line of troubleshooting and system verification. The following messages fall into
this category:



File Message: Process the file message and place an Audit message (with Extended
Audit) and Status Message on the Status Queue.
Submission Message: Same as above.
Attachment Message: Process attachment and place Status Message on the Status
Queue.
It should be noted that we might need an alternate solution for receiving status messages
from the Agents. The physical implementation of the Agents could be realized with a very
distributed model, so access to the ESF Status Queue may not possible. It would be
possible to offer this status interface as a “Web Service” implementation, once the
infrastructure was available to offer these types of services.
10.1 RECEIVING A STATUS M ESSAGE
When a status message arrives in the queue, it is the ESF Agent's responsibility to
update the status to the ESF database. A status message may contain information
for both ESF_SUBMISSION_MESSAGE as well as possibly change the state of a
submission.
1.Insert an audit message into the ESF_SUBMISSION_AUDIT, recording this event.
2.If the StatusMessage.getStatusCode() method returns a value, update the code
within the ELECTRONIC_SUBMISSION table, identified by
StatusMessage.getSubmissionId() method.
3.If the StatusMessage.getMessageText() method returns a value, insert a new
ESF_SUBMISSION_MESSAGE for the submission identified by
StatusMessage.getSubmissionId() method.
Version 1.4
Page 32
Ministry of Forests
Electronic Submission Framework
Detailed Design
4.If the notification indicator = SENDNOW then find scheduled mail job for the
submission, add new message to it and send it now.
5.If the notification indicator = REGULAR then find scheduled mail job for the
submission, add new message to it.
6.If the notification indicator = NONE or is not set, do nothing.
7.If the notification indicator = NOTIFYADMIN, create an email message with the
text of the message and send it to the administrator defined for the submission
type of the original message.
10.2 RECEIVING AN AUDIT M ESSAGE
An Audit Message is an internal ESF message sent from an Agent, updating the ESF
of certain events that fire while an Agent processes a message. Refer to the Audit
Events section for more information regarding the types of events that may occur.
Below is the processing required when an Audit Message arrives in the ESF Status
Queue:
1.Insert an audit message into the ESF_SUBMISSION_AUDIT, recording this event.
2.Insert the Audit into the ESF_SUBMISSION_AUDIT table, identifying the
AUDIT_SOURCE as AuditMessage.getOriginatorAgentID() and the
AUDIT_MESSAGE_ID as AuditMessage.getAuditMessageID().
10.3 RECEIVING A PING MES SAGE
A Ping Message is sent to the ESF Status Queue by an Agent in response to it
receiving a Ping Message on a queue it is listening on. The ESF Agent simply needs
to record the fact that a ping was returned by the Agent identified by the
PingMessage.getOriginatorAgentID() method.
Below is the processing required when an Ping Message arrives in the ESF Status
Queue:
1.Update the ESF_QUEUE_AGENT.LAST_PING_SUCCESS attribute with the current
time for the Agent identified by the PingMessage.getOriginatorAgentID()
method.
Version 1.4
Page 33
Ministry of Forests
11.
Electronic Submission Framework
Detailed Design
SECURITY
The security design of the ESF website must institute a secure and reliable mechanism for
receiving electronic submissions from industry. The design should also take a simple
approach to ensure the system can accommodate an expanded client group in the near
future, as there is a possibility of multiple projects and ministries leveraging the ESF.
11.1 AUTHENTICATION
Authentication will be implemented by the system provided by IIS to challenge a
user for a username password pair. The bulk of the target ESF users will be long to
BCeID, so their workstations will not be logged into any domain, so IIS needs to be
able to fall back from “Challenge/Response” (NTLM) to “Basic Authentication”. Due
to this default functionality, the channel between the user and the web server should
be encrypted to ensure the user id’s and passwords do not travel “in the clear”
across the Internet.
11.2 AUTHORIZATION
The authorization scheme for ESF has purposely been left simplistic. The need to
provide cross Ministry support as well as public access has prompted a design of a
simple yet effective scheme.
11.3 DEFINED ROLES
As defined the in ESF Requirement document, there are three roles that have
been identified. A fourth role has been identified (“Queue Administrator”) in
order to separate off the queue administration. This was done in order to hide
some of the sensitive information regarding the queue configuration such as
database passwords.





Submitter – A user of the ESF application that will submit a submission.
ESF Administrator – A person responsible for configuring agents and
uploading new schemas to the ESF system. The navigation for the Queue
Administration will not be available.
ESF Support – A person responsible for supporting a submission type such
as FTA. This will get them extended information on the administration side of
ESF but with Read Only access only.
Queue Administrator – A person responsible for configuring queues within
the ESF system. Only navigation for the Queue Administration will not be
available within the admin section.
Submission Browser – A user of the ESF application with extended rights
when it comes to the Submission Search screen.
Version 1.4
Page 34
Ministry of Forests
Electronic Submission Framework
Detailed Design
11.4 SUBMISSION QUERY RES TRICTIONS
While all roles defined in the ESF will have access to the search screen (ESF006), the
results will be filtered depending on who is currently logged into the system. The
access is defined as the following:
Submitter – Able to query personal submissions and submissions made by his
company/organisation.
Administrator, Support & Browser – Able to query all submissions.
11.5 SUBMISSION DETAIL RESTRICTIO NS
When viewing the details of a submission (ESF008), the screen will display status
messages of the processing of the submission after it was queued to the system for
processing by the business agent.
Submitter & Browser - The system will display only the business messages
returned by the business agent during the processing of the message
(ESF_SUBMISSION_MESSAGE).
Administrator & Support – The system will display the business processing
information as above as well as the ESF audit information (ESF_SUBMISSION_AUDIT
& ESF AUDIT_MESSAGE).
11.6 ADMINISTRATION RESTR ICTIONS
The ESF Support Role will have similar access to the administration side of ESF
however a person in this role will not have any access to affect changes. They will
be able to view all information regarding schemas and schema versions.
11.7 ENCRYPTION
The ESF application could possibly be accessed via public network paths. Because
the authentication will be relying on basic authentication and the fact that some
submission could contain private and confidential information, it is recommended the
ESF application implements strong encryption with the use of SSL.
An alternative to running the site under SSL, the Ministry could insist the application
be accessed via the BC Government VPN. This would ensure the information
travelling over the public network is secure.
11.8 TRACKING USER AND OR GANIZATION OWNERSHIP
In order to ensure user and organization data integrity within ESF, the User and Org
GUID will be stored within the ESF system. This follows standard practices within the
Ministry today.
Version 1.4
Page 35
Ministry of Forests
12.
Electronic Submission Framework
Detailed Design
SUBMISSIONS WEB SERV ICE
The ESF Submissions Web Service will allow clients of the ESF to connect directly to the ESF
programmatically rather than using the web HTML interface. This will allow external
agencies with high volumes of submissions to streamline their process by connecting their
own information systems to ESF, bypassing the manual submission upload process.
The Submissions Web Service will follow current accepted standards within the industry for
Web Service delivery. The model chosen for implementation closely follows the WS-I 1.0
Basic Profile. The service will be based on SOAP 1.1 using HTTP for the transport and WSDL
1.1 for the description of the service. At this time there is no standard government service
to provide UDDI support so this feature will not be implemented.
Apache Axis Framework will be used in the implementation of the service due to its industry
acceptance and stability. For detailed information about the design of Axis, please refer to
the Axis Architecture Guide on the Apache site. The following diagram illustrates the highlevel architecture of the Submissions Web Service and also shows how the integration is
done with the existing ESF functionality.
ESF Web User
ESF Submissions
Web Service User
IIS (Provides Authentication)
JRun Server
(esf.war)
AXIS Servlet
ESF Action Servlet
Submissions Web
Service
ESF Application Logic
Figure 1: Proposed Web Service Architecture
12.1 INTERFACE DESIGN
The Submissions Web Service will expose all of the public functionality of the ESF.
An RPC-based model will be used and the following interface will be implemented:
Version 1.4
Page 36
Ministry of Forests
12.1.1
Electronic Submission Framework
Detailed Design
ADD A TTACHMENT
An existing Submission ID will be passed into this service along with other metadata about the attachment. The attachment will be implemented using Soap with
Attachments specification, which is supported by Axis.
Parameter
submissionId
Type
long
attachmentType
String
attachmentNote
String
originalFileName
String
returns
AttachmentResult
Description
An existing ESF Submission
ID in which this attachment
will be “attached” to.
The attachment type as
defined
by
the
schema
definition within ESF. Each
Business Area will support
different attachment types.
A note from the submitter to
augment this attachment.
The name of the file used in
the attachment
AN object that contains the
result of the attempted
attachment submission.
NOTE: There is no attachmentFile parameter. The service expects one and only
one file in the attachment payload of the service request, not as a parameter.
The attachment data is not passed in as a string (as in makeSubmission or
uploadSubmission) but rather as an actual SOAP attachment and therefore is not
a parameter.
Version 1.4
Page 37
Ministry of Forests
12.1.2
Electronic Submission Framework
Detailed Design
GET S UBMISSION S TATUS
Given an ESF Submission ID, this method will return the details about the
submission including all status messages returned from the processing Business
Agents. The Status Messages will also contain Audit messages if the calling user
has the appropriate authority. The result will be returned in SubmissionStatus
complex type as detailed below:
12.1.3
GET S UPPORTED A TTACHMENT T YPES
Given a submissionType, returns all supported attachment types. Will return a
null array if the submissionType exists but does not support attachments. An
exception will be thrown if the submissionType could not be found.
Parameter
submissionType
Type
String
returns
AttachmentType[]
Description
The submission type in which to query for
supported attachment types.
An array containing a complex type
specifying the supported attachment types
as defined by the installed ESF schema.
Version 1.4
Page 38
Ministry of Forests
12.1.4
Electronic Submission Framework
Detailed Design
UPLOAD S UBMISSION
This method is identical to makeSubmission with additional parameters.
newly developed Web Service clients should make use of this method.
All
New parameters are originalFileName (allows caller to name and identify the file
that the submissionData is created from) and useCurrentUserEmail (allows caller
to specify which email address the status messages are delivered to, the address
in the submissionData or the current User’s email address).
12.1.5
GET S UPPORTED S UBMISSION T YPES
Returns a list of all supported submissions types available within the ESF. A
submission type may have multiple version of the submission available and
supported at any one time. The result will be returned in a SubmissionTypes
complex type as detailed below;
Parameter
returns
12.1.6
Type
SubmissionTypes[]
Description
An array containing a complex type
specifying the supported submission types.
IS S UPPORTED S UBMISSION T YPE
Return a boolean result indicating whether a particular submission type and
version is currently supported. It is prudent for a client to call this before
attempting to perform the makeSubmission method to ensure the submission to
be submitted is still supported.
12.1.7
MAKE S UBMISSION
This method is used to upload, validate, and finalize a new submission. The
Submissions service will only accept XML documents. ZIP files are not supported.
The XML file is passed in as a string (submissionData parameter) thus clients are
Version 1.4
Page 39
Ministry of Forests
Electronic Submission Framework
Detailed Design
responsible for converting the XML file to a string. Limitations on file size will be
machine dependant as memory heaps will affect how large a string can be
created.
The SubmissionResult will inform the client if the submission has succeeded. If
SubmissionResult.finalized is true, the submission was uploaded, validated and
finalized and the SubmissionResult.submissionId will have a non-zero value
indicating the submission id.
If the submission was not finalized,
SubmissionResult.lastSuccessfulStage will indicate if the submission was
uploaded, or validated, or finalized. The SubmissionResult.errors array will
contain any validation error messages if the file was uploaded but not validated.
The following complex type is returned to notify the calling client of the result of
the submission upload:
12.1.8
SEARCH S UBMISSION
This method is used to return a listing of known submissions that matches the
search criteria. Results are returned in pages; page number and the size of
pages are passed in as parameters. The default for pageNumber is 1, the default
for pageSize is 20. All parameters can be left blank.
The SubmissionDetails complex object is returned by the call with the results of
the search criteria.
Version 1.4
Page 40
Ministry of Forests
Electronic Submission Framework
Detailed Design
12.2 EXCEPTION HANDLING
There are three stages to each service request: initialize an ESF session, perform
security check on user/method, and the method call itself.
Exceptions can be thrown at any of the three stages. The exceptions can be mapped
directly to a SOAPException.
Each request goes through a session initialization handler to ensure that ESF is
available and ready to accept the requests. Each request then falls to a security
handler to ensure that the caller has the privileges to request that method. An
AxisFault can be thrown at either of these stages if the session fails to initialize or
the user does not have the required security.
AxisFault maps directly to a
SOAPException.
Each of the methods will throw a SubmissionsFault if any known exception is raised
during the execution of the method (i.e. unexpected parameter, data access error,
SQL error). SubmissionsFault maps to a SOAPException, but some clients will be
able to identify these specific exceptions from other SOAPExceptions.
AxisFault and SubmissionsFault map to SOAPException. Depending on how the client
is generated will depend on what exceptions are recognized. AXIS generated clients
will recognize AxisFault and SubmissionFault and can view the details in faultString.
Other clients may only recognize their own variant of SOAPException but should
provide some access to the details.
Version 1.4
Page 41
Ministry of Forests
Electronic Submission Framework
Detailed Design
Version 1.4
Page 42
Ministry of Forests
13.
Electronic Submission Framework
Detailed Design
ESF MAP VISUALIZATIO N TOOLS
Initially, a system named Tenure Map Service (TMS) was created in order to assist the
Licensee’s in creating high quality Electronic Submissions. The design for this application
was done in a way so that it could be integrated as part of the ESF at a later date should
the tool be found useful. This section covers the design and integration of TMS into ESF.
IMF
3) Initialize IMF
4) IMF Framework
4) ESF Tools
Submitter
(Business
Submission
Document)
1) Submit
3) Return Attribute Report
IMF Handler
Transformation
Services
2) Transform
ESF
Submission
Item
Document
Attribute Report
Handler
5) Return Land Use Report
SOE Handler
3) Request Report
Spatial Overlay
Engine
4) Return Report
ESF
13.1 ESF SUBMISSION ITEM DOCUMENT
In order for the functionality of TMS to be obtained in a general fashion, an intermediate
document format is defined in order for the tools within TMS to handle submissions in a
generalized spatially focused format. This generalized format has been defined in the
following schema document.
Version 1.4
Page 43
Ministry of Forests
Electronic Submission Framework
Detailed Design
Once the intermediate document is generated, the Tools can perform their operation on a
know document structure. Without this, the Tools would need to know about the structure
of each supported document type. Each business area that wants to use the Tools will be
required to create an XML Style Sheet transformation document to create a document as
specified by the submission-item.xsd. This will need to be done for every version of the
document as there may be changes to the business submission structure.
13.2 IMF HANDLER
The Map Visualization Tool is used to graphically display the submission item on a map with
other features that will be of interest to the business area. This will allow the submitter to
verify the submissions contents. The Map Visualization Tool is an IMF based application.
Once the business submission document has been transformed into the ESF Submission
Item Document, the features for a given submission item will be drawn on the map using
the “Acetate Layer” functionality of IMF. This will allow the submitter to see the features on
the map without the said features required to be in the spatial database.
The IMF initialization process will be as follows:
1. Pop a new window to display the IMF framework.
2. Zoom the Map window to the size and location for the selected submission item.
This is calculated from sum of all the map features included in the submission item.
3. Add the features to the acetate layer.
4. Display the ESF Map Visualization Tools page in the “Tools” frame of the IMF
Framework.
13.3 ATTRBIBUTE REPORT HA NDLER
This report simply shows the attributes of the submission as they are related to the spatial
features displayed on the map. The attributes are iterated over and placed in a formatted
report for viewing by the submitter.
13.4 SOE HANDLER
This tool will allow a submitter to view a Land Use Report for the spatial features included in
the Submission Item. The Tool will leverage a service provided by MSRM called Spatial
Overlay Engine (SOE). The features are transferred to SOE using HTTP and GML and a fully
formatted report is returned to ESF. The report is simply marshalled to the submitter as
formatted by SOE.
Version 1.4
Page 44
Ministry of Forests
14.
Electronic Submission Framework
Detailed Design
ATTACHMENT SUPPORT
A growing need by many business areas is the ability to augment their XML submission with
information other than well-formed XML. Data in PDF, DOC and JPG is a common
requirement for the various business areas. The ESF Attachment feature will address this
requirement.
14.1 ATTACHMENT TYPE DEFI NITION
On a schema version basis, this business area will be required to define the types of
attachments they want to accept. This is a business definition of the attachment type
rather than a technical specification such as MIME Type. The Business Area will provide an
XML document (as defined by ESF) which outlines the types of attachments that can be
submitted with each version of schema. The following sample XML document will define the
attachment types for a given system:
<?xml version="1.0" encoding="UTF-8"?>
<esfattach:Attachments
xmlns:esfattach="http://www.for.gov.bc.ca/schema/esfAttachments"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.for.gov.bc.ca/schema/esfAttachments
ESFAttachmentTypes.xsd ">
<attachment>
<attachmentName>Business Attachment 1</attachmentName>
<description>Description of the attachment use.</description>
<enabled>true</enabled>
</attachment>
<attachment>
<attachmentName>Business Attachment 2</attachmentName>
<description>Description of the attachment use.</description>
<enabled>true</enabled>
</attachment>
<attachment>
<attachmentName>Business Attachment 3</attachmentName>
<description>Example of a disabled attachment</description>
<enabled>false</enabled>
</attachment>
</esfattach:Attachments>
The following is the schema definition for the document above:
Version 1.4
Page 45
Ministry of Forests
Electronic Submission Framework
Detailed Design
The XML document specifying the accepted XML attachments can be defined as follows:



Defined when adding a new schema on “ESF105: Add New Schema”
Defined when adding a new schema version on “ESF104: Add Schema Version”
Added to an existing schema version via the “ESF103: Edit Schema Version”
14.2 UPLOADING AN ATTACHM ENT
Uploading an Attachment to augment an XML submission is performed via the “ESF105:
Submission Details” screen or via new Web Service endpoints. Once the standard XML
portion of an ESF submission has been made (validated and placed on the queue for the
Business Agent), the user may upload attachments related to the submission. The user
may continue to upload attachments while the submission remains “Submitted” or “In
Progress”. Once the submission is completed (as defined by the Business Agent by
submitting a status message that changes the submission state to “Completed”, “Rejected”
or “Partial”) the option to upload attachments will be removed from the page.
The attachment will be placed on the queue defined for the Schema Version using an ESF
AttachmentMessage. It is up for the Agent to check for this message type and handle the
attachment appropriately.
14.2.1
A TTACHMENTS U SING W EB S ERVICE
New endpoints will be added to the existing ESF Web Services in order to provide
Attachment support. The endpoint will use the “Soap with Attachments” specification for
encoding the file in the Web Service call.
14.2.2
V IRUS S CANNING F ILE U PLOADS
Because ESF is accepting binary files from the general public, effort is required to ensure
the file does not contain harmful contents such as viruses. The following scheme will be
used in order to help ensure the safety of the incoming binary data. The Ministry will
ensure “real-time” virus scanning is configured on the server and the virus scan definitions
are kept up-to-date.
1. Accept the file from the user or web service.
2. Write the file to a temporary location on the server’s file system.
3. Attempt to re-read the file from the file system after a period of time. If the file is
still available, it has not been quarantined and will be considered “safe” to process.
The process will also be repeated for files being removed from the database. They will first
be written to the temp file space to enable a second pass of virus scanning.
14.2.3
A RCHIVING A TTACHMENTS
During the submission process, ESF will hold a copy of the attachments as well provide
download ability similar to downloading the “Original Submission”. The attachments will be
archived for the duration of the submission process. Once the Business Agent completes
Version 1.4
Page 46
Ministry of Forests
Electronic Submission Framework
Detailed Design
the process by finalizing the ESF state, the attachments will be purged from ESF. This will
be enforced by the ESF Agent as it processes incoming status messages.
14.2.4
B USINESS A GENT H ANDLING
Each Business Area requiring Attachments will require enhancements to their Custom
Business Agent in order to handle the new message type. ESF will deliver the attachment
to the Agent within an ESF AttachmentMessage.
The message will have the same
submission ID as the original XML Submission. It is up to the Agent to correlate these
results. The message is defined in section “5 Messages”.
Version 1.4
Page 47
Ministry of Forests
SUPPORT
ESF ADMIN
BROWSER
SUBMITTER
ESF001
ESF002
ESF003
ESF004
ESF005
ESF006
ESF007
ESF008
ESF009
ESF010
ESF011
ESF012
ESF100
ESF101
ESF102
ESF103
ESF104
ESF105
ESF200
ESF201
ESF202
ESF203
ESF204
ESF300
ESF301
ESF302
ESF303
QUEUE ADMIN
SECURITY MATRIX
SCREEN
15.
Electronic Submission Framework
Detailed Design
X
X
X
X
X
X*
X
X**
X
X
X
X
X***
X***
X***
X***
X***
X***
X
X
X
X
X
X*
X
X**
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X*
X
X**
X
X
X
X
X
X
X
X
X
X*
X
X**
X
X
X
X
X***
X***
X***
X***
X
X
X
X
X
X
X
X
X
* Refer to section 11.4 for special handling
** Refer to section 11.5 for special handling
*** Refer to section 11.6 for special handling
Version 1.4
Page 48