WebSphere MQ Managed File Transfer
Geoff Judd
WebSphere MQ Development
IBM
1
© 2009 IBM Corporation
Agenda
Common problems transferring file data
Introduction to MQ Managed File Transfer
IBM’s Managed File Transfer Portfolio
–
2
Introducing IBM Sterling Commerce products
Key MQ Managed File Transfer concepts
Usage scenarios for MQ Managed File Transfer
© 2013 IBM
Corporation
How do most organizations move files today?
Most organizations rely on a mix of homegrown code, several legacy products and
different technologies … and even people!
FTP
– Typically File Transfer Protocol (FTP) is combined with writing and
maintaining homegrown code to address its limitations
Why is FTP use so widespread?
– FTP is widely available – Lowest common denominator
– Promises a quick fix – repent at leisure
– Simple concepts – low technical skills needed to get started
– FTP products seem “free”, simple, intuitive and ubiquitous
Legacy File Transfer products
– A combination of products often used to provide silo solutions
– Often based on proprietary versions of FTP protocol
– Can’t transport other forms of data besides files
– Usually well integrated with B2B but rarely able to work with the rest of
the IT infrastructure – especially with SOA
People
– From IT Staff to Business staff and even Security Personnel
– Using a combination of email, fax, phone, mail, memory keys…
3
© 2013 IBM
Corporation
Shortcomings of Basic FTP
Limited
Reliability
Unreliable delivery – Lacking
checkpoint restart – Files can
be lost
Transfers can terminate
without notification or any
record – corrupt or partial files
can be accidentally used
File data can be unusable
after transfer – lack of
Character Set conversion
Limited
Flexibility
Changes to file transfers often require updates to many
ftp scripts that are typically scattered across machines
and require platform-specific skills to alter
All resources usually have to be available concurrently
Often only one ftp transfer can run at a time
Typically transfers cannot be prioritized
4
Limited
Security
Often usernames and
passwords are sent with file –
as plain text!
Privacy, authentication and
encryption often not be
available
Non-repudiation often lacking
Limited visibility
and traceability
Transfers cannot be monitored and managed centrally
or remotely
Logging capabilities may be limited and may only
record transfers between directly connected systems
Cannot track the entire journey of files – not just from
one machine to the next but from the start of its
journey to its final destination
© 2013 IBM
Corporation
What is WebSphere MQ Managed File Transfer?
Adds managed file transfer capabilities to WebSphere MQ
……
WebSphere MQ Managed File Transfer
A
C
X
Y
Z
Auditable
Full logging and auditing of file transfers + archive audit data to a database
Reliable
Checkpoint restart. Exploits solid reliability of WebSphere MQ
Secure
5
B
Protects file data in transit using SSL. Provides end-to-end encryption using
AMS
Automated
Providing scheduling and file watching capabilities for event-driven transfers
Centralized
Provides centralized monitoring and deployment of file transfer activities
Any file size
Efficiently handles anything from bytes to terabytes
Integrated
Integrates with MB, WSRR, ITCAMs for Apps, DataPower + Connect:Direct
Cost Effective
Reuses investment in WebSphere MQ.
© 2013 IBM
Corporation
A consolidated transport for both files and messages
Traditional approaches to file transfer
result in parallel infrastructures
– One for files – typically built on FTP
– One for application messaging – based on
WebSphere MQ, or similar
High degree of duplication in creating
and maintaining the two infrastructures
Managed File Transfer reuses the MQ
File Transfers
Application
Messaging
network for managed file transfer and
yields:
– Operational savings and simplification
– Reduced administration effort
– Reduced skills requirements and maintenance
Consolidated Transport
for Messages & Files
6
© 2013 IBM
Corporation
Agenda
Common problems transferring file data
Introduction to MQ Managed File Transfer
IBM’s Managed File Transfer Portfolio
–
7
You are here
Introducing IBM Sterling Commerce products
Key MQ Managed File Transfer concepts
Usage scenarios for MQ Managed File Transfer
© 2013 IBM
Corporation
With Sterling Commerce, IBM offers comprehensive MFT Capabilities
Addressing multiple use cases and scenarios for both internal and multienterprise file transfer
WebSphere MQ Managed File Transfer provides file
transfer optimized for data delivery across WebSphere
MQ networks
Sterling Connect Direct provides peer-to-peer file
transfer optimized for data delivery within and between
enterprises across Connect:Direct protocol
Sterling File Gateway provides trading partner onboarding, broad protocol support, management and
visibility
For comprehensive file transfer needs IBM provides
integration between WebSphere MQ Managed File
Transfer, Sterling Connect:Direct, and Sterling File
Gateway
8
© 2013 IBM
Corporation
IBM MFT Vision
Accelerate and simplify governance of the growing volume of businesscritical data movement within and beyond the enterprise with Smarter MFT
Maximize the agility and performance of dynamic business networks by reducing
the complexity, risk, and cost of file transfer
Visibility
Security & Performance
• Single view of transfer activity
• Transaction and business monitoring
• Dashboards, analytics, and scorecards
• Assured delivery and high availability
• Protection of file data in transit and at rest
• Accelerated transport and low latency
Universal
Usability & Management
• Any transport, any protocol, and any
partner
• Global and cloud-enabled deployment
• Broad platform coverage and industry
standards
• Persona-based, easy-to-use interfaces
• Unified control and configuration of
infrastructure
• Community on-boarding and coordination
Connectivity
• Integration with BPM to drive business processes
• Leveraging ESBs to enable service orientation
• SOA Registry/Repository for lifecycle governance
9
© 2013 IBM
Corporation
Agenda
Common problems transferring file data
Introduction to MQ Managed File Transfer
IBM’s Managed File Transfer Portfolio
–
10
Introducing IBM Sterling Commerce products
Key MQ Managed File Transfer concepts
Usage scenarios for MQ Managed File Transfer
You are here
© 2013 IBM
Corporation
Components of a typical WMQ MFT network
Agents
–
Commands
–
Applications exchanging file data
Agent
Agent
Send instructions to agents
A historical record of file
transfers
WebSphere MQ
“Coordination”
Queue Manager
Coordination queue manager
–
Gathers together file transfer
events
Commands
11
Agent
Log database or file
–
The endpoints for managed
file transfer operations
Log database
or file
© 2013 IBM
Corporation
Agents
12
Act as the end points for file
transfers
Long running MQ applications that
transfer files by splitting them into
MQ messages
– Efficient transfer protocol
avoids excessive use of MQ
log space or messages
building up on queues
Multi-threaded file transfers
– Can both send and receive
multiple files at the same time
Generate a log of file transfer
activities which is sent to the
“coordination queue manager”
– This can be used for audit
purposes
Associated with one particular
queue manager (either v6 or v7)
– Agent state on queues
Applications exchanging file data
Agent
Agent
Agent
WebSphere MQ
“Coordination”
Queue Manager
Commands
Log database
or file
There are notes that accompany this slide
© 2013 IBM
Corporation
NOTES
Notes on: Agents
MFT agent processes define the end-points for file transfer. That is to say that if you want to
move files off a machine, or onto a machine – that machine would typically need to be running
an agent process
Agent processes are long running MQ applications that oversee the process of moving file data
in a managed way. Each agent monitors a ‘command’ queue waiting for messages which
instruct it to carry out work, for example file transfers
The MFT agent process needs connectivity to an MQ queue manager to do useful work. It can
connect either directly to a queue manager running on the same system, or as an MQ client
using an embedded version of the MQ client library (which is kept completely separate to any
other MQ client libraries that may or may not already have been installed onto the system)*
– Each agent requires its own set of MQ queues – which means that an agent is tied to the
queue manager where these queues are defined
– However – one queue manager can support multiple agents
* Note: availability of direct (bindings) connectivity or MQ client based connectivity is dependent on the version of MQ MFT in use
13
•
WebSphere MQ Managed File Transfer on z/OS does not support the MQ client style of connectivity
•
Managed File Transfer on distributed platforms has a ‘server’ and ‘client’ offering. The agent component of the ‘client’ offering is restricted to only
supporting MQ client style connectivity. The agent component of the ‘server’ offering may be used either connectivity options
© 2013 IBM
Corporation
Commands
Send instructions to agents and
display information about agent
configuration
–
14
Via MQ messages
Applications exchanging file data
Agent
Agent
Agent
Many implementations of
commands:
–
MQ Explorer plug-in
–
Command line programs
–
Open scripting language
–
JCL
–
Documented interface to
program to
WebSphere MQ
“Coordination”
Queue Manager
Commands
Log database
or file
There are notes that accompany this slide
© 2013 IBM
Corporation
NOTES
Notes on: Commands
15
“Commands” is the name we have given to anything which instructs the MFT agent. As
described on the previous slide, there are a wide range of command implementations including
graphical and non-graphical command-line based commands
Commands instruct the MFT agent by sending it messages. The messages themselves use a
documented format which can easily be incorporated into your own applications
The commands that are supplied with MFT can connect either as an MQ client (again based on
embedded client libraries) or directly to a queue manager located on the same system
© 2013 IBM
Corporation
Log Database & File
Keeps a historical account of transfers that
have taken place
–
Applications exchanging file data
Who, where, when… etc.
Implemented by the ‘logger’ component
Agent
Agent
Agent
which connects to the coordination queue
manager
–
Stand alone application
WebSphere MQ
“Coordination”
Queue Manager
- Can log to database or file
–
Or JEE application
- Can log to database only
Queryable via Web Gateway
Commands
–
16
Also a documented interface
Log database
or file
There are notes that accompany this slide
© 2013 IBM
Corporation
NOTES
Notes on: Log Database
Managed File Transfer can record a historical account of file transfers to a database using the
‘database logger’ component.
–
This component is available to run as a stand alone process or as a JEE application
The information used to populate the log database is generated, as MQ messages, by the MFT
agents participating in file transfers. This is routed to a collection point in the MQ network,
referred to as the ‘coordination queue manager’ (see next slides). The database logger
component subscribes to the messages produced by agents and reliably enters them into a
database.
17
© 2013 IBM
Corporation
Coordination Queue Manager
Gathers together information about
events in the file transfer network
Not a single point of failure
–
Can be made highly available
–
Messages stored + forwarded
Applications exchanging file data
Agent
Agent
Agent
WebSphere MQ
18
“Coordination”
Queue Manager
MQ v7 publish / subscribe
–
Allows multiple log databases,
command installs
–
Documented interface
Commands
Log database
or file
There are notes that accompany this slide
© 2013 IBM
Corporation
NOTES
Notes on: Coordination queue manager
The coordination queue manager is used as the gathering point for all the information about file
transfers taking place between a collection of MFT agents
The queue manager uses publish/subscribe (so it must be MQ version 7) to distribute this
information to “interested parties” which typically include:
–
The WebSphere MQ Explorer plug-in, which provides a graphical overview of MFT activity
–
The database logger component, which archives the information to a database
–
Some of the command line utilities which are part of the MFT product
The format used to publish information is documented and can be used to develop 3rd party
applications which process this data
Although there is only a single ‘coordination’ queue manager for a given collection of agents it
does not represent a point of failure:
19
–
MQ stores and forwards messages to the coordination queue manager when it is available – so if the
coordination queue manager is temporarily unavailable no log data is lost
–
The ‘coordination’ queue manager can be made highly available using standard HA techniques such as
MQ multi-instance queue manager or via a HA product such as PowerHA
© 2013 IBM
Corporation
Agenda
Common problems transferring file data
Introduction to MQ Managed File Transfer
IBM’s Managed File Transfer Portfolio
–
20
Introducing IBM Sterling Commerce products
Key MQ Managed File Transfer concepts
Usage scenarios for MQ Managed File Transfer
You are here
© 2013 IBM
Corporation
Example usage of monitoring + program execution
3. MFT transports file
to destination
Existing
Application
WMQ
MFT
Agent
1. Application writes 2. Agent monitors file
file to file system system, spots arrival
5. MFT can also start another
application to process the file
WMQ
MFT
Agent
Existing
Application
4. At destination MQ MFT
writes file to file system
of file and based on
rules, transfers the file
21
There are notes that accompany this slide
© 2013 IBM
Corporation
NOTES
Notes on: Monitoring & Program Execution
Resource monitors work in two stages:
1. Poll a resource (in this case the file system – but as we’ll see later the ‘resource’ can also be an MQ
queue) and identify that a condition has been met (perhaps the appearance of a file matching a
particular pattern)
2. Perform an action (which can include starting a managed file transfer, or running a script), optionally
propagating information about the resource (for example the name of the file triggered on) into the action
As shown on the previous slide, resource monitors are typically used to provide integration
with an existing system without needing to make changes to the system
Another function of MFT, used for integration with existing systems is the ability to execute
programs or scripts both on the source or destination systems for a file transfer. This can be
used to:
22
–
Start a program, on the source system, which generates the file data to be transferred prior to
performing the managed file transfer
–
Start, or notify, a program on the destination system when the file data has been transferred – allowing it
to process the data without having to poll
© 2013 IBM
Corporation
XML Scripting using Apache Ant
Step 1
Invoke a File Transfer
1
Step 2
If Step 1 completes Ok then
invoke program to process file
2
Step 3
If Step 1 fails then
send an email to the
Administrator
3
© 2013 IBM
Corporation
Protocol Bridging Agents
Support for transferring files located on FTP and SFTP servers
– The source or destination for a transfer can be an FTP or an SFTP server
Enables incremental modernization of FTP-based home-grown solutions
– Provides auditability of transfers across FTP/SFTP to central audit log
– Ensures reliability of transfers across FTP/SFTP with checkpoint restart
Fully integrated into graphical, command line and XML scripting interfaces
– Just looks like another MFT agent…
Files exchanged between MFT and FTP/SFTP
Agent
Agent
FTP/
SFTP
Agent
WebSphere MQ
FTP/
SFTP
Client
Protocol
Bridge
Agent
FTP/
SFTP
Server
FTP/
SFTP
Client
FTP/
SFTP
Client
24
Audit
information
© 2013 IBM
Corporation
WebSphere Message Broker Nodes
Part of
WMB
7.0.0.1
Message Broker
Execution Group
WMQ
MFT
WMQ
Agent
MFT
WMQ
Agent
MFT
Agent
Message Flow
WMQ
MFT
Agent
FTEInput
FTEOutput
FTEInput node
– Build flows that accepts file transfers from the WMQ MFT network
FTEOutput node
– Build flows that are designed to send a file across a WMQ MFT network
When WMQ MFT nodes are used in a flow an MFT agent is automatically
25
started in the Message Broker Execution Group
© 2013 IBM
Corporation
Integration with IBM Sterling Connect:Direct
Agent
Available
2Q2011
C:D
Node
Agent
Agent
WebSphere MQ
C:D
Bridge
Agent
C:D
Node
Reference
MFT
Audit
C:D
Node
C:D
Node
C:D
Audit
The Connect:Direct Bridge capability supports
managed file transfers that span MFT and C:D
Trading Partner
Inside the MFT audit trail…
The audit information for each MFT transfer
references related C:D audit information
with a joined up audit trail
26
© 2013 IBM
Corporation
Interoperation with DataPower B2B Appliance XB60
Documented and tested configurations for integrating with DataPower Appliances
– WebSphere DataPower XB60 B2B Appliance – for B2B connectivity
– WebSphere DataPower IX50 Integration Appliance – for ESB connectivity
Enables sending files to trading partners over a range of protocol transports
– via DataPower Appliances acting as B2B gateways
Multi-protocol transfers to B2B trading partners
Company A
Agent
Agent
WebSphere MQ
Agent
XB60
SFTP
AS2
Agent
Internal Network
27
HTTPS
Agent
Company B
Company C
… etc
…etc
DMZ
Internet
Trading Partner
© 2013 IBM
Corporation
Securing file data with SSL and WMQ AMS
WMQ MFT supports transport
Agent
svrconn WebSphere
channel
MQ
sndr/rcvr
channels
Queue
Manager
WebSphere
MQ
Queue
Manager
level encryption using SSL
Agent
Data is encrypted before it is
sent over a channel and
decrypted when it is received
When combined with WMQ
Agent
svrconn WebSphere
channel
MQ
Queue
Manager
sndr/rcvr
channels
WebSphere
MQ
Queue
Manager
Advanced Message Security
Agent
– Allows file data to be encrypted
at the source system and only
decrypted when it reaches the
destination system
– Data is secure even when at
rest on a queue
28
© 2013 IBM
Corporation
Staged migration to messaging
Pain-point:
– Hard to migrate to an event driven architecture as lots of applications communicate
by transferring files
Managed File Transfer Helps:
– Deliver files as message payloads and vice versa
– Monitor queues and transfer message payloads to files
WebSphere
MQ Managed
File Transfer
29
© 2013 IBM
Corporation
Options for converting data between files and messages
One file to one message
WMQ
MFT
One file to a group of messages
WMQ
MFT
One file becomes one
message
The file can be split based on:
–
Size
–
Binary delimiter
–
Regular expression
One message to one file
WMQ
MFT
A group of messages (or all messages on the queue) to one file
WMQ
MFT
One message becomes one
file
Optionally, a delimiter can be
inserted between each
message used to compose the
file
30
© 2013 IBM
Corporation
Monitoring queues for the arrival of messages
The WMQ MFT agent can monitor queues
for the arrival of messages, then perform an
action, such as transferring the payload fro
Remember we said MFT
can monitor for files arriving…
Existing
Application
the messages as a file (as per the previous
WMQ
MFT
Agent
slide)
Conditions that can be monitored for:
– Queue not empty
Well, it can also monitor for
messages arriving on a queue…
– Complete group of messages
Existing
Application
31
WMQ
MFT
Agent
© 2013 IBM
Corporation
Historical Roadmap
WebSphere MQ V7.5
WebSphere MQ File Transfer Edition
became
WebSphere MQ Managed File Transfer
FTE V7.0.2
FTE V7
Reliable, managed file transfer
Remote management and audit
Core platforms including z/OS
Zero programming needed
Command line and GUI interfaces
File auditing across backbone
Globalization
Q4 2008
4Q 2010
Q4 2009
FTE V7.0.1
Archive transfer audit log to external DB
Enhanced directory monitoring
ANT XML Scripting of multi-step transfer jobs
Enhanced transfer request error reporting
Unattended silent install
Support for zLinux
Enhanced z/OS Performance, Tape, GDG
FTE V7.0.4
Integration with existing IBM
Sterling Connect:Direct networks
Bridge to transfer files into out of
existing C:D networks
Enhanced Explorer tooling with
visibility of file transfers sent
into/out of C:D networks
Q2 2009
32
Ability to bridge to FTP networks
iSeries® support
Initial DataPower XB60 integration
Consumeability enhancements
Security enhancements
2Q 2012
2Q 2011
WMB 7.0.0.1
FTE input
and output
nodes
FTE V7.0.3
Web browser (ad hoc) file transfers
Convert payloads between files and messages
Automatically start agents and DB logger on Windows
View deployed agents in MQ Explorer (GUI)
End-to-end encryption using WebSphere MQ AMS
Extensions to platform support
© 2013 IBM
Corporation
Resources
Information Center:
–
http://publib.boulder.ibm.com/infocenter/wmqfte/v7r0/index.jsp (pre-V7.5)
–
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r5/index.jsp (7.5 onwards)
Redbooks / Redguides / Redpapers:
–
Getting Started with WebSphere MQ Managed File Transfer V7
•
–
IBM WebSphere MQ Managed File Transfer Solution Overview
•
–
http://www.redbooks.ibm.com/abstracts/redp4533.html
B2B Enabled Managed File Transfer using WebSphere DataPower B2B Appliance XB60 and
WebSphere MQ Managed File Transfer
•
–
http://www.redbooks.ibm.com/abstracts/redp4532.html
Managed File Transfer for SOA using IBM WebSphere MQ Managed File Transfer
•
–
http://www.redbooks.ibm.com/abstracts/sg247760.html
http://www.redbooks.ibm.com/abstracts/redp4603.html
IBM Sterling Managed File Transfer Integration and WebSphere Connectivity for a Multi-Enterprise
Solution
•
http://www.redbooks.ibm.com/redpieces/abstracts/sg247927.html
Trial Download:
–
http://www.ibm.com/software/integration/wmq/filetransfer/
Early Design Program
– Interested in participating in the development of future versions of MFT?
33
• Ask your local IBM representative to nominate you for the MFT EDP program
© 2013 IBM
Corporation
Any Questions?
■
34
© 2013 IBM
Corporation