User Guider
Secure Cloud Data Migrator (SCDM)
Trend Micro | 10101 N. De Anza Blvd. | Cupertino, CA 95014| ph:408.257.1500 | toll free: 800.228.5651
www.trendmicro.com
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
Abstract
SCDM is to migrate the on-premise DC files and Mysql Data to AWS S3 and
RDS with efficient method.
This document briefly introduces the principle, installation, features and
trouble shootings.
Note
It is an evaluation purpose solution; the component may be updated
frequently. Some items in the UI may be changed as this document
descripted, so please track the publish web to get the latest information.
Revision History
Revision
1
2
3
4
Date
6/3/2011
6/25/2011
Author
Page 2 of 16
Comment
create
Update the changes
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
Contents
Abstract ............................................................................................. 2
Note .................................................................................................. 2
Revision History ................................................................................ 2
1 Introduction ................................................................................ 4
2 Installation.................................................................................. 4
2.1
Launch EC2 instance inherited from predefined AMI ...................... 5
2.1.1
Choose the right AMI. ......................................................... 5
2.1.2
Select the instance type=C1.medium .................................... 5
2.2
Generate SSL key pair for client tools and configure the EC2 instance
5
2.2.1
Assign an Elastic IP for this instance (optional) ...................... 5
2.2.2
Login in to EC2 instance with root account and configure SSL
client certification ........................................................................... 5
2.3
Install the client tool (Data Extractor) and Mysql client.................. 7
2.4
Configure SCDM client tool and do simple test. ............................ 8
3 Feature introduction .................................................................. 11
3.1
File Upload ............................................................................. 11
3.2
DB Upload .............................................................................. 12
4 Troubleshooting ........................................................................ 14
4.1
Networking:............................................................................ 14
4.2
Error Handing ......................................................................... 15
4.2.1
Networking disconnect between AWS and your DC ................ 15
4.2.2
Data conflict in Mysql DB data migration .............................. 15
4.2.3
Some manual actions cause the unexpected result. ............... 15
5 Conclusion & feedback .............................................................. 16
Page 3 of 16
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
1 Introduction
SCDM is a point to point cloud data migration solution, which integrates the
openSSL to secure the data transfer over WAN.
Currently, it is as an evaluation tool to support the data (files and MySQL
data) from on-premises DC to Amazon Web Service data storage service (S3
and RDS).
There are two major components in this solution –Client Data Extractor and
AWS predefined EC2 instance. Secure & high efficient protocol is to improve
the transfer efficiency between the client and AWS EC2 instance over WAN.
Client data extractor is installed into on-premises cloud instance, and
connects on-premise DC’s data over intranet. User can upload files or Mysql
data with SCDM client UI.
Predefined EC2 instance as a data receiver is installed in customer AWS
environment and created from predefined AMI, which receives the data from
data extractor and save the data into S3 or RDS (bases on the data type).
2 Installation
Before installation, please confirm following information:
1. Should have an Amazon account and knowledge to use AWS service,
especially on EC2 instance, S3, RDS and so on.
Trend Micro | 10101 N. De Anza Blvd. | Cupertino, CA 95014| ph:408.257.1500 | toll free: 800.228.5651
www.trendmicro.com
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
1
2
3
4
5
6
2. Your on-premise DC real WAN upload throughput > 10Mbps will be
better, or you may not get the obvious benefits to compare the
Amazon official solution.
3. It is an evaluation purpose, and you should read and accept license
and agreement in the client installation process.
There are four steps to install the evaluation solution.
7
2.1 Launch EC2 instance inherited from predefined AMI
8
9
10
11
12
13
2.1.1 Choose the right AMI.
Choose the community AMI in US East or US West to create EC2 instance
(instance name is US_West_SCDM_Evaluation and
US_East_SCDM_Evaluation). The AMI ID may be changed in the final version,
but please double check the right AMI ID from final publish web
14
15
16
17
2.1.2 Select the instance type=C1.medium
18
19
20
Other steps are same as launching instance.
21
22
2.2 Generate SSL key pair for client tools and configure the
EC2 instance
23
2.2.1 Assign an Elastic IP for this instance (optional)
24
25
26
27
28
2.2.2 Login in to EC2 instance with root account and configure SSL
client certification
Entry the path of “/home/SDTHome”.
Page 5 of 16
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
1
2
3
4
5
6
7
8
9
10
11
12
Generate the client Key with “SDTKeyGen”
“clientCert.pem” and “clientKey.pem” will be existed in “/home/SDTHome/”
after running SDTKeyGen.
Download the “clientCert.pem” and “clientKey.pem” to local machine for
client tool.
For example (putty method):
Page 6 of 16
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Download the client setup file (SDTSetup.msi and setup.exe) with same
method.
Then, configure the EC2 instance via SDTS3Cfg and reboot EC2 instance.
Note:
Before run the SDTS3Cfg, you should prepare your S3 bucket (bucket name
shouldn’t have blank char and only support lower case letters), access key
and secret key.
After that, you can run the SDTS3Cfg directly, and reboot the instance after
configure successfully, for example:
15
16
17
2.3 Install the client tool (Data Extractor) and Mysql client
18
19
20
21
22
23
24
25
26
Note:
1) Before install client tool, please make sure your computer has
installed .NET Framework 3.5 or later version.
2) For the windows UAC issue, client tool may don’t have permission to
access their files. This suggests you don’t change the default installation
path.
Install Mysql client is to support the Mysql data migration, which isn’t impact
the file migration to S3.
Page 7 of 16
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
1
2
Mysql client installation please refers to the official Mysql step, and must
select the “Include Bin Directory in Windows Path”, for example:
3
4
5
6
7
8
9
Entry into the download folder and click the “setup.exe” to install the “Secure
Cloud Data Migrator”, the step is same as normal windows software
installation. You should read and accept the content in the license and
agreement. At the beginning of installation, it may pop up a warning, please
click the yes to continue.
10
11
12
13
14
After successful installation, your desktop will have a shortcut named” Secure
Cloud Data Migrator”, which is a link to the main procedure.
15
16
17
18
2.4 Configure SCDM client tool and do simple test.
Double click the desktop shortcut of “secure Cloud Data Migrator”, select the
setting tab, and begin the client configuration.
Page 8 of 16
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
At first, select your downloaded Key and cert (“clientKey.pem” and
“clientCert.pem”)
Set the S3 bucket name as your filled in the EC2 instance for client display
using.
Set the right AWS EC2 IP address or public DNS name.
About the bandwidth limitation, all bases on your needs.
Others please don’t change in this version.
Then click the Save All button.
16
17
Page 9 of 16
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Before test the tool, please make sure your network is allowed to access
AWS EC2 instance with UDP: 9000, and allowed to get the response from
that EC2 instance with UDP: 9000. Both send and receive should be allowed
in the test. (Source and destination for EC2 instance UDP: 9000 should be
allowed)
Then you can test the networking status between client tool and AWS EC2
instance in File Upload tag.
Click the files button to select any files, and then click the start button to test
the network status (make sure your selected file path only support English
letters and don’t include blank in the file path).
Successfully status:
Failed status:
Page 10 of 16
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
1
2
3
4
5
6
7
8
9
10
11
12
13
14
If there appears the init error as above information, pls check following
information:
1. Your AWS EC2 instance is running and the UDP port 9000 is listening.
2. Your intranet is allowed to connect the AWS EC2 instance with UDP 9000
and receive the response from that.
3. Your Setting tag fills the right content, and has saved the information?
4. Please turn off the windows firewall, or check your local additional firewall
to allow UDP connection.
5. More info please refers to troubleshooting session or contacts us.
3 Feature introduction
15
3.1 File Upload
16
17
File upload is to migrate the local files to AWS S3.
Page 11 of 16
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Warning:
Currently, because of an open source tool’s compatible issue (s3fs), the
directory upload isn’t supported in this version. So directory related feature is
disabled in this version.
If you want to upload some files at a time, please copy the files into one
folder, and select all of them to upload.
3.2 DB Upload
DB upload supports the Mysql database upload (table scheme, table data),
But destination database should be exist at first (or test button will be failed).
Meanwhile, this tool supports the table and row level data migration for your
actual needs.
Before using this tool, you should check the network status from AWS EC2
instance to Mysql database service. You can login into AWS EC2 instance,
and use Mysql client to verify it.
Page 12 of 16
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
1
2
3
4
5
6
7
8
9
10
11
At first, fill the right source DB information, click the
to list all
the tables of the database, and then the tables will be displayed in the left
list view.
Choose the upload tables and set the temp local backup path (for manual
operation when error occurred, and the
is an optional for
user to clean the temp backup, at the same time the option also impact the
server temp backup in S3).
12
13
: Automatically select all tables in this database; the row filter
feature is disabled at this time.
14
There are four options for user to use:
: Automatically create the table schema before data migration.
15
16
17
: By default, when table’s migration is started, migration tool
will lock this table. If check this option, migration process will unlock the
table.
18
19
20
21
22
23
24
25
26
: It is a faster model to support many tables migration. By
default, migration tool will migrate the table one by one. If user check this
option, all tables data in this database will be merged together to migrate.
We suggest user use this option to improve the efficiency, especially for
many tables migration.
The next is to configure the destination DB information. At first, you should
fill the right information of AWS Mysql database.
Page 13 of 16
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
1
Then click the
2
3
4
5
This is the successful result, or there is something wrong in your
configuration and pop up message will display the error message for your
reference. Such as:
6
7
8
9
10
button to wait for the response.
Warning:
As previous warning, there is the compatible issue with open source module
(s3fs), the directory isn’t supported in this version, so please don’t fill any
11
12
13
content in
disabled in this version.
14
15
16
17
18
19
20
21
22
After above configuration is right, you can click the
button
to begin the migration process. However, above process’s success can’t make
sure your migration process will be successful.
23
24
25
26
27
28
29
, and it is
Database migration leverages your understanding on database, tables,
schemas and data. User should understand if migration is reasonable at first.
For example, if you want to migrate several rows (with row filter) from local
Mysql to AWS RDS, you should make sure the primary key isn’t conflict, or
you will be failed at last.
4 Troubleshooting
4.1 Networking:
Secure cloud data migrator is anther channel to support on-premise DC data
to Amazon storage, and it leverages the EC2 virtual instance.
So the networking status from on-premise DC to Amazon storage should be
smooth.
Page 14 of 16
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
1
2
3
4
5
6
7
8
9
1. Between AWS EC2 instance and on-premises DC is a Two-way UDP
communication. The Server Port is 9000.
2. Because it is a secure channel via Openssl, and server should verify
the client certification. So client should get the right Cert & Key for
server to verify, and the Cert & Key should be generated from AWS
EC2 instance inherited from predefined AMI.
3. The network status between AWS EC2 instance and AWS storage (S3
or RDS) refer to the AWS guider.
4.2 Error Handing
10
11
12
The tool will handle some errors in the transfer process. However, it isn’t
designed as final product to trail, so some error should be handled by user
self.
13
14
15
16
17
18
19
20
21
22
23
24
25
26
4.2.1 Networking disconnect between AWS and your DC
For file upload, the client will try to reconnect the AWS EC2 instance for 6
times (around 10 minutes) if reconnect successfully, the data migration will
continue with the previous position. Or client will abort this time file transfer.
27
28
29
30
31
32
33
34
35
36
37
4.2.2 Data conflict in Mysql DB data migration
DB data migration will be more complex than file upload; user should have
db migration knowledge before doing this migration, such as key conflict,
table lock issue, schema issue and so on.
38
39
40
4.2.3 Some manual actions cause the unexpected result.
Some unexpected actions may cause the unknown result, but user can refer
to following suggestions to do trouble shooting.
If user manually cancel the file upload, please wait more than 1 minute to restart the file upload process, because this file is locked in the server for 1
minute to wait for the old session timeout.
About DB upload, it doesn’t try to re-connect, user should base on the temp
storage data to manually handle the error via file upload tag and Mysql client
tool. However, this protocol will support around 1 minute’s network
interruption over the WAN.
Actually, SCDM’s DB upload leverages DB’s feature to support that in this
version (However, it isn’t limit to Mysql database from the principle). SCDM
uses DB client to export the data to local file storage, and through secure
channel to upload the file to S3, and then import the data to RDS. Saving
the temp data to local and S3 storage is the consideration for security,
because this tool doesn’t provide some error handling options for user.
Page 15 of 16
Secure Cloud Data Migrator (SCDM)
Date printed: 3/7/2016
1
2
4.2.3.1
AWS EC2 Instance
Make sure following procedures are running.
3
4
5
If above two procedure aren’t running in the EC2 instance, please verify your
installation process at first, or contact us.
6
7
4.2.3.2
Networking issue
Refer above networking troubleshooting information.
8
9
10
11
12
13
14
15
4.2.3.3
Poor performance
From our testing, 10Mbps WAN upload throughput, RTT~=270ms networking
(from China to AWS US East), the file upload speed is around 6Mbps. From
US DC to AWS US East (RTT~=87ms), file upload will be >130Mbps real
upload speed.
16
17
18
4.2.3.4
Other problems
If other field related problem please read this document at first to make sure
the installation and configuration is right. Then contact us to get help.
19
20
21
22
23
24
25
26
27
28
29
30
31
So if poor performance appears, please check your real network throughout
and any other reason limit the performance (disk IO or network control)
5 Conclusion & feedback
This tool is for evaluation purpose; you can free to use and study. Meanwhile
we strongly hope you can send us your feedback at last with following
schema.
1.
2.
3.
4.
Why use this tool?
Benefits from this tool?
What should to be improved?
Other suggestions…
Feedback Email Account: cn_research@trendmicro.com.cn
Page 16 of 16