DIFFSYNC
A file synchronization program
CLASS PROJECT – EEL6883
BY
ADAM RADWAN
GARGI CHIPALKATTI
FILE SYNCHRONIZATION





Process of making sure that two or more locations
contain the same up-to-date files
One-way (Mirroring) or two-way file synchronization
Useful for mobile workers, multiple computer users;
used for backup purposes
Version control tools and File synchronization tools
Various existing solutions, none of which were found
by OpenPublic Networks, LLC to be robust enough
for production use in a distributed environment
GOALS




Develop a file synchronization program which is
capable of working over a diverse range of
geographical placement
Must be capable of running securely over common
public network backbone, such as the Internet
Capable of two-way synchronization of files for
changes to served web pages
Ability for a non-functional server to gracefully
degrade while other servers detect this and recover
appropriately to keep services running
RSYNC




Open Source software for Unix systems
Mirroring takes place with only one transmission in
each direction
Used to synchronize local directories or directories
via a remote shell such as RSH or SSH
Two checksums: a weak “rolling“ 32-bit checksum
and a strong 128-bit MD4 checksum
UNISON





Runs on Unix-like operating systems (Linux, Mac OS
X, and Solaris), and on Windows
Can synchronize files across platforms
Detects 'conflicts' where a file has been modified on
both sources, and displays these to the user
Uses the rsync algorithm
Designed to be robust in the event of a program or
system crash or a communication failure
DIFFSYNC






Uses open source software – rsync, inotify
Uses PHP scripting language
Built on top of a MySQL 5.1 triple-master replicated
database system running in chrooted jail for security
Custom SSH-based VPN for three independent pointto-point TUN links between the servers
Apache-based web server for verification running in
chrooted jail for security
phpMyAdmin used for database management
DIFFSYNC FOR THREE SERVERS
Podice
Location: Fayetteville, NC
IP: 216.117.160.235
CPU: Pentium 4 2.8 Ghz
Storage: 80GB, 512 MB
OS: Gentoo Linux
Akimbo
Location: Orlando, FL
IP: 72.189.195.130
CPU: Pentium III 800 Mhz
Storage: 20GB, 128 MB
OS: Gentoo Linux
Philomena
Location: Ft. Lauderdale, FL
IP: 65.111.166.49
CPU: AMD 2000+
Storage: 40GB, 256 MB
OS: Gentoo Linux
CREATION OF DIRECTORY & FILE
~/TestArea $ mkdir 1
~/TestArea $ nano ./1/X
Array
(
[0] => Array
(
[dir] => ./TestArea/
[event] => CREATE,ISDIR
[file] => 1
)
[1] => Array
(
[dir] => ./TestArea/
[event] => OPEN,ISDIR
[file] => 1
)
[2] => Array
(
[dir] => ./TestArea/
[event] => CLOSE_NOWRITE,CLOSE,ISDIR
[file] => 1
)
)
Array
(
[0] => Array
(
[dir] => ./TestArea/1/
[event] => CREATE
[file] => X
)
[1] => Array
(
[dir] => ./TestArea/1/
[event] => OPEN
[file] => X
)
[2] => Array
(
[dir] => ./TestArea/1/
[event] => MODIFY
[file] => X
)
[3] => Array
(
[dir] => ./TestArea/1/
[event] => CLOSE_WRITE,CLOSE
[file] => X
)
)
MOVE DIRECTORY & FILE
~/TestArea $ mv ./1/X ./1/Y
~/TestArea $ mv ./1 ./2
Array
(
[0] => Array
(
[dir] => ./TestArea/1/
[event] => MOVED_FROM
[file] => X
)
[1] => Array
(
[dir] => ./TestArea/1/
[event] => MOVED_TO
[file] => Y
)
Array
(
[0] => Array
(
[dir] => ./TestArea/
[event] => MOVED_FROM
[file] => 1
)
[1] => Array
(
[dir] => ./TestArea/
[event] => MOVED_TO
[file] => 2
)
)
)
ATTRIB DIRECTORY & FILE
~/TestArea $ chmod 777 ./1
~/TestArea $ chmod 777 ./1/X
Array
(
[0] => Array
(
[dir] => ./TestArea/
[event] => ATTRIB
[file] => 1
)
)
Array
(
[0] => Array
(
[dir] => ./TestArea/1/
[event] => ATTRIB
[file] => X
)
)
DELETE DIRECTORY
~/TestArea $ rm –r ./1
Array
(
[0] => Array
(
[dir] => ./TestArea/
[event] => OPEN,ISDIR
[file] => 1
)
[1] => Array
(
[dir] => ./TestArea/1/
[event] => OPEN,ISDIR
[file] =>
)
[2] => Array
(
[dir] => ./TestArea/1/
[event] => DELETE
[file] => X
)
[3] => Array
(
[dir] => ./TestArea/
[event] => CLOSE_NOWRITE,CLOSE,ISDIR
[file] => 1
)
[4] => Array
(
[dir] => ./TestArea/1/
[event] => CLOSE_NOWRITE,CLOSE,ISDIR
[file] =>
)
[5] => Array
(
[dir] => ./TestArea/
[event] => DELETE,ISDIR
[file] => 1
)
[6] => Array
(
[dir] => ./TestArea/1/
[event] => DELETE_SELF
[file] =>
)
[7] => Array
(
[dir] => ./TestArea/1/
[event] => IGNORED
[file] =>
)
)
DELETE FILE
~/TestArea $ rm ./1/X
Array
(
[0] => Array
(
[dir] => ./TestArea/1/
[event] => DELETE
[file] => X
)
)
EDIT FILE
~/TestArea $ nano ./1/X
Array
(
[0] => Array
(
[dir] => ./TestArea/1/
[event] => OPEN
[file] => X
)
[1] => Array
(
[dir] => ./TestArea/1/
[event] => ACCESS
[file] => X
)
[2] => Array
(
[dir] => ./TestArea/1/
[event] => CLOSE_NOWRITE,CLOSE
[file] => X
)
)
Array
(
[0] => Array
(
[dir] => ./TestArea/1/
[event] => MODIFY
[file] => X
)
[1] => Array
(
[dir] => ./TestArea/1/
[event] => OPEN
[file] => X
)
[2] => Array
(
[dir] => ./TestArea/1/
[event] => CLOSE_WRITE,CLOSE
[file] => X
)
)
COPY FILE
~/TestArea $ cp ./1/X ./1/Y
Array
(
[0] => Array
(
[dir] => ./TestArea/1/
[event] => OPEN
[file] => X
)
[1] => Array
(
[dir] => ./TestArea/1/
[event] => CREATE
[file] => Y
)
[2] => Array
(
[dir] => ./TestArea/1/
[event] => OPEN
[file] => Y
)
)
Array
(
[3] => Array
(
[dir] => ./TestArea/1/
[event] => ACCESS
[file] => X
)
[4] => Array
(
[dir] => ./TestArea/1/
[event] => MODIFY
[file] => Y
)
[5] => Array
(
[dir] => ./TestArea/1/
[event] => CLOSE_WRITE,CLOSE
[file] => Y
)
[6] => Array
(
[dir] => ./TestArea/1/
[event] => CLOSE_NOWRITE,CLOSE
[file] => X
)
)
DIFFICULTIES ENCOUNTERED





Must have a kernel module installed on the host
server for inotify events to be recorded
Problems detecting changes made to newly-created
directories within the specified inotify watch point
MySQL replication is sensitive to binary log changes
when one node is disconnected
Accurate prediction and parsing of inotify events for
complete analysis of changes made to filesystem
Not enough time in the semester to complete
integration of all features into program
CONCLUSION

Fundamentals of DiffSync program prove that file
synchronization is possible for servers in remote
locations

Additional work will be completed to make this
program ready for a production environment

Any questions?
THE END
THANK YOU!