MobiStore - Mobile internet
File Storage Platform
Chetna Kaur
Proposed Work
Develop a distributed internet storage platform that
will allow internet enabled devices to manage
storage over the internet.
What is an Internet File System
User’s view

Storage provided on the Internet.

Can access the data stored in the File System on the go.

No need to buy expensive memory.

Seamless integration into the users file system eg: Gmail Drive’
Technically,




Not a Real File System
Simulates a file system by having a central repository of folder and file
links.
Files are stored on multiple storage servers.
API’s are provided for users to log in, create and manage files and
folders.
Motivation - Why another Internet File System?




# of mobile internet users increasing exponentially.
Mobile clients have limited memory but growing
memory requirements. – Potential User Base
Current internet file systems cannot cater to these
clients due to client resource constraints (Memory,
Bandwidth).
Why? What are the issues involved? How can I
build a file system platform that can provide
storage services to all types of clients?
Other sources of motivation

Show Gmail drive demo here
Solution
The Block Level Meta File System
-
-
-
Files are stored and accessed at a block
level.
Allows users with limited memory and
bandwidth requirements to access large
files.
File is downloaded in chunks as needed.
Challenges

Where Will Data Be Stored?
Requiring lot’s of space is the easy part of the problem!
 Availability
 Scalability
 Privacy and security
 Efficient data updates

Storage Access and Management Problems




Customizing system to users’/applications’ needs
Designing and managing block level meta data
Adaptivity to existing or new applications
 How do you share storage on top of blocks?
Communication between components
Important Design Decisions



System Interface
Scalable and High Availability Architecture
Data Structures




What Meta-Data to capture?
 Fast Indexing
 User Based Views
 Ease of file sharing
Efficient Algorithms for creation of new files and allocation
of new data blocks
File caching strategy
Technology
System Design
Meta File System Server
WWW
Distributed Storage Server
Access Server
System Interface and Functionality




Simple API for user login (password based
authentication).
APIs to create, and delete files and directories.
APIs to read and write to files at a block level
granularity.
API to share files with friends
Access Server - Functionality


Users connect to the file system and access their files via the
Access Server.
Client requests for File Block.





The Access Server gets the file information from the MFS Server.
Checks if file available in Cache.
If not, it connects to the storage server fetches the first block from
the SS.
Delegates the task of fetching rest of the blocks to cache
manager.
Send the requested data block to the client.
Key Features





Asynchronous Communication
Multithreaded – New thread for every connection
(uses a thread pool).
Block Level File Caching and Pre-Fetching.
Cache Manager runs in separate thread, prefetches next few blocks.
Cache Replacement Policy – Least Recently Used.
Storage Server



Essentially a flat file store.
Each file can be distributed across several actual
files on one or more storage servers.
How then can we allocate new files and blocks on
storage servers?
 Handled
by the Meta File System Server.
Meta File System



Stores Meta-data related to users and files
Uses hash maps for quick indexing of user name
spaces and files.
Efficient allocation of new blocks and files to
storage servers.
 Tries
to first allocate contiguous blocks so that they can be
pre-fetched together.
 If not possible, tries to allocate all blocks on same storage
server.
Design of the Meta File System
Main Contributions





Propose a Scalable, Highly-Available Architecture for a Distributed Internet File
System.
Design and implement an efficient Meta-File structure for a block level file
system implementation.
Implemented a simple file caching and pre-fetching mechanism, to improve
efficiency of block level file access.
Design and implement APIs for accessing files block by block for client with
resource constraints.
Implement a proof of concept system, and built a basic web based file storage
application on this platform to demonstrate its capabilities.
Open Research Areas





Persistence of Meta File System structure
Improved File Caching Mechanism in the Access
Server
Test the performance of the Block Allocation
Algorithm and improve it.
Towards a Google Docs type of implementation
(Multiple users can write different blocks of data
simultaneously)
Handling Multimedia Files
Technologies Used

+ Front End Web Application
PHP
+ Server
Apache, C++, Adaptive Communication
Environment, STL/BOOST
+ Communication
JSON over HTTP
Questions?