The Hadoop Distributed File System: Architecture and Design by Dhruba Borthakur Presented by Bryant Yao Introduction What is it? It’s a file system! ◦ Supports most of the operations a normal file system would. Open source implementation of GFS (Google File System). Written in Java Designed primarily for GNU/Linux ◦ Some support for Windows Design Goals HDFS is designed to store large files (think TB or PB). HDFS is designed for a computer cluster/s made up of racks. Rack 1 Rack 2 Cluster Write once, read many model ◦ Useful for reading many files at once but not single files. Streaming access of data ◦ Data is coming to you constantly and not in waves Make use of commodity computers Expect hardware to fail “Moving computation is cheaper than moving data” Master/Slave Architecture Namenode Datanodes Master/Slave Architecture cont. 1 master, many slaves The master manages the file system namespace and regulates access to files by clients. Data distributed across slaves. The slaves store the data as “blocks”. What is a block? ◦ A portion of a file. ◦ Files are broken down into and stored as a sequence of blocks. File 1 A B C Broken down into blocks A, B, and C. Task Flow Namenode Master Handles metadata operations ◦ Stored in a transaction log called EditLog Manages datanodes ◦ Passes I/O requests to datanodes ◦ Informs the datanode when to perform block operations. ◦ Maintains a BlockMap which keeps track of which blocks each datanode is responsible for. Stores all files’ metadata in memory ◦ File attributes, number of replicas, file’s blocks, block locations, and checksum of a block. Stores a copy of the namespace in the FsImage on disk. Datanode Slave Handles data I/O. Handles block creation, deletion, and replication Local storage is optimized so files are stored over multiple file directories ◦ Storing data into a single directory Data Replication Makes copies of the data! Replication factor determines the number of copies. ◦ Specified by namenode or during file creation Replication is pipelined! Pipelining Data Replication Blocks are split into portions (4KB). 1 2 3 Assume a block is split into 3 portions: A, B, and C. A 1 B 2 3 A 1 C 2 B 3 A Replication Policy Communication bandwidth between computers in a rack is greater than between a computer outside of the rack. We could replicate data across racks…but this would consume the most bandwidth. We could replicate data across all computers in a rack…but if the rack dies we’re in the same position as before. Replication Policy cont. Assume only three replicas are created. ◦ Split the replicas between 2 racks. ◦ Rack failure is rare so we’re still able to maintain good data reliability while minimizing bandwidth cost. Version 0.18.0 ◦ 2 replicas in current rack (2 different nodes) ◦ 1 replica in remote rack Version 0.20.3.x ◦ 1 replica in current rack 2 replicas in remote rack (2 different nodes) What happens if replication factor is 2 or > 3? ◦ No answer in this paper. ◦ Some other papers state that the minimum is 3. ◦ The author wrote a separate paper stating every replica after the 3rd is placed randomly. Reading Data Read the data that’s closest to you! ◦ If the block/replica of data you want is on the datanode/rack/data center you’re on, read it from there! Read from datanodes directly. ◦ Can be done in parallel. Namenode is used to generate the list of datanodes which host a requested file as well as getting checksum values to validate blocks retrieved from the datanodes. Writing Data Data is written once Split into blocks, typically of size 64MB ◦ The larger the block size, the less metadata stored by the namenode Data is written to a temporary local block on the client side and then flushed to a datanode, once the block is full. ◦ If a file is closed while the temporary block isn’t full, the remaining data is flushed to the datanode. If the namenode dies during file creation, the file is lost! Hardware Failure Imagine a file is broken into 3 blocks spread over three datanodes. 1 Block A 2 Block B 3 Block C If the third datanode died, we would have no access to block C and we can’t retrieve the file. 1 Block A 2 Block B 3 Block C Designing for Hardware Failure Data replication Safemode ◦ Heartbeat ◦ Block report Checkpoints Re-replication Checkpoints EditLog + = File System Namespace FsImage Checkpoints FsImage is a copy of the system taken before any changes have occurred. EditLog is a log of all the changes to the namenode since it’s startup. Upon the start up of the namenode, it applies all changes to the FsImage to create an up to date version of itself. ◦ The resulting FsImage is the checkpoint. If either the FsImage or EditLog is corrupt, the HDFS will not start! Heartbeat and Blockreport A heartbeat is a message sent from the datanode to the namenode. ◦ Periodically sent to the namenode, letting the namenode know it’s “alive.” ◦ If it’s dead, assume you can’t use it. Blockreport ◦ A list of blocks the datanode is handling. Safemode Upon startup, the namenode enters “safemode” to check the health status of the cluster. Only done once. Heartbeat is used to ensure all datanodes are available to use. Blockreport is used to check data integrity. ◦ If the number of replicas retrieved is different from the number of replicas expected, there is a problem. Replicated A A A Found A A Other Can view file system through FS Shell or the web Communicates through TCP/IP File deletes are a move operation to a “trash” folder which auto-deletes files after a specified time (default is 6 hours). Rebalancer moves data from datanodes which have are close to filling up their local storage. Relation with Search Engines Originally built for Nutch. ◦ Intended to be the backbone for a search engine. HDFS is the file system used by Hadoop. ◦ Hadoop also contains a MapReducer which has many applications, like indexing the web! Analyzing large amounts of data. Used by many, many companies ◦ Google,Yahoo!, Facebook, etc. It can store the web! ◦ Just kidding . “Pros/Cons” The goal of this paper is to describe the system, not analyze it. It gives a great beginning overview. Probably could’ve been condensed/organized better. Some information is missing ◦ SecondaryNameNode ◦ CheckpointNode ◦ Etc. Pros/Cons of HDFS In and Beyond the Paper Pros ◦ ◦ ◦ ◦ It accomplishes everything it set out to do. Horizontally scalable – just add a new datanode! Cheap cheap cheap to build. Good for reading and storing large amounts of data. Cons ◦ Security ◦ No redundancy of namenode Single point of failure ◦ The namenode is not scalable ◦ Doesn’t handle small files well ◦ Still in development, many features missing Questions? Thank you for listening!