Wednesday, November 15, 2023

NameNode, DataNode And Secondary NameNode in HDFS

HDFS has a master/slave architecture. With in an HDFS cluster there is a single NameNode and a number of DataNodes, usually one per node in the cluster.

In this post we'll see in detail what NameNode and DataNode do in Hadoop framework. Apart from that we'll also talk about Secondary NameNode in Hadoop which can take some of the work load of the NameNode.

NameNode in HDFS

The NameNode is the centerpiece of an HDFS file system. NameNode manages the file system namespace by storing information about the file system tree which contains the metadata about all the files and directories in the file system tree.

Metadata stored about the file consists of file name, file path, number of blocks, block Ids, replication level.

This metadata information is stored on the local disk. Namenode uses two files for storing this metadata information.

  • FsImage
  • EditLog

We’ll discuss these two files, FsImage and EditLog in more detail in the Secondary NameNode section.

NameNode in Hadoop also keeps, location of the DataNodes that store the blocks for any given file, in it’s memory. Using that information Namenode can reconstruct the whole file by getting the location of all the blocks of a given file.

Client application has to talk to NameNode to add/copy/move/delete a file. Since block information is also stored in NameNode so any client application that wishes to use a file has to get BlockReport from NameNode. The NameNode returns list of DataNodes where the data blocks are stored for the given file.

DataNode in HDFS

Data blocks of the files are stored in a set of DataNodes in Hadoop cluster.

Client application gets the list of DataNodes where data blocks of a particular file are stored from NameNode. After that DataNodes are responsible for serving read and write requests from the file system’s clients. Actual user data never flows through NameNode.

The DataNodes store blocks, delete blocks and replicate those blocks upon instructions from the NameNode.

DataNodes in a Hadoop cluster periodically send a blockreport to the NameNode too. A blockreport contains a list of all blocks on a DataNode.

Secondary NameNode in HDFS

Secondary NameNode in Hadoop is more of a helper to NameNode, it is not a backup NameNode server which can quickly take over in case of NameNode failure.

Before going into details about Secondary NameNode in HDFS let’s go back to the two files which were mentioned while discussing NameNode in Hadoop– FsImage and EditLog.

  • EditLog– All the file write operations done by client applications are first recorded in the EditLog.
  • FsImage– This file has the complete information about the file system metadata when the NameNode starts. All the operations after that are recorded in EditLog.

When the NameNode is restarted it first takes metadata information from the FsImage and then apply all the transactions recorded in EditLog. NameNode restart doesn’t happen that frequently so EditLog grows quite large. That means merging of EditLog to FsImage at the time of startup takes a lot of time keeping the whole file system offline during that process.

Now you may be thinking only if there is some entity which could take over this job of merging FsImage and EditLog and keep the FsImage current that will save a lot of time. That’s exactly what Secondary NameNode does in Hadoop. Its main function is to check point the file system metadata stored on NameNode.

The process followed by Secondary NameNode to periodically merge the fsimage and the edits log files is as follows-

  1. Secondary NameNode gets the latest FsImage and EditLog files from the primary NameNode.
  2. Secondary NameNode applies each transaction from EditLog file to FsImage to create a new merged FsImage file.
  3. Merged FsImage file is transferred back to primary NameNode.

The start of the checkpoint process on the secondary NameNode is controlled by two configuration parameters which are to be configured in hdfs-site.xml.

  • dfs.namenode.checkpoint.period - This property specifies the maximum delay between two consecutive checkpoints. Set to 1 hour by default.
  • dfs.namenode.checkpoint.txns - This property defines the number of uncheckpointed transactions on the NameNode which will force an urgent checkpoint, even if the checkpoint period has not been reached. Set to 1 million by default.

Following image shows the HDFS architecture with communication among NameNode, Secondary NameNode, DataNode and client application.

NameNode and DataNode in HDFS

Reference -

That's all for this topic NameNode, DataNode And Secondary NameNode in HDFS. If you have any doubt or any suggestions to make please drop a comment. Thanks!

>>>Return to Hadoop Framework Tutorial Page


Related Topics

  1. File Read in HDFS - Hadoop Framework Internal Steps
  2. Java Program to Write File in HDFS
  3. Replica Placement Policy in Hadoop Framework
  4. What is SafeMode in Hadoop
  5. HDFS High Availability

You may also like-

  1. What is Big Data
  2. Introduction to Hadoop Framework
  3. Uber Mode in Hadoop
  4. Capacity Scheduler in YARN
  5. Data Compression in Hadoop
  6. MapReduce Flow in YARN
  7. Try-With-Resources in Java Exception Handling
  8. Just In Time Compiler (JIT) in Java