Sizing NameNode Heap Memory

The NameNode memory allocation plays a critical role in ensuring overall cluster performance. The NameNode is the core metadata server of Hadoop. This is the most critical piece of the system. NameNode stores the file system image and the file system journal. The NameNode keeps all of the filesystem layout information (files, blocks, directories, permissions, etc) and the block locations. The filesystem layout is persisted on disk and the block locations are kept solely in memory. When a client opens a file, the name node tells the client the locations of all the blocks in the file; the client then no longer needs to communicate with the NameNode for data transfer.

HADOOP_HEAPSIZE
Sets the JVM heap size for all Hadoop  servers such as HDFS, YARN, and MapReduce.  It is an integer passed to the JVM as the maximum memory (Xmx) argument
HADOOP_NAMENODE_OPTS
Specific to the NameNode and sets all JVM flags, which must be specified. HADOOP_NAMENODE_OPTS overrides the HADOOP_HEAPSIZE Xmx value for the NameNode.

It’s difficult to give a precise formula because memory usage depends on the number of blocks per file, the filename length, and the number of directories in the filesystem; plus, it can change from one Hadoop release to another. Some working examples:

Estimating NameNode Heap Memory Needed
(Reference: Cloudera Documentation & Hadoop: The Definitive Guide
Cluster A: 200 hosts of 24 TB each = 4800 TB.

    Blocksize=128 MB, Replication=1
    Cluster capacity in MB: 200 * 24,000,000 MB = 4,800,000,000 MB (4800 TB)
    Disk space needed per block: 128 MB per block * 1 = 128 MB storage per block
    Cluster capacity in blocks: 4,800,000,000 MB / 128 MB = 36,000,000 blocks

At capacity, with the recommended allocation of 1 GB of memory per million blocks, Cluster A needs 36 GB of maximum heap space.

Cluster B: 200 hosts of 24 TB each = 4800 TB.

    Blocksize=128 MB, Replication=3
    Cluster capacity in MB: 200 * 24,000,000 MB = 4,800,000,000 MB (4800 TB)
    Disk space needed per block: 128 MB per block * 3 = 384 MB storage per block
    Cluster capacity in blocks: 4,800,000,000 MB / 384 MB = 12,000,000 blocks

At capacity, with the recommended allocation of 1 GB of memory per million blocks, Cluster B needs 12 GB of maximum heap space.

Both Cluster A and Cluster B store the same number of block files. In Cluster A, however, each block file is unique and represented by one block on the NameNode; in Cluster B, only one-third are unique and two-thirds are replicas.