Datanode denied communication with namenode


Error Scenario:

Datanode denied communication with namenode:

We usually get this error message while adding a data node to an existing HDFS cluster or at the time of cluster setup. Below is the snapshot of this error message from log file in data node.

Root Cause:

This error message is receive if the hadoop user (for example hduser) on namenode machine doesn’t have full (write) permissions to the directory location <dfs.datanode.data.dir> on data node machine. This directory exists in the local file system of the data node.

In the above example scenario, the name node (172.16.2.211) is not having write access to /home/hduser/hdfs directory on data node and that’s why

When start-dfs.sh is triggered from namenode,

  1. namenode is trying to start the data node daemon and
  2. create the data/current directory under <dfs.datanode.data.dir> location on Datanode-dn1.
  3. Since, hduser@Namenode-nn doesn’t have write access to Datanode-dn1, datanode is denying the hduser@Namenode-nn to create directories on datanode.

Resolution:

By giving full permissions to hduser@Namenode-nn for <dfs.datanode.data.dir> location we can avoid this error message.

Give  full permissions with the help of below command to <dfs.datanode.data.dir>

With this command (3rd digit 7), ) any other users will have read (4) + execute (1) + write (2) access to this directory.

So, now hduser@Namenode-nn can create sub directories data/current under /home/user/hdfs on data node.


Profile photo of Siva

About Siva

Senior Hadoop developer with 4 years of experience in designing and architecture solutions for the Big Data domain and has been involved with several complex engagements. Technical strengths include Hadoop, YARN, Mapreduce, Hive, Sqoop, Flume, Pig, HBase, Phoenix, Oozie, Falcon, Kafka, Storm, Spark, MySQL and Java.

Leave a comment

Your email address will not be published. Required fields are marked *