ソリューション:datanodeが起動できない「Incompatible namespaceIDs in...」

hadoopを起動するとdatanodeは起動できません.次のコマンドを使用してログ情報を表示します.

tail -100f /usr/local/hadoop/libexec/../logs/hadoop-hadoop-datanode-master.log

次のエラーが表示されます.

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /data/hadoop/hdfs/data: namenode namespaceID = 1265744122; datanode namespaceID = 908607799

理由:namenode formatのたびにnamenode Idが再作成され、/data/hadoop/hdfs/dataの下に前回formatの下のidが含まれ、namenode formatはnamenodeの下のデータを空にしたが、青空datanodeの下のデータがなく、起動時に失敗し、fotmatの前に/data/hadoop/hdfs/data以下のすべてのディレクトリを空にする.

[hadoop@master name]$ cd /data/hadoop/hdfs
[hadoop@master hdfs]$ ll
total 16
drwxr-xr-x 6 hadoop hadoop 4096 Apr 22 18:42 data
drwxrwxr-x 5 hadoop hadoop 4096 Apr 22 18:42 name

[root@master hdfs]# rm -rf data/*
[root@master hdfs]# su hadoop

[hadoop@master root]$ jps
12426 Jps
11532 JobTracker
11278 NameNode
11675 TaskTracker
[hadoop@master root]$ stop-all.sh
Warning: $HADOOP_HOME is deprecated.
stopping jobtracker
master: stopping tasktracker
node1: stopping tasktracker
stopping namenode
master: no datanode to stop
node1: no datanode to stop
node1: stopping secondarynamenode
[hadoop@master root]$ start-all.sh
Warning: $HADOOP_HOME is deprecated.
starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-namenode-master.out
node1: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-datanode-node1.out
master: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-datanode-master.out
node1: starting secondarynamenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-node1.out
starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-master.out
master: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-master.out
node1: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-node1.out
[hadoop@master root]$ jps
13035 JobTracker
13243 Jps
13157 TaskTracker
12911 DataNode

この時点でDataNodeが起動しているのに気づきましたが、NameNodeがいなくなりました.慌てずに、以下のnamenodeを再フォーマットすればいいです.

[hadoop@master hadoop]$ stop-all.sh
Warning: $HADOOP_HOME is deprecated.
stopping jobtracker
master: stopping tasktracker
node1: stopping tasktracker
no namenode to stop
master: stopping datanode
node1: stopping datanode
node1: stopping secondarynamenode
[hadoop@master hadoop]$ jps
13581 Jps
[hadoop@master hadoop]$ hadoop namenode -format
Warning: $HADOOP_HOME is deprecated.
14/04/22 18:52:54 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/192.168.1.100
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.1.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
************************************************************/
14/04/22 18:52:54 INFO util.GSet: VM type       = 32-bit
14/04/22 18:52:54 INFO util.GSet: 2% max memory = 0.61875 MB
14/04/22 18:52:54 INFO util.GSet: capacity      = 2^17 = 131072 entries
14/04/22 18:52:54 INFO util.GSet: recommended=131072, actual=131072
14/04/22 18:52:54 INFO namenode.FSNamesystem: fsOwner=hadoop
14/04/22 18:52:54 INFO namenode.FSNamesystem: supergroup=supergroup
14/04/22 18:52:54 INFO namenode.FSNamesystem: isPermissionEnabled=false
14/04/22 18:52:54 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/04/22 18:52:54 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/04/22 18:52:54 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/04/22 18:52:55 INFO common.Storage: Image file of size 112 saved in 0 seconds.
14/04/22 18:52:55 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/data/hadoop/hdfs/name/current/edits
14/04/22 18:52:55 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/data/hadoop/hdfs/name/current/edits
14/04/22 18:52:55 INFO common.Storage: Storage directory /data/hadoop/hdfs/name has been successfully formatted.
14/04/22 18:52:55 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.1.100
************************************************************/
[hadoop@master hadoop]$ jps
13631 Jps
[hadoop@master hadoop]$ start-all.sh
Warning: $HADOOP_HOME is deprecated.
starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-namenode-master.out
node1: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-datanode-node1.out
master: starting datanode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-datanode-master.out
node1: starting secondarynamenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-secondarynamenode-node1.out
starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-jobtracker-master.out
node1: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-node1.out
master: starting tasktracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-hadoop-tasktracker-master.out
[hadoop@master hadoop]$ jps
13699 NameNode
13972 JobTracker
14107 TaskTracker
13823 DataNode
14179 Jps

すべてのノードが起動したことに気づき、OK、大成功!