hadoop で ‘container [pid=..,containerID=..] is running beyond physical memory limits. current usage..’を解決する
error
17/01/20 08:22:36 INFO mapreduce.Job: Task Id : attempt_<>, Status : FAILED
Container [pid=<pid>,containerID=container_<id>] is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 6.2 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_<id> :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 70327 70251 70246 70246 (python) 3778 502 510001152 89480 /usr/bin/python mr_test.py reducer
|- 70246 70244 70246 70246 (bash) 0 0 13598720 712 /bin/bash -c /path/to/java -XX:-UseGCOverheadLimit -Xmx4000m -
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
解決方法：Memoryを大きくしてあげる
-Dmapreduce.reduce.memory.mb=5120
17/01/20 08:22:36 INFO mapreduce.Job: Task Id : attempt_<>, Status : FAILED
Container [pid=<pid>,containerID=container_<id>] is running beyond physical memory limits. Current usage: 2.0 GB of 2 GB physical memory used; 6.2 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_<id> :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 70327 70251 70246 70246 (python) 3778 502 510001152 89480 /usr/bin/python mr_test.py reducer
|- 70246 70244 70246 70246 (bash) 0 0 13598720 712 /bin/bash -c /path/to/java -XX:-UseGCOverheadLimit -Xmx4000m -
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
-Dmapreduce.reduce.memory.mb=5120
これをJarのあとに追加するだけ！
Author And Source
この問題について(hadoop で ‘container [pid=..,containerID=..] is running beyond physical memory limits. current usage..’を解決する), 我々は、より多くの情報をここで見つけました https://qiita.com/nakamasato/items/5ebda74ad9170245d1c8著者帰属：元の著者の情報は、元のURLに含まれています。著作権は原作者に属する。
Content is automatically searched and collected through network algorithms . If there is a violation . Please contact us . We will adjust (correct author information ,or delete content ) as soon as possible .