mapreduce執行時報java.net.ConnectException
解決方案
在執行mapreduce程序時出現java.net.ConnectException: 拒絕連接異常,在錯誤信息中看到" localhost:35334"字眼,懷疑是沒有配置主機名的原因, (1)嘗試對每台虛擬機設置主機名,同時更新ssh密鑰信息。
(2)更改hdfs-site.xml和yarn-site.xml相關內容,將原來的ip號改為主機名形式,並在./etc/hadoop/slaves文件下追加slave節點的主機名,將這三個文件分別傳送到slave節點上。
(3)刪除master下namenode和tmp路徑下的所有信息,刪除slave節點下nodename和tmp路徑下的所有信息.對master節點重新進行format
./bin/hadoop namenode -format
配置 core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://master:9000</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property> </configuration>
yarn-site.xml
<configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>master</value> <description>host Single hostname that can be set in place of setting all yarn.resourcemanager*address resources. Results in default ports for ResourceManager components.</description> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> <description></description> </property> </configuration>
slaves
node1
node2
錯誤詳細信息
[root@bogon bin]# ./hadoop jar /home/wordcount.jar /home/test.txt /home/test/rs.txt
19/08/27 23:33:17 INFO client.RMProxy: Connecting to ResourceManager at /192.168.120.128:8032
19/08/27 23:33:33 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/08/27 23:33:35 INFO input.FileInputFormat: Total input files to process : 1
19/08/27 23:33:35 INFO mapreduce.JobSubmitter: number of splits:1
19/08/27 23:33:35 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
19/08/27 23:33:36 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1566918823206_0003
19/08/27 23:33:36 INFO impl.YarnClientImpl: Submitted application application_1566918823206_0003
19/08/27 23:33:37 INFO mapreduce.Job: The url to track the job: http://bogon:8088/proxy/application_1566918823206_0003/
19/08/27 23:33:37 INFO mapreduce.Job: Running job: job_1566918823206_0003
19/08/27 23:43:02 INFO mapreduce.Job: Job job_1566918823206_0003 running in uber mode : false
19/08/27 23:43:02 INFO mapreduce.Job: map 0% reduce 0%
19/08/27 23:43:03 INFO mapreduce.Job: Job job_1566918823206_0003 failed with state FAILED due to: Application application_1566918823206_0003
failed 2 times due to Error launching appattempt_1566918823206_0003_000002. Got exception: java.net.ConnectException:
Call From bogon/192.168.120.128 to localhost:35334 failed on connection exception: java.net.ConnectException: 拒絕連接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor44.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511)
at org.apache.hadoop.ipc.Client.call(Client.java:1453)
at org.apache.hadoop.ipc.Client.call(Client.java:1363)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy82.startContainers(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:128)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy83.startContainers(Unknown Source)
at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:122)
at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:311)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: 拒絕連接
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:690)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:412)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1568)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
... 19 more
. Failing the application.
19/08/27 23:43:03 INFO mapreduce.Job: Counters: 0