使用YARN命令管理Hadoop作業
作者：尹正傑
版權聲明：原創作品，謝絕轉載！否則將追究法律責任。
一.YARN命令概述
如下圖所示，Hadoop的yarn命令可以管理YARN的各個方面。雖然ResourceManager Web UI非常有用，但當不能使用Web UI時，就需要使用命令來處理問題。 Hadoop的yarn命令具有廣泛的使用范圍，從某種意義上來說，它可以幫助我們管理大量的Hadoop任務，例如獲取和殺死正在運行的應用程序，獲取作業和守護程序日志，甚至管理ResourceManager的上下線。 可以使用yarn命令從命令行監視和管理應用程序。例如，可以使用"yarn application -list running"命令查看正在運行的作業的狀態。 同樣，可以使用"yarn application -kill <Application ID>"命令從命令行正常刪除正在運行的應用程序，而不是使用Linux kill命令殺死進程。
最好用的yarn命令是application子命令，它允許快速查看某個時間點的集群時間點的使用情況。 如下圖所示，yarn top命令提供了有關"正在運行(running)","已提交(submitted)","待定(pending)","已完成(completed)","已終止(killed)","失敗(failed)"的摘要信息，還顯示為集群的足有作業分配的內存量和CPU內核量。
本篇博客的重點是監控和管理集群，我們需要了解如何使用yarn命令監控YARN集群。 通過該命令可以查看基本作業信息並執行其他任務，例如"yarn application -kill application_1604218163813_0003"來殺死Application ID為"application_1604218163813_0003"正在運行的Job。 使用"yarn application"命令可以執行以下管理任務: (1)列出集群中運行的應用程序; (2)殺死正在運行的應用程序; (3)獲取正在運行的應用程序的狀態;
二.查看YARN應用程序
1>."yarn application -list"命令
如下圖所示，可以使用"yarn application -list"命令，立即查看集群中的所有應用程序(我們稱為作業，YARN稱為應用)。
[root@hadoop101.yinzhengjie.com ~]# yarn application -list 20/11/01 17:06:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:06:30 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):2 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URLapplication_1604218163813_0007 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 RUNNING UNDEFINED 50% http://hadoop104.yinzhengjie.com:8705application_1604218163813_0008 word count MAPREDUCE root root.yinzhengjie.development ACCEPTED UNDEFINED 0% N/A[root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]#
2>."yarn application -list -appStates <States>"命令
使用"yarn application -list"命令可以檢索所有作業的列表，而不考慮它們的狀態。 作業可以具有以下狀態： "所有(ALL)","新的(NEW)","新的保存(NEW_SAVING)","提交(SUBMITTED)","接受(ACCEPTED)","正在運行(RUNNING)","已完成(FINISHED)","FAILED(失敗)"和"被殺死的(KILLED)"。
[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all 20/11/01 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:17:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):8 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1604218163813_0007 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007 application_1604218163813_0008 word count MAPREDUCE root root.yinzhengjie.development FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008 application_1604218163813_0003 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 KILLED KILLED 100% http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003 application_1604218163813_0004 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004 application_1604218163813_0005 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005 application_1604218163813_0006 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006 application_1604218163813_0001 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001 application_1604218163813_0002 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002 [root@hadoop101.yinzhengjie.com ~]#
[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all 20/11/01 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:17:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):8 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1604218163813_0007 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007 application_1604218163813_0008 word count MAPREDUCE root root.yinzhengjie.development FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008 application_1604218163813_0003 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 KILLED KILLED 100% http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003 application_1604218163813_0004 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004 application_1604218163813_0005 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005 application_1604218163813_0006 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006 application_1604218163813_0001 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001 application_1604218163813_0002 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates killed 20/11/01 17:17:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:17:27 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [KILLED] and tags: []):1 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1604218163813_0003 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 KILLED KILLED 100% http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003 [root@hadoop101.yinzhengjie.com ~]#
[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all 20/11/01 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:17:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):8 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1604218163813_0007 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007 application_1604218163813_0008 word count MAPREDUCE root root.yinzhengjie.development FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008 application_1604218163813_0003 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 KILLED KILLED 100% http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003 application_1604218163813_0004 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004 application_1604218163813_0005 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005 application_1604218163813_0006 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006 application_1604218163813_0001 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001 application_1604218163813_0002 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates failed 20/11/01 17:17:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:17:32 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [FAILED] and tags: []):0 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL [root@hadoop101.yinzhengjie.com ~]#
[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all 20/11/01 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:17:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):8 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1604218163813_0007 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007 application_1604218163813_0008 word count MAPREDUCE root root.yinzhengjie.development FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008 application_1604218163813_0003 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 KILLED KILLED 100% http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003 application_1604218163813_0004 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004 application_1604218163813_0005 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005 application_1604218163813_0006 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006 application_1604218163813_0001 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001 application_1604218163813_0002 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates finished 20/11/01 17:25:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:25:04 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [FINISHED] and tags: []):7 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1604218163813_0007 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007 application_1604218163813_0008 word count MAPREDUCE root root.yinzhengjie.development FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008 application_1604218163813_0004 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004 application_1604218163813_0005 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005 application_1604218163813_0006 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006 application_1604218163813_0001 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001 application_1604218163813_0002 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002 [root@hadoop101.yinzhengjie.com ~]#
三.檢查應用程序的狀態
1>.通過"yarn application -status <Application ID>"查看特定作業的狀態
[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all 20/11/01 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:17:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):8 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1604218163813_0007 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007 application_1604218163813_0008 word count MAPREDUCE root root.yinzhengjie.development FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008 application_1604218163813_0003 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 KILLED KILLED 100% http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003 application_1604218163813_0004 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004 application_1604218163813_0005 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005 application_1604218163813_0006 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006 application_1604218163813_0001 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001 application_1604218163813_0002 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# yarn application -status application_1604218163813_0008 20/11/01 17:33:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:33:54 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 20/11/01 17:33:55 INFO conf.Configuration: resource-types.xml not found 20/11/01 17:33:55 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 20/11/01 17:33:55 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE 20/11/01 17:33:55 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE Application Report : Application-Id : application_1604218163813_0008 Application-Name : word count Application-Type : MAPREDUCE User : root Queue : root.yinzhengjie.development Application Priority : 0 Start-Time : 1604221581251 Finish-Time : 1604221620578 Progress : 100% State : FINISHED Final-State : SUCCEEDED Tracking-URL : http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008 RPC Port : 2890 AM Host : hadoop103.yinzhengjie.com Aggregate Resource Allocation : 58152 MB-seconds, 27 vcore-seconds Aggregate Resource Preempted : 0 MB-seconds, 0 vcore-seconds Log Aggregation Status : SUCCEEDED Diagnostics : Unmanaged Application : false Application Node Label Expression : <Not set> AM container Node Label Expression : <DEFAULT_PARTITION> TimeoutType : LIFETIME ExpiryTime : UNLIMITED RemainingTime : -1seconds [root@hadoop101.yinzhengjie.com ~]#
2>."yarn applicationattempt -list <Application ID>"命令
[root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates all 20/11/01 17:17:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:17:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):8 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1604218163813_0007 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0007 application_1604218163813_0008 word count MAPREDUCE root root.yinzhengjie.development FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0008 application_1604218163813_0003 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 KILLED KILLED 100% http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003 application_1604218163813_0004 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0004 application_1604218163813_0005 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0005 application_1604218163813_0006 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0006 application_1604218163813_0001 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0001 application_1604218163813_0002 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 FINISHED SUCCEEDED 100% http://hadoop105.yinzhengjie.com:19888/jobhistory/job/job_1604218163813_0002 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# yarn applicationattempt -list application_1604218163813_0008 20/11/01 17:44:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:44:51 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of application attempts :1 ApplicationAttempt-Id State AM-Container-Id Tracking-URL appattempt_1604218163813_0008_000001 FINISHED container_1604218163813_0008_01_000001 http://hadoop101.yinzhengjie.com:8088/proxy/application_1604218163813_0008/ [root@hadoop101.yinzhengjie.com ~]#
3>."yarn applicationattempt -status <Application Attempt ID>"命令
[root@hadoop101.yinzhengjie.com ~]# yarn applicationattempt -list application_1604218163813_0008 20/11/01 17:51:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:51:51 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of application attempts :1 ApplicationAttempt-Id State AM-Container-Id Tracking-URL appattempt_1604218163813_0008_000001 FINISHED container_1604218163813_0008_01_000001 http://hadoop101.yinzhengjie.com:8088/proxy/application_1604218163813_0008/ [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# yarn applicationattempt -status appattempt_1604218163813_0008_000001 20/11/01 17:52:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:52:02 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Application Attempt Report : ApplicationAttempt-Id : appattempt_1604218163813_0008_000001 State : FINISHED AMContainer : container_1604218163813_0008_01_000001 Tracking-URL : http://hadoop101.yinzhengjie.com:8088/proxy/application_1604218163813_0008/ RPC Port : 2890 AM Host : hadoop103.yinzhengjie.com Diagnostics : [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]#
四.kill正在執行的作業("yarn application -kill <Application ID>")
[root@hadoop101.yinzhengjie.com ~]# yarn application -list 20/11/01 17:59:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:59:08 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):1 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1604218163813_0013 word count MAPREDUCE root root.yinzhengjie.testing RUNNING UNDEFINED 5% http://hadoop104.yinzhengjie.com:2692 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# yarn application -kill application_1604218163813_0013 20/11/01 17:59:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:59:18 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Killing application application_1604218163813_0013 20/11/01 17:59:18 INFO impl.YarnClientImpl: Killed application application_1604218163813_0013 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# yarn application -list 20/11/01 17:59:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:59:22 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# yarn application -list -appStates killed 20/11/01 17:59:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 17:59:27 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total number of applications (application-types: [], states: [KILLED] and tags: []):2 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1604218163813_0013 word count MAPREDUCE root root.yinzhengjie.testing KILLED KILLED 10 0% http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0013 application_1604218163813_0003 word count MAPREDUCE root root.yinzhengjie.operation.op_queue01 KILLED KILLED 100% http://hadoop101.yinzhengjie.com:8088/cluster/app/application_1604218163813_0003 [root@hadoop101.yinzhengjie.com ~]#
五.檢查節點狀態
[root@hadoop101.yinzhengjie.com ~]# yarn node -all -list 20/11/01 18:09:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 18:09:42 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Total Nodes:3 Node-Id Node-State Node-Http-Address Number-of-Running-Containers hadoop104.yinzhengjie.com:6666 RUNNING hadoop104.yinzhengjie.com:8042 0 hadoop102.yinzhengjie.com:6666 RUNNING hadoop102.yinzhengjie.com:8042 0 hadoop103.yinzhengjie.com:6666 RUNNING hadoop103.yinzhengjie.com:8042 0 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]#
六.檢查YARN的隊列狀態
[root@hadoop101.yinzhengjie.com ~]# yarn queue -status yinzhengjie 20/11/01 18:15:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 18:15:48 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 Queue Information : Queue Name : root.yinzhengjie State : RUNNING Capacity : .0% Current Capacity : .0% Maximum Capacity : -100.0% Default Node Label expression : <DEFAULT_PARTITION> Accessible Node Labels : [root@hadoop101.yinzhengjie.com ~]#
七.獲取作業的日志
[root@hadoop101.yinzhengjie.com ~]# ll total 0 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# yarn logs -applicationId application_1604218163813_0008 > application_1604218163813_0008.log 20/11/01 18:27:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/11/01 18:27:09 INFO client.RMProxy: Connecting to ResourceManager at hadoop101.yinzhengjie.com/172.200.6.101:8032 [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# ll total 140 -rw-r--r-- 1 root root 140715 Nov 1 18:27 application_1604218163813_0008.log [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# [root@hadoop101.yinzhengjie.com ~]# ll -h total 140K -rw-r--r-- 1 root root 138K Nov 1 18:27 application_1604218163813_0008.log [root@hadoop101.yinzhengjie.com ~]#
八.YARN管理命令
1>.節點的上線與下線
2>.高可用ResourceManager
3>.其他