cdh6 service启动遇到问题 ：提示主机unhealthy
打开日志显示的是
............................
Caused by: java.net.ConnectException: Connection refused (Connection refused)
.....................
msg:Role not started due to unhealthy host
...............................
所有日志:
ty.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:134) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at com.cloudera.server.common.BoundedQueuedThreadPool$2.run(BoundedQueuedThreadPool.java:94) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) 2020-07-15 14:46:38,456 WARN scm-web-507:org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver: Resolved [java.lang.NullPointerException] to ModelAndView: materialized View is [com.cloudera.server.web.common.JamonModelAndView$JamonView@53128fcb]; model is {} 2020-07-15 14:46:53,445 INFO scm-web-424:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/services/13/healthStatusBar 2020-07-15 14:46:53,501 INFO scm-web-424:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/services/13/healthStatusBar, Status:200 2020-07-15 14:47:07,482 INFO scm-web-423:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/parcel/details 2020-07-15 14:47:07,750 INFO scm-web-423:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/parcel/details, Status:200 2020-07-15 14:47:13,499 INFO scm-web-423:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/parcel/details 2020-07-15 14:47:13,537 INFO scm-web-423:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/parcel/details, Status:200 2020-07-15 14:47:18,784 INFO scm-web-419:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/parcel/details 2020-07-15 14:47:18,831 INFO scm-web-419:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/parcel/details, Status:200 2020-07-15 14:47:24,043 INFO scm-web-419:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/parcel/details 2020-07-15 14:47:24,100 INFO scm-web-419:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/parcel/details, Status:200 2020-07-15 14:47:24,446 INFO scm-web-419:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/services/13/healthStatusBar 2020-07-15 14:47:24,535 INFO scm-web-419:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/services/13/healthStatusBar, Status:200 2020-07-15 14:47:32,420 INFO scm-web-422:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/hardware/hosts/1/healthStatusBar 2020-07-15 14:47:32,456 WARN scm-web-425:com.cloudera.server.web.cmf.HealthReportHelper: Error getting health report from Host Monitor. org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused (Connection refused) at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:104) at com.sun.proxy.$Proxy206.getAvroHealthReports(Unknown Source) at com.cloudera.cmf.protocol.firehose.nozzle.TimeoutNozzleIPC.getAvroHealthReports(TimeoutNozzleIPC.java:127) at com.cloudera.cmon.NozzleIPCWrapper.getHealthReports(NozzleIPCWrapper.java:599) at com.cloudera.server.web.cmf.HealthReportHelper$GetHealthReportCallable.call(HealthReportHelper.java:502) at com.cloudera.server.web.cmf.HealthReportHelper.getHealthReport(HealthReportHelper.java:393) at com.cloudera.server.web.cmf.HealthCheckController.hostStatusHealthCheckJSON(HealthCheckController.java:448) at com.cloudera.server.web.cmf.HealthCheckController$$FastClassBySpringCGLIB$$40ee95d3.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:84) at com.cloudera.server.web.cmf.aop.RetryExecution$1.call(RetryExecution.java:32) at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32) at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210) at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:131) at com.cloudera.server.web.cmf.aop.RetryExecution.retryOperation(RetryExecution.java:24) at sun.reflect.GeneratedMethodAccessor1549.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671) at com.cloudera.server.web.cmf.HealthCheckController$$EnhancerBySpringCGLIB$$498cf7ed.hostStatusHealthCheckJSON(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) at javax.servlet.http.HttpServlet.service(HttpServlet.java:645) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:200) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:158) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:134) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at com.cloudera.server.common.BoundedQueuedThreadPool$2.run(BoundedQueuedThreadPool.java:94) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) at sun.net.www.http.HttpClient.New(HttpClient.java:339) at sun.net.www.http.HttpClient.New(HttpClient.java:357) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315) at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:77) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72) at org.apache.avro.ipc.Requestor.request(Requestor.java:147) at org.apache.avro.ipc.Requestor.request(Requestor.java:101) at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88) ... 119 more 2020-07-15 14:47:32,474 INFO scm-web-422:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/hardware/hosts/1/healthStatusBar, Status:200 2020-07-15 14:47:34,602 INFO avro-servlet-hb-processor-12:com.cloudera.server.common.AgentAvroServlet: (39 skipped) AgentAvroServlet: heartbeat processing stats: average=64ms, min=25ms, max=1307ms. 2020-07-15 14:47:39,488 INFO scm-web-425:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/hardware/hostsBringUp 2020-07-15 14:47:39,520 INFO scm-web-425:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command HostsBringUp HostsCmdArgs{args=[jzmaster1]}. 2020-07-15 14:47:39,586 INFO scm-web-425:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 8 steps in sequence 2020-07-15 14:47:39,587 INFO scm-web-425:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start on service HDFS 2020-07-15 14:47:39,638 INFO scm-web-425:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command HdfsStartWithFailovers SvcCmdArgs{targetRoles=[DbRole{id=181, name=hdfs-SECONDARYNAMENODE-8be12fc4fb5b84bd7c1978b6303e70c3, hostName=jzmaster1}, DbRole{id=190, name=hdfs-NAMENODE-8be12fc4fb5b84bd7c1978b6303e70c3, hostName=jzmaster1}, DbRole{id=191, name=hdfs-BALANCER-8be12fc4fb5b84bd7c1978b6303e70c3, hostName=jzmaster1}], args=[]}. Service: DbService{id=24, name=hdfs} 2020-07-15 14:47:39,652 INFO scm-web-425:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 2 steps in sequence 2020-07-15 14:47:39,653 INFO scm-web-425:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start on service HDFS 2020-07-15 14:47:39,682 INFO scm-web-425:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command Start SvcCmdArgs{targetRoles=[DbRole{id=181, name=hdfs-SECONDARYNAMENODE-8be12fc4fb5b84bd7c1978b6303e70c3, hostName=jzmaster1}, DbRole{id=190, name=hdfs-NAMENODE-8be12fc4fb5b84bd7c1978b6303e70c3, hostName=jzmaster1}, DbRole{id=191, name=hdfs-BALANCER-8be12fc4fb5b84bd7c1978b6303e70c3, hostName=jzmaster1}], args=[]}. Service: DbService{id=24, name=hdfs} 2020-07-15 14:47:39,683 INFO scm-web-425:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: Added BringUp command to service DbService{id=24, name=hdfs}. 2020-07-15 14:47:39,686 INFO scm-web-425:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence 2020-07-15 14:47:39,686 INFO scm-web-425:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Starting 2 roles on service 2020-07-15 14:47:39,714 INFO scm-web-425:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start this SecondaryNameNode on role SecondaryNameNode (jzmaster1) 2020-07-15 14:47:39,792 INFO scm-web-425:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=24, name=hdfs} Role: DbRole{id=181, name=hdfs-SECONDARYNAMENODE-8be12fc4fb5b84bd7c1978b6303e70c3, hostName=jzmaster1} 2020-07-15 14:47:39,795 ERROR scm-web-425:com.cloudera.cmf.model.DbCommand: Command null(Start) has completed. finalstate:FINISHED, success:false, msg:Role not started due to unhealthy host jzmaster1. 2020-07-15 14:47:39,796 ERROR scm-web-425:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 2277 Role not started due to unhealthy host jzmaster1. 2020-07-15 14:47:39,798 INFO scm-web-425:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start this NameNode on role NameNode (jzmaster1) 2020-07-15 14:47:39,819 WARN scm-web-425:com.cloudera.cmf.service.config.PortNumberParamSpec: Host jzmaster1 missing heartbeat/ephemeral port info, skipping check 2020-07-15 14:47:39,830 INFO scm-web-425:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=24, name=hdfs} Role: DbRole{id=190, name=hdfs-NAMENODE-8be12fc4fb5b84bd7c1978b6303e70c3, hostName=jzmaster1} 2020-07-15 14:47:39,830 ERROR scm-web-425:com.cloudera.cmf.model.DbCommand: Command null(Start) has completed. finalstate:FINISHED, success:false, msg:Role not started due to unhealthy host jzmaster1. 2020-07-15 14:47:39,831 ERROR scm-web-425:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 2278 Role not started due to unhealthy host jzmaster1. 2020-07-15 14:47:39,831 ERROR scm-web-425:com.cloudera.cmf.model.DbCommand: Command 2276(Start) has completed. finalstate:FINISHED, success:false, msg:Failed to start service. 2020-07-15 14:47:39,832 ERROR scm-web-425:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 2276 Failed to start service. 2020-07-15 14:47:39,836 ERROR scm-web-425:com.cloudera.cmf.model.DbCommand: Command 2275(HdfsStartWithFailovers) has completed. finalstate:FINISHED, success:false, msg:Failed to start the HDFS service 2020-07-15 14:47:39,842 ERROR scm-web-425:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 2275 Failed to start the HDFS service 2020-07-15 14:47:39,843 ERROR scm-web-425:com.cloudera.cmf.model.DbCommand: Command 2274(HostsBringUp) has completed. finalstate:FINISHED, success:false, msg:Failed to start all the roles on selected hosts. 2020-07-15 14:47:39,940 INFO scm-web-425:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/hardware/hostsBringUp, Status:200 2020-07-15 14:47:40,540 WARN scm-web-425:com.cloudera.cmf.command.flow.SeqFlowCmd: Invalid command state json com.cloudera.enterprise.JsonUtil2$JsonRuntimeException: com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input at [Source: (String)""; line: 1, column: 0] at com.cloudera.enterprise.JsonUtil2.valueFromString(JsonUtil2.java:193) at com.cloudera.enterprise.JsonUtil2.valueFromBytes(JsonUtil2.java:290) at com.cloudera.cmf.command.flow.SeqFlowCmd.from(SeqFlowCmd.java:217) at com.cloudera.cmf.command.flow.SeqFlowCmd.getProgress(SeqFlowCmd.java:327) at com.cloudera.cmf.command.CmdWorkCommand.getProgress(CmdWorkCommand.java:181) at com.cloudera.server.web.cmf.commands.CommandController.getCommandProgress(CommandController.java:96) at com.cloudera.server.web.cmf.commands.CommandController.commandDetailsJson(CommandController.java:114) at com.cloudera.server.web.cmf.commands.CommandController$$FastClassBySpringCGLIB$$b64180f6.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:84) at com.cloudera.server.web.cmf.aop.RetryExecution$1.call(RetryExecution.java:32) at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32) at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210) at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:131) at com.cloudera.server.web.cmf.aop.RetryExecution.retryOperation(RetryExecution.java:24) at sun.reflect.GeneratedMethodAccessor1549.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671) at com.cloudera.server.web.cmf.commands.CommandController$$EnhancerBySpringCGLIB$$b31eec26.commandDetailsJson(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) at javax.servlet.http.HttpServlet.service(HttpServlet.java:645) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:200) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:158) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:134) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at com.cloudera.server.common.BoundedQueuedThreadPool$2.run(BoundedQueuedThreadPool.java:94) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input at [Source: (String)""; line: 1, column: 0] at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59) at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4145) at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4000) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3004) at com.cloudera.enterprise.JsonUtil2.valueFromString(JsonUtil2.java:189) ... 119 more 2020-07-15 14:47:49,668 INFO scm-web-508:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/statusContent 2020-07-15 14:47:49,675 INFO scm-web-422:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar 2020-07-15 14:47:49,714 INFO scm-web-422:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar, Status:200 2020-07-15 14:47:49,793 INFO scm-web-508:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/statusContent, Status:200 2020-07-15 14:47:53,485 INFO scm-web-425:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/services/13/healthStatusBar 2020-07-15 14:47:53,522 INFO scm-web-425:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/services/13/healthStatusBar, Status:200 2020-07-15 14:47:54,160 INFO scm-web-507:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/do 2020-07-15 14:47:54,222 INFO scm-web-507:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ClusterInspector BasicCmdArgs{args=[DEFAULT]}. 2020-07-15 14:47:54,227 INFO scm-web-507:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command clusterHostInspector HostInspectorCmdArgs{inspection=DEFAULT, inspectorInput=com.cloudera.cmf.inspector.InspectorInput@54e728d1}. Host: DbHost{id=2, hostId=8d4cc534-afe3-4ecc-b394-6766caa611de, hostName=jzmaster2} 2020-07-15 14:47:54,271 ERROR scm-web-507:com.cloudera.cmf.model.DbCommand: Command null(clusterHostInspector) has completed. finalstate:FINISHED, success:false, msg:Can only run host inspector when host is healthy. 2020-07-15 14:47:54,272 INFO scm-web-507:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command clusterHostInspector HostInspectorCmdArgs{inspection=DEFAULT, inspectorInput=com.cloudera.cmf.inspector.InspectorInput@6e0aa505}. Host: DbHost{id=3, hostId=525476b3-3158-4165-997f-9f89b4896a61, hostName=jznode6} 2020-07-15 14:47:54,280 INFO scm-web-507:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command clusterHostInspector HostInspectorCmdArgs{inspection=DEFAULT, inspectorInput=com.cloudera.cmf.inspector.InspectorInput@1a4722aa}. Host: DbHost{id=4, hostId=b3b4de66-337f-4122-a985-2d112dc2cf58, hostName=jznode3} 2020-07-15 14:47:54,286 INFO scm-web-507:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command clusterHostInspector HostInspectorCmdArgs{inspection=DEFAULT, inspectorInput=com.cloudera.cmf.inspector.InspectorInput@75f3277f}. Host: DbHost{id=9, hostId=26b51ec3-691a-4671-a114-ffacb58e0781, hostName=jznode4} 2020-07-15 14:47:54,293 INFO scm-web-507:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command clusterHostInspector HostInspectorCmdArgs{inspection=DEFAULT, inspectorInput=com.cloudera.cmf.inspector.InspectorInput@e6bb578}. Host: DbHost{id=7, hostId=aeaae19b-5393-49d2-b8a5-80e5d5ca307d, hostName=jznode8} 2020-07-15 14:47:54,301 INFO scm-web-507:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command clusterHostInspector HostInspectorCmdArgs{inspection=DEFAULT, inspectorInput=com.cloudera.cmf.inspector.InspectorInput@506b27b7}. Host: DbHost{id=5, hostId=9921134a-c106-4f67-b451-dfd859d00331, hostName=jznode5} 2020-07-15 14:47:54,307 INFO scm-web-507:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command clusterHostInspector HostInspectorCmdArgs{inspection=DEFAULT, inspectorInput=com.cloudera.cmf.inspector.InspectorInput@367aaa18}. Host: DbHost{id=10, hostId=ffd10518-2c85-4a2d-8daa-0b09b535b82e, hostName=jznode2} 2020-07-15 14:47:54,313 INFO scm-web-507:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command clusterHostInspector HostInspectorCmdArgs{inspection=DEFAULT, inspectorInput=com.cloudera.cmf.inspector.InspectorInput@2fcb1f19}. Host: DbHost{id=11, hostId=f19e645f-1ba2-4d8e-8376-707540b85ac9, hostName=jzmaster3} 2020-07-15 14:47:54,320 INFO scm-web-507:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command clusterHostInspector HostInspectorCmdArgs{inspection=DEFAULT, inspectorInput=com.cloudera.cmf.inspector.InspectorInput@42c04d63}. Host: DbHost{id=6, hostId=d49c33ca-3a4a-431d-8eaa-406a4085bec5, hostName=jznode7} 2020-07-15 14:47:54,327 INFO scm-web-507:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command clusterHostInspector HostInspectorCmdArgs{inspection=DEFAULT, inspectorInput=com.cloudera.cmf.inspector.InspectorInput@1e6b67be}. Host: DbHost{id=8, hostId=5722dda5-d768-49e8-a743-551bde1e7968, hostName=jznode1} 2020-07-15 14:47:54,542 INFO scm-web-507:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/do, Status:200 2020-07-15 14:47:54,614 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:47:54,676 WARN ActionablesProvider-0:com.cloudera.cmf.service.config.PortNumberParamSpec: Host jzmaster1 missing heartbeat/ephemeral port info, skipping check 2020-07-15 14:47:54,740 WARN ActionablesProvider-0:com.cloudera.cmf.service.config.PortNumberParamSpec: Host jzmaster1 missing heartbeat/ephemeral port info, skipping check 2020-07-15 14:47:59,689 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:04,738 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:09,787 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:09,898 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:09,973 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:10,193 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:10,281 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:10,747 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:10,781 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:11,037 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:11,165 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:15,535 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'clusterHostInspector' 2020-07-15 14:48:15,570 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 2285(clusterHostInspector) has completed. finalstate:FINISHED, success:true, msg:Inspector ran successfully. 2020-07-15 14:48:15,592 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'clusterHostInspector' 2020-07-15 14:48:15,602 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 2279(clusterHostInspector) has completed. finalstate:FINISHED, success:true, msg:Inspector ran successfully. 2020-07-15 14:48:15,630 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:15,722 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:16,032 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'clusterHostInspector' 2020-07-15 14:48:16,047 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 2283(clusterHostInspector) has completed. finalstate:FINISHED, success:true, msg:Inspector ran successfully. 2020-07-15 14:48:16,080 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:16,336 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'clusterHostInspector' 2020-07-15 14:48:16,344 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 2284(clusterHostInspector) has completed. finalstate:FINISHED, success:true, msg:Inspector ran successfully. 2020-07-15 14:48:16,394 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'clusterHostInspector' 2020-07-15 14:48:16,400 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 2281(clusterHostInspector) has completed. finalstate:FINISHED, success:true, msg:Inspector ran successfully. 2020-07-15 14:48:16,416 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:16,480 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:16,917 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'clusterHostInspector' 2020-07-15 14:48:16,923 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 2288(clusterHostInspector) has completed. finalstate:FINISHED, success:true, msg:Inspector ran successfully. 2020-07-15 14:48:16,956 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:17,262 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'clusterHostInspector' 2020-07-15 14:48:17,274 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 2289(clusterHostInspector) has completed. finalstate:FINISHED, success:true, msg:Inspector ran successfully. 2020-07-15 14:48:17,304 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:17,990 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'clusterHostInspector' 2020-07-15 14:48:17,998 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 2282(clusterHostInspector) has completed. finalstate:FINISHED, success:true, msg:Inspector ran successfully. 2020-07-15 14:48:18,018 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:18,270 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'clusterHostInspector' 2020-07-15 14:48:18,281 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 2286(clusterHostInspector) has completed. finalstate:FINISHED, success:true, msg:Inspector ran successfully. 2020-07-15 14:48:18,301 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) is still active on service null. 2020-07-15 14:48:21,718 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'clusterHostInspector' 2020-07-15 14:48:21,725 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 2287(clusterHostInspector) has completed. finalstate:FINISHED, success:true, msg:Inspector ran successfully. 2020-07-15 14:48:21,747 INFO CommandPusher:com.cloudera.cmf.command.CommandHelpers: Command (ClusterInspector, 2290) has finished on service null. 2020-07-15 14:48:21,747 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 2290(ClusterInspector) has completed. finalstate:FINISHED, success:false, msg:Command completed with 10/11 successful subcommands 2020-07-15 14:48:23,480 INFO scm-web-422:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/services/13/healthStatusBar 2020-07-15 14:48:23,538 INFO scm-web-422:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/services/13/healthStatusBar, Status:200 2020-07-15 14:48:34,679 INFO avro-servlet-hb-processor-7:com.cloudera.server.common.AgentAvroServlet: (57 skipped) AgentAvroServlet: heartbeat processing stats: average=60ms, min=25ms, max=1307ms.
处理方法是关掉集群所有cdh agent
然后先开启service，再打开agent
server机器: service cloudera-scm-server stop
agent机器:service cloudera-scm-agent stop
server机器: service cloudera-scm-server start
agent机器:service cloudera-scm-agent start
异常原因就是:
CDH启动顺序错误
正常启动顺序，先启动server，再启动agent，而造成的原因是自己先启动了agent后启动的server，所以抛异常
导致这次异常的原因：
主节点服务器根目录/ 满了，其中/opt/cloudera占用了很大空间，就直接执行了移动/opt，挂载/dev/vdb1的操作,而没有先停止cdh server和agent：
mv /opt /data/opt_backup
mount /dev/vdb1 /opt
mv /data/opt_backup/* /opt
更新：后面出现有个cdh agent节点无法启动问题：
问题一致无法解决，最后用了最笨的办法，重新覆盖安装该节点的agent，搞掂！