Netty源碼學習（四）Netty服務器是如何啟動的？

本文會分析Netty服務器的啟動過程，采用的范例代碼是Netty編寫的Echo Server。

 

0. 聲明acceptor與worker

由於Netty采用的reactor模型，所以需要聲明兩組線程，一組作為boss/acceptor，另一組作為worker

boss/acceptor負責監聽綁定的端口，accept新接入的連接，然后將這些連接轉交給worker，worker會處理這些連接上的讀寫事件。

也就是下面的代碼：

EventLoopGroup bossGroup = new NioEventLoopGroup(1);//boss/acceptor線程組

EventLoopGroup workerGroup = new NioEventLoopGroup();//worker線程組

 

1. 聲明ServerBootstrap，並調用bind方法開始監聽端口

ServerBootstrap.bind()方法最終會調用到AbstractBootstrap.doBind()方法，其源碼如下：

    private ChannelFuture doBind(final SocketAddress localAddress) {
        final ChannelFuture regFuture = initAndRegister();
        final Channel channel = regFuture.channel();
        if (regFuture.cause() != null) {
            return regFuture;
        }

        if (regFuture.isDone()) {//如果Channel已經register成功，則直接調用doBind0方法
            // At this point we know that the registration was complete and successful.
            ChannelPromise promise = channel.newPromise();
            doBind0(regFuture, channel, localAddress, promise);//doBind0方法后文再做分析
            return promise;
        } else {//否則添加一個回調函數，在Channel register成功后再調用doBind0方法
            // Registration future is almost always fulfilled already, but just in case it's not.
            final PendingRegistrationPromise promise = new PendingRegistrationPromise(channel);
            regFuture.addListener(new ChannelFutureListener() {
                @Override
                public void operationComplete(ChannelFuture future) throws Exception {
                    Throwable cause = future.cause();
                    if (cause != null) {
                        // Registration on the EventLoop failed so fail the ChannelPromise directly to not cause an
                        // IllegalStateException once we try to access the EventLoop of the Channel.
                        promise.setFailure(cause);
                    } else {
                        // Registration was successful, so set the correct executor to use.
                        // See https://github.com/netty/netty/issues/2586
                        promise.registered();

                        doBind0(regFuture, channel, localAddress, promise);
                    }
                }
            });
            return promise;
        }
    }

    final ChannelFuture initAndRegister() {
        Channel channel = null;
        try {
            channel = channelFactory.newChannel();//創建一個新的Channel，這里的channelFactory對象是在初始化ServerBootstrap時調用channel()方法的時候被設置的，在Echo Server范例中，此處會構造一個NioServerSocketChannel
            init(channel);//見下文分析，初始化Channel，設置屬性與選項，以及為這個NioServerSocketChannel添加一個ChannelInitializer，其目的是在這個Channel被register到boss EventLoopGroup的時候，自動添加一個SerrverBootstrapAcceptor
        } catch (Throwable t) {
            if (channel != null) {
                // channel can be null if newChannel crashed (eg SocketException("too many open files"))
                channel.unsafe().closeForcibly();
            }
            // as the Channel is not registered yet we need to force the usage of the GlobalEventExecutor
            return new DefaultChannelPromise(channel, GlobalEventExecutor.INSTANCE).setFailure(t);//如果初始化Channel的時候發生異常，則將其包裝一下然后返回
        }

        ChannelFuture regFuture = config().group().register(channel);//group()方法得到的是boss EventLoopGroup，register方法很重要，后文再做分析
        if (regFuture.cause() != null) {
            if (channel.isRegistered()) {
                channel.close();
            } else {
                channel.unsafe().closeForcibly();
            }
        }

        // If we are here and the promise is not failed, it's one of the following cases:
        // 1) If we attempted registration from the event loop, the registration has been completed at this point.
        //    i.e. It's safe to attempt bind() or connect() now because the channel has been registered.
        // 2) If we attempted registration from the other thread, the registration request has been successfully
        //    added to the event loop's task queue for later execution.
        //    i.e. It's safe to attempt bind() or connect() now:
        //         because bind() or connect() will be executed *after* the scheduled registration task is executed
        //         because register(), bind(), and connect() are all bound to the same thread.

        return regFuture;
    }


        @Override
    void init(Channel channel) throws Exception {
        final Map<ChannelOption<?>, Object> options = options0();
        synchronized (options) {
            setChannelOptions(channel, options, logger);//設置的channel的options
        }

        final Map<AttributeKey<?>, Object> attrs = attrs0();
        synchronized (attrs) {
            for (Entry<AttributeKey<?>, Object> e: attrs.entrySet()) {
                @SuppressWarnings("unchecked")
                AttributeKey<Object> key = (AttributeKey<Object>) e.getKey();
                channel.attr(key).set(e.getValue());//設置Channel的attribute
            }
        }

        ChannelPipeline p = channel.pipeline();//獲取Channel的pipeline，此時為DefaultChannelPipeline，且只有head與tail兩個節點

        final EventLoopGroup currentChildGroup = childGroup;
        final ChannelHandler currentChildHandler = childHandler;
        final Entry<ChannelOption<?>, Object>[] currentChildOptions;
        final Entry<AttributeKey<?>, Object>[] currentChildAttrs;
        synchronized (childOptions) {
            currentChildOptions = childOptions.entrySet().toArray(newOptionArray(childOptions.size()));
        }
        synchronized (childAttrs) {
            currentChildAttrs = childAttrs.entrySet().toArray(newAttrArray(childAttrs.size()));
        }

        p.addLast(new ChannelInitializer<Channel>() {//向pipeline中添加一個ChannelInitializer對象
            @Override
            public void initChannel(final Channel ch) throws Exception {
                final ChannelPipeline pipeline = ch.pipeline();
                ChannelHandler handler = config.handler();
                if (handler != null) {
                    pipeline.addLast(handler);
                }

                ch.eventLoop().execute(new Runnable() {
                    @Override
                    public void run() {
                        pipeline.addLast(new ServerBootstrapAcceptor(//為當前Channel添加一個ServerBootstrapAcceptor，其目的后文會做分析
                                ch, currentChildGroup, currentChildHandler, currentChildOptions, currentChildAttrs));
                    }
                });
            }
        });
    }

主要干的事情：

a. 聲明了一個Channel對象（實際上是NioServerSocketChannel對象），這個Channel會被register到之前聲明的boss NioEventLoopGroup里，然后再調用doBind0方法綁定端口，后文會分析這兩個方法的調用鏈

b. 向這個Channel的pipeline里添加一個ChannelInitializer對象，這個對象繼承於ChannelInboundHandler接口，后續初始化的時候會向pipeline里再加一個ServerBootstrapAcceptor對象，它也繼承於ChannelInboundHandler接口，這個對象的作用我們會在文末介紹。

 

2. register方法

register方法的實際調用者為前文聲明的boss NioEventLoopGroup，其register實現位於MultithreadEventLoopGroup中，調用鏈如下：

MultithreadEventLoopGroup.register()

    @Override
    public ChannelFuture register(Channel channel) {
        return next().register(channel);//next方法會從boss NioEventLoopGroup管理的NioEventLoop中挑一個並返回，所以調用的是NioEventLoop.register()方法。而其實際實現位於SingleThreadEventLoop中
    }

SingleThreadEventLoop.register()

    @Override
    public ChannelFuture register(Channel channel) {
        return register(new DefaultChannelPromise(channel, this));
    }

    @Override
    public ChannelFuture register(final ChannelPromise promise) {
        ObjectUtil.checkNotNull(promise, "promise");
        promise.channel().unsafe().register(this, promise);//調用channel的unsafe()方法，通過單步可以知道，這里會獲得一個AbstractNioMessageChannel.NioMessageUnsafe對象，這個對象是在NioServerSocketChannel初始化時創建的。但是register()方法的實際實現位於AbstractUnsafe中
        return promise;
    }


AbstractUnsafe.register()

        @Override
        public final void register(EventLoop eventLoop, final ChannelPromise promise) {
            if (eventLoop == null) {
                throw new NullPointerException("eventLoop");
            }
            if (isRegistered()) {
                promise.setFailure(new IllegalStateException("registered to an event loop already"));
                return;
            }
            if (!isCompatible(eventLoop)) {
                promise.setFailure(
                        new IllegalStateException("incompatible event loop type: " + eventLoop.getClass().getName()));
                return;
            }

            AbstractChannel.this.eventLoop = eventLoop;

            if (eventLoop.inEventLoop()) {
                register0(promise);//如果當前線程是EventLoop線程，則直接調用register0方法
            } else {
                try {
                    eventLoop.execute(new Runnable() {//否則給EventLoop提交一個task，最終還是調用register0方法
                        @Override
                        public void run() {
                            register0(promise);
                        }
                    });
                } catch (Throwable t) {
                    logger.warn(
                            "Force-closing a channel whose registration task was not accepted by an event loop: {}",
                            AbstractChannel.this, t);
                    closeForcibly();
                    closeFuture.setClosed();
                    safeSetFailure(promise, t);
                }
            }
        }

        private void register0(ChannelPromise promise) {
            try {
                // check if the channel is still open as it could be closed in the mean time when the register
                // call was outside of the eventLoop
                if (!promise.setUncancellable() || !ensureOpen(promise)) {
                    return;
                }
                boolean firstRegistration = neverRegistered;
                doRegister();//將channel注冊到Selector，具體實現位於AbstractNioChannel中
                neverRegistered = false;
                registered = true;

                // Ensure we call handlerAdded(...) before we actually notify the promise. This is needed as the
                // user may already fire events through the pipeline in the ChannelFutureListener.
                pipeline.invokeHandlerAddedIfNeeded();//這里會將ServerBootstrapAcceptor添加到當前Channel中

                safeSetSuccess(promise);
                pipeline.fireChannelRegistered();//觸發pipeline的channelRegistered事件
                // Only fire a channelActive if the channel has never been registered. This prevents firing
                // multiple channel actives if the channel is deregistered and re-registered.
                if (isActive()) {
                    if (firstRegistration) {
                        pipeline.fireChannelActive();
                    } else if (config().isAutoRead()) {
                        // This channel was registered before and autoRead() is set. This means we need to begin read
                        // again so that we process inbound data.
                        //
                        // See https://github.com/netty/netty/issues/4805
                        beginRead();
                    }
                }
            } catch (Throwable t) {
                // Close the channel directly to avoid FD leak.
                closeForcibly();
                closeFuture.setClosed();
                safeSetFailure(promise, t);
            }
        }


AbstractNioChannel.doRegister()

    @Override
    protected void doRegister() throws Exception {
        boolean selected = false;
        for (;;) {
            try {
                selectionKey = javaChannel().register(eventLoop().unwrappedSelector(), 0, this);//終於，調用JDK NIO的方法，將傳入的channel綁定到event loop關聯的Selector上，這個Selector是在NioEventLoop初始化時構造的。需要注意的是這里設置的interest ops是0，此時沒有任何作用，后續會在doBind0方法里做具體設置
                return;
            } catch (CancelledKeyException e) {
                if (!selected) {
                    // Force the Selector to select now as the "canceled" SelectionKey may still be
                    // cached and not removed because no Select.select(..) operation was called yet.
                    eventLoop().selectNow();
                    selected = true;
                } else {
                    // We forced a select operation on the selector before but the SelectionKey is still cached
                    // for whatever reason. JDK bug ?
                    throw e;
                }
            }
        }
    }

調用鏈很長，干的事情倒是比較簡單：將傳入的Channel注冊到某個EventLoop關聯的Selector上，然后觸發一些相關的回調函數。

 

3. doBind0方法

調用鏈如下：

AbstractBootstrap.doBind0()

    private static void doBind0(
            final ChannelFuture regFuture, final Channel channel,
            final SocketAddress localAddress, final ChannelPromise promise) {

        // This method is invoked before channelRegistered() is triggered.  Give user handlers a chance to set up
        // the pipeline in its channelRegistered() implementation.
        channel.eventLoop().execute(new Runnable() {
            @Override
            public void run() {
                if (regFuture.isSuccess()) {
                    channel.bind(localAddress, promise).addListener(ChannelFutureListener.CLOSE_ON_FAILURE);
                } else {
                    promise.setFailure(regFuture.cause());
                }
            }
        });
    }

AbstractChannel.bind()

    @Override
    public ChannelFuture bind(SocketAddress localAddress, ChannelPromise promise) {
        return pipeline.bind(localAddress, promise);
    }

DefaultChannelPipeline.bind()

    @Override
    public final ChannelFuture bind(SocketAddress localAddress, ChannelPromise promise) {
        return tail.bind(localAddress, promise);
    }

AbstractChannelHandlerContext.bind()

    @Override
    public ChannelFuture bind(final SocketAddress localAddress, final ChannelPromise promise) {
        if (localAddress == null) {
            throw new NullPointerException("localAddress");
        }
        if (isNotValidPromise(promise, false)) {
            // cancelled
            return promise;
        }

        final AbstractChannelHandlerContext next = findContextOutbound();
        EventExecutor executor = next.executor();
        if (executor.inEventLoop()) {//根據當前線程是否為event loop線程而采取策略
            next.invokeBind(localAddress, promise);
        } else {
            safeExecute(executor, new Runnable() {
                @Override
                public void run() {
                    next.invokeBind(localAddress, promise);
                }
            }, promise, null);
        }
        return promise;
    }

    private void invokeBind(SocketAddress localAddress, ChannelPromise promise) {
        if (invokeHandler()) {
            try {
                ((ChannelOutboundHandler) handler()).bind(this, localAddress, promise);
            } catch (Throwable t) {
                notifyOutboundHandlerException(t, promise);
            }
        } else {
            bind(localAddress, promise);
        }
    }

DefaultChannelPipeline.bind()
        @Override
        public void bind(
                ChannelHandlerContext ctx, SocketAddress localAddress, ChannelPromise promise)
                throws Exception {
            unsafe.bind(localAddress, promise);//這里的unsafe是AbstractNioMessageChannel.NioMessageUnsafe
        }

AbstractChannel.bind()
        @Override
        public final void bind(final SocketAddress localAddress, final ChannelPromise promise) {
            assertEventLoop();

            if (!promise.setUncancellable() || !ensureOpen(promise)) {
                return;
            }

            // See: https://github.com/netty/netty/issues/576
            if (Boolean.TRUE.equals(config().getOption(ChannelOption.SO_BROADCAST)) &&
                localAddress instanceof InetSocketAddress &&
                !((InetSocketAddress) localAddress).getAddress().isAnyLocalAddress() &&
                !PlatformDependent.isWindows() && !PlatformDependent.maybeSuperUser()) {
                // Warn a user about the fact that a non-root user can't receive a
                // broadcast packet on *nix if the socket is bound on non-wildcard address.
                logger.warn(
                        "A non-root user can't receive a broadcast packet if the socket " +
                        "is not bound to a wildcard address; binding to a non-wildcard " +
                        "address (" + localAddress + ") anyway as requested.");
            }

            boolean wasActive = isActive();
            try {
                doBind(localAddress);
            } catch (Throwable t) {
                safeSetFailure(promise, t);
                closeIfClosed();
                return;
            }

            if (!wasActive && isActive()) {
                invokeLater(new Runnable() {
                    @Override
                    public void run() {
                        pipeline.fireChannelActive();//這里會設置對ACCEPT事件的監聽，后文再做分析
                    }
                });
            }

            safeSetSuccess(promise);
        }


NioServerSocketChannel.bind()
    @Override
    protected void doBind(SocketAddress localAddress) throws Exception {
        if (PlatformDependent.javaVersion() >= 7) {
            javaChannel().bind(localAddress, config.getBacklog());//調用JDK的NIO提供的方法，將傳入的channel與指定的端口綁定，並設置backlog大小
        } else {
            javaChannel().socket().bind(localAddress, config.getBacklog());
        }
    }

很長很長的調用鏈，最終做的事情是調用JDK NIO提供的bind函數，將channel與指定的端口綁定。

 

在AbstractChannel.bind()方法中，提交了一個異步任務，里面只有一行代碼：pipeline.fireChannelActive()，這行代碼經過一系列調用之后，會執行AbstractChannel.AbstractUnsafe.doBeginRead()方法

然后又會執行到AbstractNioChannel.doBeginRead()方法，其代碼如下：

    @Override
    protected void doBeginRead() throws Exception {
        // Channel.read() or ChannelHandlerContext.read() was called
        final SelectionKey selectionKey = this.selectionKey;
        if (!selectionKey.isValid()) {
            return;
        }

        readPending = true;

        final int interestOps = selectionKey.interestOps();
        if ((interestOps & readInterestOp) == 0) {
            selectionKey.interestOps(interestOps | readInterestOp);//設置interestOps
        }
    }

雖然此時selectionKey.interestOps()還沒有被設置，但是readInterestOp是一個全局變量，在NioServerSocketChannel初始化的時候就會被設置為SelectionKey.OP_ACCEPT。

於是在doBeginRead方法中，Channel所關注的IO事件就會被設置為ACCEPT事件了。

這樣如果有客戶端連接進來，就會觸發關聯的EventLoop里的相關代碼，並作出處理了。

 

4. ServerBootstrapAcceptor

ServerBootstrapAcceptor的關鍵代碼是channelRead方法

        @Override
        @SuppressWarnings("unchecked")
        public void channelRead(ChannelHandlerContext ctx, Object msg) {
            final Channel child = (Channel) msg;

            child.pipeline().addLast(childHandler);//給child channel的pipeline添加用戶自定義的handler

            setChannelOptions(child, childOptions, logger);//設置child channel的options

            for (Entry<AttributeKey<?>, Object> e: childAttrs) {
                child.attr((AttributeKey<Object>) e.getKey()).set(e.getValue());//設置child channel的attribute
            }

            try {//將child channel注冊到worker event loop group里，Netty會根據round-robin算法選擇一個worker線程來做綁定
                childGroup.register(child).addListener(new ChannelFutureListener() {
                    @Override
                    public void operationComplete(ChannelFuture future) throws Exception {
                        if (!future.isSuccess()) {
                            forceClose(child, future.cause());
                        }
                    }
                });
            } catch (Throwable t) {
                forceClose(child, t);
            }
        }

在服務器收到新鏈接的時候，這個函數會被觸發，然后設置Channel的各種屬性與關聯的pipeline

這個Channel接着會被交付給worker eventloop group里的一個worker，然后這個Channel上發生的任何讀寫事件都是由這個worker來處理了